This Web page has been archived on the Web.

1999 April Report of the Auditor General of Canada

Main Points

National Child Benefit

Background and other observations

6.3 One of the clear achievements of the negotiations leading to the NCB was the partners' agreement on the overall goals of both the federal and the provincial elements of the program. They also have committed to a new kind of joint accountability to the public. What is distinctive about this commitment is that no level of government is more responsible for reporting on the results than another. Each is accountable for the overall program.

6.4 It will be a significant challenge in the first few years, before evaluation results are available, to demonstrate precisely how the National Child Benefit has contributed to reduced depth of poverty and increased employment among its recipients. In assessing progress, the NCB accountability report will also need to be clear about any trade-offs among goals and to discuss the implications. Otherwise, readers may expect that all goals can be achieved at the same time.

6.5 There are potential gaps in the quality of financial and other information. The credibility of the accountability reports depends on the comparability, accuracy and verifiability of information from all parties - not only those who negotiated the arrangement (provinces, territories and the federal government) but also those who are responsible for designing and implementing only specific sub-programs (Ontario municipalities and First Nations). This is also a challenge for audit offices that serve the partners in this arrangement.

6.6 We think the key ingredients in reporting the NCB's results to the public are transparency and adequacy of the information. This means:

• appropriate care is taken to ensure that the information is credible and, at least for financial information, subject to audit;
• data are adequate to determine if the program's overall goals are being achieved; and
• data are sufficiently comparable that outcomes of different provincial approaches can be compared.

The Department has said that this chapter will be of considerable assistance in its work to build an accountability regime for the NCB. Some issues are already being addressed. The remaining issues identified in this study will be raised by the federal government in its capacity as co-chair of the federal-provincial-territorial group working on the NCB.

Employability Assistance for People with Disabilities

Background and other observations

6.9 Human Resources Development Canada (HRDC) and provincial partners have set about to solve implementation issues in a collaborative fashion. For example, the provinces have committed to annual accountability to HRDC. However, so far there is no specific commitment to the format, substance, or timing of any overall annual report, nor are there specific goals or targets over any time frame. We recommend that HRDC prepare a brief overall annual report for EAPD, comparing activities, expenditures, program outputs and performance of the partners.

6.10 The partners still have important work to do. When so many jurisdictions are involved, those with oversight and audit responsibilities face the challenge of helping to ensure that the quality of financial and performance reporting is maintained and enhanced.

The Department has said that the case study will be useful in its work with provinces. It agrees in principle with the recommendation to produce an overall annual report on EAPD expenditures and performance. It pointed out that it will be important to note that EAPD is only one part of overall programming for people with disabilities.

Preface

This chapter looks at the challenges for accountability that characterize the two recently implemented social programs that are already part of the social union: the National Child Benefit and Employability Assistance for People with Disabilities.

The case studies

6.12 Employability Assistance for Persons with Disabilities (EAPD) is also cited as an example of a social union program. It is a relatively more traditional form of federal cost-sharing in provincially delivered programs. Its aim is to help people with disabilities overcome the barriers they face in the work force by helping them to prepare for, obtain and maintain employment.

Challenges for accountability are great

6.14 For EAPD, provinces have committed in bilateral agreements to publish reports annually starting in 1999-2000. They have also agreed to share and make public the results of evaluations and to conduct a review of the arrangement at the end of its third year, in 2001.

6.15 Notwithstanding these commitments, it is not clear exactly how accountability for overall results of these programs will be achieved. An important question for this Office is, How should the Auditor General, in fulfilling his obligations to Parliament, look at programs that are not fully federal? Our Office clearly has an interest in arrangements that use federal resources. Parliament expects us to cover such arrangements in our work and to report on them (see Appendix A ). Since all levels of government agree on the need to be accountable to citizens and legislatures for these programs, our task is simply to assist in this endeavour. The case studies in this chapter show that there is no single best approach to demonstrating accountability, but there are some basic principles that can usefully be followed.

Canada's approach to social programs is changing

6.17 Annual Premiers' Conferences. Each summer, premiers meet at Annual Premiers' Conferences to discuss matters of mutual interest without federal government participation. At the 1995 Premiers' Conference, after major cuts in federal transfers to provinces had been announced, premiers reiterated their desire to work together on national matters affecting areas of provincial jurisdiction and to speak with a common voice in the national debate on social policy reform.

6.18 Ministerial social policy councils. In 1995, premiers formed a Ministerial Council on Social Policy Renewal to deal specifically with social policy issues. At the 1996 Premiers' Conference, they invited the federal government to join them in establishing a reconstituted Federal-Provincial-Territorial Council on Social Policy Renewal to advance the agenda of collaboration. This Council oversees social policy initiatives involving more than one department. In November 1996, it agreed that its first priorities were an integrated child benefit and improved services to persons with disabilities. The overall goals of co-operation were to reduce duplication and renew the social safety net.

6.19 First Ministers' Meetings. All premiers and the Prime Minister attend First Ministers' Meetings. In 1996, First Ministers began to discuss a collaborative and accountable approach to Canada's social programs. In communiqués, they referred to the National Child Benefit as part of a broader National Children's Agenda. They cited child poverty and persons with disabilities as collective priorities. The 1997 Speech from the Throne referred to the agreement among all levels of government to "work together to develop this broader agenda for children, including clear outcome measures by which to gauge success."

6.20 At the First Ministers' Meeting in December 1997, they reiterated the need to negotiate "a set of principles for social policy, such as mobility and monitoring social policy outcomes." They affirmed the National Child Benefit as a possible model for future social programs: the federal government would increase the Canada Child Tax Benefit, while provinces would use their resulting savings in welfare expenditures to undertake programs consistent with mutually agreed-upon goals. They also requested that social services ministers conclude the development of a vision statement and national framework for assistance to persons with disabilities.

6.21 Ministers Responsible for Social Services. After the 1996 First Ministers' Meeting, the task of program design for the National Child Benefit and Employability Assistance for People with Disabilities had been assigned to Ministers Responsible for Social Services. This group had many of the same members as the Council on Social Policy Renewal. In face to face meetings and through electronic communication, federal-provincial-territorial ministers and their public servants set about to design the NCB and EAPD.

6.22 Working groups of public servants. Officials were organized into the NCB Working Group and the EAPD Working Group. Each group was co-chaired by a provincial and a federal official. There were also sub-groups to work on specific issues. While Human Resources Development Canada (HRDC) had lead responsibility at the federal level, the Department of Finance, Revenue Canada, Indian and Northern Affairs Canada and Health Canada were all represented. The groups reported to Ministers Responsible for Social Services and to the Council on Social Policy Renewal. Each member also reported to his or her superiors and minister.

6.23 Public consultation. Most of the ministerial meetings also had a public component: the decisions that ministers wished to make public were announced through press releases, interviews, and postings to the Internet (http://socialunion.gc.ca/menu_e.html). As well, the groups and their ministers received input from a variety of formal and informal consultations with experts, advocates for the poor and the disabled, and other non-governmental organizations.

Quebec's policies are consistent, even if not formally aligned

Focus of the study

• How can credible reporting on these two programs best be achieved?
• What are realistic expectations for measuring the program outcomes?
• Are there information gaps that will need to be filled?

6.27 Appropriate accountability arrangements must include the provision of relevant, verifiable information to parliamentarians and others to assess whether funds have been spent for approved purposes and expected outcomes have been achieved. Providing such information when many partners are involved is not an easy task. We believe that making some general suggestions while the accountability arrangements are still being implemented will help achieve appropriate arrangements in a timely fashion.

6.28 Further details on our approach to the case studies can be found in About the Study at the end of the chapter. Appendix A contains information on our earlier work that addressed related issues in HRDC, more general issues of performance measurement and the quality of provincial accountability information in a federal-provincial context. Appendix B provides historical context for the NCB. Appendix C compares the two programs.

The National Child Benefit

Introduction

6.30 Another important feature of the NCB is that all participants have committed to joint public reporting on the program and on the extent to which it is achieving its intended outcomes. What is distinctive about this commitment is that no level of government is more responsible than another for reporting on the results. All are responsible for the overall product. We believe that this is the first time this kind of co-operative effort has been undertaken in Canada.

6.31 The case study sought to:

• help develop an awareness of what has been achieved so far with the National Child Benefit, and of the factors that may delay or constrain the establishment of appropriate accountability arrangements and credible measurement of results; and
• assist the reader of the annual "public accountability reports" by discussing what can reasonably be expected from them.

NCB supplement: The federal contribution

6.33 The federal contribution to the NCB is an estimate based on economic models and Revenue Canada data. The amounts of the NCB supplement that will actually be paid depend on family incomes in the previous year. We did not verify the models, and it is too early to know if they have correctly anticipated total levels of benefits. Variances could also occur because of economic and labour market changes.

6.34 Although millions of non-poor families receive the base Canada Child Tax Benefit, only low-income families benefit from the NCB supplement. This supplement is being introduced over several years. In 1997 the Working Income Supplement benefit to working poor families was restructured from a flat $500 per family to a per child basis: $605 for the first child, an additional $405 for the second child and $330 more for each additional child. In 1998 this became the NCB supplement to all families with income under $20,922. The supplement increased the base CCTB to a maximum of $1,625 for one child, $3,050 for two and $4,475 for three.

6.35 However, as a tax benefit recipients could also receive less than the maximum amount. For each dollar of net income over $20,921, the maximum NCB supplement is phased out by 12.1 percent for a one-child family, 20.2 percent for a two-child family and 26.8 percent for larger families, reducing to zero when income reaches $25,922. The 1999 Budget maintained approximately the same rates of phase-out when it increased the maximum NCB supplement by an additional $180 per child in 1999 and a further $170 per child in 2000. With a larger benefit to be phased out, the cut-off point at which the supplement is reduced to zero will increase to $27,750 in 1999 and $29,590 in 2000.

6.36 The 1997 federal Budget indicated (see Appendix B ) that although the intent of the NCB was to reduce social assistance by the amount of the supplement, a family on social assistance would be no worse off as a result of the NCB. However, the discontinued Working Income Supplement was not counted as income, whereas the NCB supplement was to be considered income and deducted dollar for dollar from social assistance benefits in most jurisdictions. Therefore, some families on social assistance who had received the Working Income Supplement in 1997 would potentially receive less in 1998. To avoid this, provinces and the federal government agreed to share 1998-99 costs of transitional assistance (under provincial social assistance budgets) for families whose income would otherwise have declined. It is not known whether this arrangement will continue beyond June 1999.

Reinvestments: The contributions of provincial, territorial, municipal and First Nations governments

• income support programs and tax measures for families with children and with incomes below certain levels;
• earnings supplements for families with children and with employment income in a specified range;
• extension to working poor families of in-kind benefits available for children in families receiving social assistance (for example, health benefits);
• improvement in overall child benefits to families receiving social assistance;
• social services, such as child care, that support parents' attachment to the work force; and
• other initiatives aimed at preventing and reducing child poverty, such as child nutrition and teen parent programs.

6.39 Not all provinces reduced social assistance payments by the full amount of the NCB supplement. Newfoundland and New Brunswick chose to maintain social assistance payments at their previous levels, and thus had no savings to reinvest. Nevertheless, each instituted additional programs aimed at low-income families: New Brunswick allocated a portion of what it would have saved, and Newfoundland the entire amount. Although Alberta initially reduced social assistance payments by the amount of the NCB supplement, it found that it was spending less than anticipated on its main reinvestment initiative, a child health benefit. It therefore increased shelter benefits for families on social assistance who have children.

6.40 In June 1998, provinces (except Quebec) and territories announced the level of their commitments to the NCB: a total of $329.4 million between July 1998 and June 1999. (These commitments will increase in July 1999 as the NCB supplement increases.) The First Nations commitment is $27.8 million for the same period.

Observations

NCB Goals Involve Trade-offs and Performance Measurement Challenges

6.42 The partners negotiated three objectives for the National Child Benefit:

• reduce the depth of child poverty;
• increase parents' attachment to the work force; and
• reduce administrative overlap and duplication.

Reducing the depth of poverty

6.45 One key question for assessing the success of the NCB will be, How deep is the depth of poverty? As defined in the preceding paragraph, depth of poverty is the amount by which family income is below a poverty line. Using Statistics Canada's low-income cut-off as a poverty line, HRDC estimates that in 1996 the aggregate depth of poverty for all families with children living below that line was $7 billion. (The total depth of poverty is greater because it also includes persons without children.)

6.46 A second key question will be, How much has the NCB reduced the depth of poverty? The federal commitment for the NCB supplement to the base Canada Child Tax Benefit is $850 million beginning in July 1998, rising to $1.7 billion beginning in July 2000. The depth of poverty could theoretically be reduced by the entire amount if the value of all the provincial benefits were included. (Some provincial NCB programs, aimed at lowering the welfare wall by improving health benefits or subsidizing child care, might improve the economic situation of poor working families without providing direct income.) The depth of poverty could also be reduced by more than the supplement alone by counting the additional amounts earned by parents who enter the work force as a direct result of the NCB.

6.47 The choice of poverty measure has an impact on measures of program performance. The resources required to meet any specific target for reducing the depth of poverty and to measure the degree of program success for a given level of expenditure will depend on how poverty is defined. A higher poverty line will mean that the aggregate depth of poverty is greater and therefore that more resources are required to reduce it. Ministers have decided to use two measures of low income for the first NCB report: Statistics Canada's low-income cut-offs (LICOs) and the low-income measure (LIM). While this Office and other credible bodies may comment on overall criteria, data quality and other implementation and administrative issues, how and where a poverty line is constructed depends in the final analysis on fundamental social and political value judgments, not just technical considerations.

Increasing parents' attachment to the work force

Reducing overlap and duplication

There are trade-offs in the short term between two goals

6.51 In their accountability reports on the NCB, the partners will need to clearly discuss any such trade-offs and their implications. Otherwise, readers of the reports may expect that all the goals can be achieved simultaneously.

The goals are general, without specific targets or time frames

The relationship of goals to performance expectations can be better explained

6.54 In the NCB, many of these relationships are made clear. For example, in the September 1997 background booklet, "Building a Better Future for Canadian Children", good use is made of research demonstrating the importance of early childhood intervention (see Appendix B ). Evidence that parental attachment to the work force contributes to better outcomes for children was not presented, perhaps because research is not yet conclusive on this issue (as we note in paragraph 6.48 ). The issue is very complex. Recent HRDC-sponsored research tends to support the conclusion that work force attachment does not negatively affect outcomes for children. Positive outcomes are correlated with education and with parents who read to their children, and less positive outcomes with being on welfare. However, it is not easy to sort out cause and effect so that social policy can be designed. This is why HRDC continues to sponsor such research.

Accountability, Results Measurement, Audit and Evaluation Issues

Who is accountable to whom?

6.57 A key mechanism for demonstrating accountability is credible reporting. While mechanisms for reporting to the public tend to vary according to the nature of each program, arrangements for reporting to legislatures are well defined and involve auditors general and legislative committees. The partners in the NCB state that one advantage of accountability to the public is that it will "minimize administrative reporting." However, accountability is unlikely to be served best by minimized administrative reporting. Administrative data are part of the accountability and operational relationship between partners. They are also necessary for reporting both outputs and performance outcomes. The goal, therefore, should be to report appropriately. At the same time, if accountability to the public means that performance reports on the overall program present the necessary relevant, credible and auditable information to the public rather than to internal administrations, this would be one appropriate way to deal with the NCB's interjurisdictional aspects.

Key Accountability Ingredient: Adequate Information

6.59 However, smaller provinces and perhaps First Nations and municipalities may lack the resources to obtain accurate, comparable and relevant data and to verify them. This is an issue that can be addressed in the same collaborative spirit as has prevailed in the NCB design and implementation to date. For example, if asked, the federal government or larger provinces might work with partners to help them build capacity.

6.60 Data do not have to be perfect, but they should be as good as possible. One important reason is that in negotiations it is important for the parties to have accurate information about their own and their partners' contributions. This information is also useful for operational purposes. Later on, in reporting results and expenditures, data from several jurisdictions and levels of government will have to be aggregated and synthesized to assess whether progress is being made. The key ingredient to credible reporting, therefore, is that each partner take steps to ensure that:

• the information presented is credible and, at least for financial information, subject to audit;
• data are adequate to determine if overall program goals are being achieved; and
• data are sufficiently comparable that outcomes of different provincial approaches can be compared.

6.61 The reinvestments raise particularly challenging questions of compliance, comparability and verifiability of data, and audit assurance. Clearly, each partner is responsible for data on its own contributions and, where assurance is required, can arrange for its own auditor to verify the information. Another possibility is that the partners' auditors can co-ordinate this task themselves. Whatever arrangement is finally arrived at, each partner is responsible for obtaining assurance from other partners that all the information is accurate and relevant. It will be important that readers know what information has been verified or is consistent with independent analysis, what has been or will be audited or otherwise independently verified, and what information must be accepted on trust.

6.62 We examined information made available to federal officials about provincial reinvestments during 1998, as the NCB was being implemented. Although we did not perform a formal audit, we do have some preliminary observations.

Reinvestments are within the parameters of the NCB goals and objectives but there are information gaps

6.64 First Nations and Ontario municipalities will have funds available for reinvestments, but did not participate in the negotiations or the design of the accountability framework. As of February 1999, they were still formulating plans and had not announced them to the general public. (According to Indian and Northern Affairs Canada, some First Nations reinvestment programs have been announced to their own communities.) We do not yet know what measures they, like the provinces and territories, will take to ensure that their information is of adequate quality.

6.65 With respect to First Nations programs, Indian and Northern Affairs Canada has informed us that it believes the steps taken to date will fulfil the obligation to account for the performance of the NCB in First Nations communities. It points out that since the reporting required from First Nations is based on the fiscal year, it will not receive information on the operations of reinvestment programs until July 1999. Indian and Northern Affairs believes that it will then be in a better position to assess the quality of data and the capacity of First Nations to provide it. It has said it is committed to working in partnership with First Nations to assess and evaluate the long-term impacts of the NCB on reserves.

6.66 First Nations directly administer at the community level an estimated $592 million in social assistance programs. Indian and Northern Affairs has indicated that the First Nations commitment for on-reserve reinvestments in the first full year is $27.8 million - the fourth-largest reinvestment amount after Ontario, Quebec, and British Columbia. First Nations communities are very diverse in size, population, socio-economic conditions, governance and capacity. Indian and Northern Affairs points out that the introduction of a largely federal-provincial-territorial program into this diverse population, with concomitant reporting and accountability expectations, presents unique challenges. It has informed us that its regional offices have been working with First Nations and, where appropriate, with provinces and territories to develop mutually acceptable frameworks that will guide the implementation, funding and reporting requirements of the NCB reinvestment initiatives. Its aim is to have frameworks that are both consistent with the objectives of the NCB and flexible enough to address the different priorities and needs of First Nations communities.

6.67 Because municipalities pay 20 percent of social assistance expenditures, Ontario has asked them to design their own NCB reinvestment programs. Those will account for 20 percent of Ontario's reinvestments. At December 1998, municipalities had only begun to design their programs. The province has informed us that it is putting in place a requirement that municipalities provide it with the necessary information to demonstrate that they have met their obligations under the NCB. It indicated that it is a high priority for Ontario to have accurate and timely information on municipal strategies, expenditures, and program performance and it does not anticipate any gaps in information on municipal reinvestments.

6.68 Finally, there is no information about Quebec's reinvestments, since Quebec has participated in the discussions only as an observer. As we indicated in paragraph 6.24 , Quebec has instituted programs that would likely have qualified as reinvestments. HRDC estimates that if Quebec had been a party to the agreement, its reinvestment envelope would have been approximately $150 million.

Are reinvestments "new" money?

Are the data on the size of reinvestments reasonably accurate and is the degree of inaccuracy, if any, known and acceptable?

6.71 Estimating the amounts to be reinvested is not a straightforward task. For example, they depend on the number of children in families receiving social assistance, not on total population. Thus, Alberta plans to reinvest less per capita than Saskatchewan because it has fewer children in families on social assistance. Another complication is that reinvestment levels are related to the previous year's income of families on social assistance, because this is how the amount of a parent's Canada Child Tax Benefit is determined. Since families whose previous year's income was over $25,921 do not receive the NCB supplement, a province would not be entitled to reduce their social assistance. If the economy were to take a sudden downturn and the number of people on social assistance increased, the commitment to reinvestment would increase. However, the calculation would have to take into account that some of the new social assistance recipients might have had too high an income the previous year to qualify for the NCB supplement, and for these individuals there would be no savings to reinvest.

6.72 Federal officials have told us they are satisfied that provincial estimates of planned reinvestments are accurate and that correct levels of reinvestment are planned. We have not verified these calculations, and would need more information to do so. We have not seen estimates of an acceptable range of accuracy for the projections of provincial reinvestments, nor have we seen plans to verify the actual levels. We have no reason to believe that the calculations are inaccurate or that planned and actual expenditures will be less than agreed, except perhaps in the first year (see paragraph 6.80 ).

Are there known data gaps and problems and are plans in place to fix them?

6.74 One of the key remaining issues is how to obtain and link data on employment, social assistance, income tax and family structure without compromising privacy. For example, a two-way "data exchange" between provinces or municipalities and Revenue Canada could help reconcile differences in data on family structure, movement of children between caregivers, etc., so that overpayments and underpayments could be dealt with quickly and accurately, perhaps automatically. To enhance comparability and increase efficiency, there may be a need to reconcile differences between data systems of the different jurisdictions. A related question will be whether data will be available for evaluation and analysis.

Means to monitor and inform all participants and stakeholders

What are the means for dispute settlement?

The First Accountability and Performance Report

6.78 The partners stated that the goals for the report are to:

• place the NCB in the context of child poverty;
• establish some baseline data;
• provide some statistics on program recipients and expenditures; and
• provide some preliminary information on how the program will be monitored and assessed, what indicators will be used, and what studies will be undertaken.

Data for outcome indicators need to be accurate and comparable

6.81 In the accountability and performance reports, the federal government and its partners have, in our view, individually and collectively committed to ensuring that relevant and comparable data are available. An important question will be whether this is accomplished.

6.82 The partners have announced that the first report would contain two kinds of measures:

• program statistics to indicate the inputs and outputs of the NCB - who has benefited, overall expenditures, and so forth; and
• outcome indicators - to assess the impact of the NCB on low-income families with children.

Outcome indicators could include:

• Changes in the percentage of the total income of low-income families that result from employment:

By monitoring this outcome indicator, we would be able to measure the degree to which low-income families with children increase their attachment to the workforce over time.

• Changes in the number of families with children on social assistance:

This indicator would reflect the degree to which low-income families with children move off social assistance, as compared to being in the workforce and/or receiving income supplements.

• Changes in depth of poverty:

The NCB is an important step being taken to reduce the depth of child poverty over time. The depth of poverty indicates the income gap for families with children measured against one or more lines of low-income measurement.

6.84 Experts and advocacy groups were asked for their comments on these proposed measures. In response to their suggestions, the report will likely contain revised measures of outcome, offer caveats about their accuracy and validity, and present additional relevant background information for assessing NCB success.

What the indicators tell us

6.86 In the booklet entitled "Building a Better Future for Canada's Children", the federal-provincial-territorial governments promise that the accountability and performance report will contain performance measures for the program as a whole:

Governments are now identifying performance outcomes that will measure the National Child Benefit's success in achieving its objectives. They are also evaluating how closely results can be linked to the investments made in the National Child Benefit, recognizing that this program will be only one of many factors that will affect the well-being of children and families.

Performance outcomes will be incorporated into public reports that will describe the National Child Benefit's success in meeting its objectives. These reports will ensure that governments are fully accountable to the public. They will also allow provinces and territories to evaluate and share the results of their various National Child Benefit reinvestment initiatives.

Role of evaluation

Accomplishments to Date

Reaching agreement: Clear evidence of common goals and trust

6.90 There was a good level of co-operation in all of these groups and committees. When ministers met, for example, they used common briefing notes agreed on by their officials for each agenda item. Comments of those involved suggest that these meetings met the criterion (discussed in Chapter 5 of this Report) of dealing with complex relationships with co-ordination, mutual trust and confidence.

6.91 A significant aspect of reaching agreement was the decision to link the goal of reducing child poverty to the goal of increasing parents' attachment to the work force. While this linkage makes performance measurement a greater challenge, it is clear that agreement on these policy goals has been a critical factor in the progress to date.

Implementation: Continuing commitment to outcome measures and accountability

Transparency and consultation: Public was informed at many key steps

6.94 At several points in the process, public announcements were made. An Internet site was established that provided all the documents released after meetings of Ministers Responsible for Social Services (http://socialunion.gc.ca/ncb_e.html). All this was in addition to special efforts by Revenue Canada to inform the public and respond to its concerns in order to ensure as smooth a transition as possible when the new benefit was implemented in July 1998.

Conclusion

6.96 The challenge for all involved in this program is to be able to assure taxpayers that moneys are spent with due regard to economy and efficiency, and with appropriate means to measure and report on the effectiveness of not only individual parts of the program but also the program as a whole. This needs to be done in a way that respects the jurisdictional competence of the different parties involved. We believe that it is possible to do so and that many steps in the right direction have already been taken. Indeed, if the issues we have discussed are successfully addressed by all parties concerned (including audit offices), accountability for the expenditures and outcomes of the National Child Benefit will be just as rigorous and comprehensive as it would had only one jurisdiction been responsible.

How to achieve accountability for the NCB: Five suggestions

• Ask how and when we will know if desired results have been achieved . In other words, have reasonable expectations for the accountability reports-at current levels of funding, the NCB cannot be expected on its own to fully eliminate child poverty or the "welfare wall". Some impacts will not be visible in the first or even the second year, but means of collecting the data to show the impacts must be in place now.
• Ask how reliable the information is. How good are the data? How can one know? Information could be unverified and still be reliable, but in what circumstances should only those supplying the information make this judgment? Where there is potential for conflict of interest (an interest in reporting mainly positive results, for example) ask not only whether there is a need to consider obtaining independent assurance but also whether doing so would be worth the cost.
• Accept that the choice of outcome measures is partly political. It is reasonable to question whether a chosen measure is appropriate, but not whether final accountability for the choice should rest with elected officials. Consider asking under what circumstances it might be appropriate to delegate the responsibility to independent third parties for implementing or revising the measures and evaluating the results.
• Encourage transparency and openness by treating the publication of negative trends in outcomes as opportunities for learning or improving the program, not for laying blame.
• Accept that in an interjurisdictional situation like the NCB, accountability is shared. Seek answers from those best able to answer. Overall accountability can be achieved only through holding partners accountable for not only their own programs but also overall outcomes.

We can report that many of the issues raised by the Auditor General are being addressed by the federal government and its provincial and territorial partners. For example, outcome indicators and evaluation issues have been the subject of consultations with experts and advocacy groups and work will continue on these issues. In addition, the Federal-Provincial-Territorial NCB Working Group is currently preparing the first NCB Progress Report. All efforts are being made to ensure that the information contained in the Progress Report is timely, comparable, credible and complete. The remaining issues identified by the Auditor General will be raised by the federal government in its capacity as Co-Chair of the NCB Working Group.

Employability Assistance for People with Disabilities

Introduction

6.99 The EAPD initiative represents a small, targeted portion of total spending on persons with disabilities. Its objectives emphasize employability; that is, it supports programs aimed at helping people with disabilities achieve the capacity to enter and remain in the regular labour market, as distinct from supporting participation in sheltered workshops and related kinds of employment supports.

6.100 This case study focusses on the accountability arrangements of EAPD and the elements to be included in future reports on its results, particularly the performance indicators that have been identified and the challenges implicit in designing and using these indicators. Our aim was to assess whether EAPD has some of the characteristics of a collaborative arrangement, to identify the factors that may delay or constrain the achievement of appropriate accountability arrangements and to make suggestions for improvement at an early stage in the process of implementing this initiative.

6.101 As the population ages, the numbers of persons with disabilities will increase. The most recent Statistics Canada data (1991, based on the since-discontinued Health and Activity Limitation Survey) indicate that 16 percent of Canadians (or approximately 4.2 million people) report some level of disability. Fifty percent of those over 65 report some level of disability.

6.102 The last five years have seen major efforts to revise and improve services while reducing costs. The 1994 government-wide Program Review, the Mainstream '92 initiative and the 1996 report of the Federal Task Force on Disability Issues (the Scott Task Force) have all had a significant impact. The Standing Committee on Human Resources Development and Status of Persons with Disabilities produced four reports between 1990 and 1995, dealing with such matters as the legislative and regulatory framework, the continued "marginalization of persons with disabilities", improvements to the tax system, and the integration of persons with disabilities from the concept of a "warehouse" toward an "open house".

6.103 The process of arriving at a new program arose from federal-provincial-territorial discussions on social policy renewal. Annual Premiers' Conferences, First Ministers' Meetings, meetings of Ministers Responsible for Social Services, and meetings of the Federal-Provincial-Territorial Council on Social Policy Renewal all made it a priority to improve income support for disabled persons through increased employment. Indeed, one measure of program success would be savings realized by income support programs due to increased labour market participation by persons with disabilities. In addition to the clear benefits for persons with disabilities, their increased pay would mean increased government revenue and reduced income support expenditures over the long term. With this direction in mind, in February 1997 officials began to review programming then funded under the Vocational Rehabilitation of Disabled Persons Act .

6.104 The first order of business was to develop a mutually acceptable multilateral framework that would guide the negotiations of subsequent bilateral contribution agreements between the federal government and the provinces and territories. Three informal task teams worked in tandem to develop an accountability framework, proposals for changes to the funding allocation formula and a policy framework for EAPD. The chief policy thrusts are a clearer focus on employability and accountability, bolstered by commitments to measuring results and to ongoing evaluation of the program. HRDC held formal consultations with representatives of the community of people with disabilities in April and June 1997. Provinces also incorporated the input from their own consultations into the work of the Federal-Provincial-Territorial Working Group.

Observations

Governance and Accountability Arrangements

6.106 The partners agreed that the initiative would emphasize "annual accountability to consumers and the general public." As we discuss more fully in the case study on the National Child Benefit ( paragraph 6.56 ), accountability to the public need not diminish accountability to Parliament; the key is that information must be adequate so reporting is credible.

Performance measures are varied: Some present challenges to implement and use

6.108 The primary quantitative indicators of results focus on the "supply side" - increasing the number of employable persons with disabilities who are ready to enter the labour market. Other programs for persons with disabilities have dealt with the "demand side" of the equation - stimulating the demand for employees who are disabled by working with employers, business and community groups to change stereotypes, identify appropriate training and job opportunities and demonstrate how persons with disabilities can fit into the workplace. The challenge is to ascertain, when indicators show a positive or a negative trend, whether employment opportunities have been affected by external labour market factors (such as a general rise or fall in the unemployment rate) rather than by EAPD.

6.109 Our 1997 Report Chapter 22, Crown Corporations: Making Performance Measurement Work, describes a good performance indicator as one that is meaningful, reliable and practical. "Meaningful" means the indicator is understandable (clear, concrete/measurable, in context), relevant (related to objectives, significant, useful, attributable) and comparable (over time and with other organizations or standards). "Reliable", in this context, means it is an accurate representation (valid/free from bias) of what is being measured, is replicable or verifiable, is free from error or manipulation and balances other indicators. "Practical" means that it is feasible, both financially and temporally, to gather data.

6.110 The quantitative indicators described in the multilateral framework were developed through an analysis of EAPD's component programs, the input of the client community, and HRDC's experience with results indicators in the Employment Insurance program. The measures tend to focus on activities and intermediate outcomes rather than on long-term results. The indicators agreed on in the multilateral framework show some of the challenges that arise when trying to measure results, as the following paragraphs discuss.

6.111 Number of people employed, or sustained in employment in the event of vocational crises. This indicator actually contains two measures that have meaning and relevance, where it is feasible to gather the data. The bilateral agreements recognize this and split them. However, the task of defining the measures is not finished. For instance, the partners will need to define operationally what "sustained in employment" means.

6.112 Number of people actively participating in or successfully completing an EAPD program and, if unsuccessful, the reasons. The number of active participants is meaningful as a "throughput" measure, and is practical to collect. Determining why participants may not have succeeded could provide important information for lessons learned and program redesign. However, unless all jurisdictions have similar definitions of what it means to complete a program, and similar methods of data collection, the information will not necessarily be comparable.

6.113 Number of people not served, on waiting lists or unable to access interventions. This indicator in the multilateral framework contains three measures, the first and last of which were omitted from all bilateral agreements. The "number of people on waiting lists" is a measure of unfulfilled demand for services, but different standards and practices for compiling waiting lists in different jurisdictions will cause data variations, making comparisons difficult. Length of time on the waiting list is also important. Several bilateral agreements therefore added a measure for "waiting periods".

6.114 Savings to income support programs as a result of increased earnings through employment. This potentially valuable indicator presents major challenges of practicality in obtaining accurate and reliable data. The federal-provincial accountability task team recognized this complexity and proposed that "all provinces should be engaged in a discussion around the methodological design required to capture this element". The idea behind the measure is simple - a person whose employment income increases as a result of this program will need less income support or perhaps none, enabling government to save correspondingly. However, the question immediately arises as to whether, for how long, and in what proportion the disabled person's increase in income is to be claimed as "savings" to income support programs. The answer will require good research into the causal linkages among EAPD, employment earnings and savings to income support programs. Virtually all bilateral agreements state that joint evaluation of this indicator will be "subject to the feasibility of collecting the data and establishing the systems linkages required". A further complication is that federal and provincial privacy laws must be respected.

6.115 Number of people who have received support and have maintained employment or advanced in their jobs. This indicator was incorporated into only one bilateral agreement. The other indicators already capture most of the required information. Advancement in employment would be useful to know but actually obtaining the data would present significant challenges: the meaning of "advancement" would have to be defined in ways that would allow for a comparison of the results of different programs in various jurisdictions.

6.116 In general, it is too early to assess how successfully the program has addressed the challenge of gathering and presenting data that are reliable and comparable across jurisdictions and that reflect the program's performance. The partners are aware of the challenge. For example, they have agreed to modify in the bilateral agreements the primary indicators agreed to in the multilateral framework. They agreed that some of the indicators would be reported in annual reports; participants would use formal evaluation to measure the others and to assess the reasons for any lack of success. This is clearly a step in the right direction.

Roles and responsibilities are stated in general terms

It is too early to say whether reporting will be credible

6.119 Readers of accountability reports will want to ask the following kinds of questions.

6.120 Are the data presented reasonably accurate and is the degree of any inaccuracy known and acceptable? One important aspect of reporting credibly on program results is describing the reliability and validity of the information presented and the methods used to collect and verify the data, giving an indication of their accuracy, and noting any related factors that may have influenced the results. The technical and practical challenges of obtaining valid, reliable and consistent data are many, including the challenges of attributing results to the measures and tracking clients. The parties will need to continue working to resolve many of these issues if information on results is to be credible, timely, meaningful and useful.

6.121 Is there knowledge of what data gaps and problems exist and are there plans in place to fix them? There may be, for example, differences among the parties in their standards of program completion, measurements of waiting lists, and calculations of savings to income support programs. We are aware that this is a concern for officials, but we do not know precisely what plans they have to deal with these issues.

6.122 Is there reasonable provision for review, audit and program evaluation? The parties have agreed to review the arrangements under the EAPD agreements after the end of the third year (2000-01). We expect that all parties will routinely review and adjust their programs, in order to improve services and capitalize on best practices and lessons learned from pilot projects. The multilateral framework commits the partners to reasonable review efforts (which normally would result in adjustments to improve efficiency and effectiveness, and in more transparency). The key will be how these review provisions in the framework are implemented.

6.123 The bilateral agreements require the provinces to produce audited financial statements of program expenditures. These statements are sent to HRDC for approval and reimbursement of expenses. There is no requirement for independent verification of other performance information. We believe that the federal government should seek some assurance about the accuracy of the systems that provide the information.

6.124 The bilateral agreements also require the provinces each to produce annual reports and multi-year program and expenditure plans. These annual reports are to be published. However, it is not clear that they will contain comparative or analytical cross-provincial data. We believe this type of data would contribute to accountability for the federal expenditure, by providing information to identify best practices and to make valid comparisons of provincial program performance. We recognize that comparisons can be misinterpreted and oversimplified. It will be important to make the appropriate qualifications in the overall annual report that we recommend HRDC prepare (see paragraph 6.129 ).

6.125 The agreements also require provinces to conduct program evaluations of longer-term impacts (though no due dates are specified) and to measure progress using the identified performance indicators. These evaluations may serve to confirm the provinces' performance information. HRDC, as one of the parties, agrees to undertake ongoing evaluations with its partners and to make the evaluation results public. The partners are aware of the challenges to fulfilling the evaluation requirement. A group of provincial and HRDC regional officials is working out the details of how, when and by whom the various aspects of evaluation will be undertaken. HRDC's Evaluation and Data Development group is actively involved in defining approaches to the EAPD evaluation and has developed a discussion draft Proposed Framework for the Evaluation of EAPD. Two federal-provincial accountability sub-working groups were created to develop options for the evaluation and to establish a common set of definitions for the results indicators. This presumably will involve identification of common issues, supporting data, time frames, roles and responsibilities, as well as an approach to national issues and a national synthesis of results. So far, the majority of provinces have agreed to proceed with an assessment of the feasibility of evaluations that would examine the design and delivery features of the EAPD interventions existing across Canada. This work may also determine if it will be possible, given the data being collected, to carry out a longitudinal survey of EAPD participants and to follow their outcomes.

Conclusion and Recommendation

Accountability, performance measures and evaluation remain at issue

6.127 There are no plans to publish any overall annual report. The first provincial annual reports will be produced for the year 1999-2000. However, no deadline has been set for tabling or submitting these reports and there is no provision for an overall annual report.

6.128 There are no specific goals or targets over any specified time frame. HRDC officials cite the lack of baseline data and the complexity of the services delivered by provinces and third parties as reasons for the lack of defined performance expectations or established targets. They also consider it more important to meet the needs of individuals than to set targets, though some provinces have made some projections. Stakeholders and officials are also concerned that setting such targets would encourage "creaming"- that is, selecting first the clients with the greatest chance of success so as to yield early positive results. This would discriminate against persons who need longer and/or more interventions to achieve success. While this means that overly simplistic targets are inappropriate, we believe that well-defined goals can, and still should, be established for each EAPD program.

6.129 To add to the annual reports by the partners, Human Resources Development Canada should, in collaboration with its partners, prepare a brief overall annual report on Employability Assistance for People with Disabilities, comparing partners' activities, expenditures, program outputs and performance.

6.130 The report should include:

• performance targets both for the program overall and for the individual provincial programs;
• a description of limitations in the accuracy and the comparability of data;
• an indication of which data have been audited;
• information on program adjustments and lessons learned from reviews of the federal-provincial arrangements;
• performance measurements; and
• evaluation studies.

Department's response: The Department appreciates the contribution of this case study to our work with provinces in implementing Employability Assistance for People with Disabilities (EAPD). The Auditor General has outlined many of the key challenges we face in meeting the commitment to accountability and results indicators agreed to by federal and provincial-territorial Ministers Responsible for Social Services. The case study provides a useful analysis that will inform our ongoing work with provinces.

The Department agrees in principle with the Auditor General's recommendation to produce an overall annual report on EAPD. Work is under way, in collaboration with provinces, to ensure consistency in reporting on EAPD-funded programs. The Department will consider the suggestions and comments of the Auditor General in the development of annual report. In preparing the reports and in measuring and comparing results and outcomes, it will be important to note that EAPD-funded programs constitute only part of overall programming for people with disabilities and should be considered in the unique context of each province.

About the Study

Objective and Scope

• How can credible reporting on these two programs best be achieved?
• What are realistic expectations for measuring the program outcomes?
• Are there information gaps that will need to be filled?

• an understanding of potential accountability gaps and the context and factors that may delay or constrain improvement in accountability and results measurement; and
• an appropriate response to public accountability and performance reports, reports on social outcomes, and jointly designed performance management systems of these programs.

Criteria

• indicators and measures should be clear and concrete; and
• information should be understandable, balanced, attributable, and reliable.

• Data on expenditures and for outcome indicators and evaluations are credible and comparable so that overall outcomes of the NCB can be measured and evaluated. The accountability report informs readers whether financial information has been audited, and whether arrangements are in place to ensure that due care has been taken with respect to the information presented, including data used for performance reporting.
• Reinvestments are within the parameters of the NCB goals and objectives, and represent new programs or enhancements to existing ones.
• There is knowledge of the data gaps and problems that exist and there are plans in place to fix them.
• There are means to monitor and inform all participants and stakeholders about individual reinvestment expenditures, outputs and outcomes, as well as the outcomes of the entire effort.

Approach

Study Team

Allison Fader
Martin Dompierre
Ali Jakubec

For further information, please contact Lewis Auerbach.

Appendix A

Auditor General's Related Work

Human Resources Development Canada (HRDC)

Performance reporting

Meeting the expectations of Canadians for programs that work and finding more cost-effective means of delivering them requires managing for results. When managing for results, ministers, senior officials, managers, staff, central agencies and Parliament make decisions based on what a program is achieving for Canadians - the results that citizens value - and at what cost. In particular, there is a focus on outcomes, that is, the benefits realized. Putting this approach into practice means agreeing on expected outcomes, measuring progress toward them, using the information and reporting results.

Quality of provincial information

More recently, in our 1996 Chapter 26, Canada Infrastructure Works Program - Lessons Learned, we observed that provincially supplied information had not been verified:

In the majority of files we examined, federal officials endorsed provincial assessments without requiring direct supporting information, or at least ensuring that appropriate information had been provided to, and analyzed by, the province concerned.

We believe that undefined and diffused accountability erodes the authority and responsibility of the federal, provincial and territorial governments and their officials and makes it difficult for anyone to measure their respective performance. It also means that accountability for Canada's progress toward its stabilization goal is obscured. Nevertheless, in our opinion the federal government remains accountable to Parliament for establishing an effective management structure to respond to climate change.

Appendix B

Historical Context - National Child Benefit

Early social programs of the `40s, `50s, and `60s such as Unemployment Insurance, Old Age Assistance and Family Allowance were direct federal expenditures administered entirely by the federal government. Each such program required an Act of Parliament and, unless funded separately like Unemployment Insurance, required annual appropriations as part of the federal Budget. The Office of the Auditor General can, and did, audit the expenditures as well as the administration of such programs. As well, parliamentarians had opportunities through the Budget debates to question the expenditures and the administration of these programs.

The 1978 revisions to the Family Allowance Act increased child allowances but made them taxable. Making child allowances taxable had the effect of targeting the benefit more toward lower and middle income families. From an accountability perspective, the relationship of Parliament and the nature of program delivery remained the same - the program was still an expenditure in the Budget cycle. The Child Tax Credit introduced in 1978 to further supplement the Family Allowance for families of modest means was an even more significant change. Because tax credits form a part of the income tax system, the credit did not require annual review by Parliament; the form of financial assistance was not direct payment but rather forgone revenue, that is, a tax expenditure.

In 1992, family allowances were eliminated completely and replaced by the Canada Child Tax Benefit. One consequence for accountability (as we observed in our 1994 Report Chapter 6, Study of Key Federal Social Programs) was that " because the new Child Tax Benefit is now delivered through the tax system, it is not included in budgetary spending. Thus, it becomes difficult to make comparisons or analyze trends in program spending in this area." In our 1998 Chapter 8, Effectiveness Measurement and Reporting in the Department of Finance (which is responsible for tax policy and evaluations of taxes and tax expenditures), we noted that effectiveness measuring and reporting of the child tax benefit were "insufficient".

Whatever the shortcomings of past child support programs, at least the role and responsibility of the federal government was clear. In the case of the 1992 changes, for example, a budget was presented and debated, and Parliament enacted the necessary changes to the Income Tax Act and replaced the Family Allowance Act .

Current situation

Policy context: Research results demonstrate importance of early childhood

Most Canadian children are doing well and are physically, emotionally and socially healthy. Unfortunately, some children are not doing as well. Many experience emotional, behavioural or learning problems that affect school performance and personal development. Others suffer from physical problems such as disease, disability and injury.

Such difficulties can and do affect children being raised in all family situations-in two-parent as well as single-parent families, in every neighbourhood and community throughout the country. But decades of research in Canada and abroad have shown that low family income is strongly linked with these problems. Compared to children from middle- and upper-income families, poor children run a greater risk of inadequate nutrition, mental and physical health problems, poor school achievement and difficulties with the law.

Living on a low income can exert a heavy toll not only on children and their parents but also on the economy and society. Although many low-income children grow up and out of poverty, its harmful effects in childhood can often linger long into their adult years. The costs include increased expenditures for welfare, employment insurance, social services, health care, the courts and the penal system; lost tax revenues; wasted productivity; and missed opportunities for both individuals and society.

The 1997 federal Budget

• Right now, the combined effect of federal and provincial programs is to reduce the child benefits of parents who leave welfare to enter the workforce. Parents should not be put in the position of penalizing their children in order to take a job.
• [The benefit would be directed to working families, who would see their incomes rise. Although the intention was to reduce social assistance by the amount of the benefit] ...the overall income of families on social assistance will be protected.
• In addition to being a win for Canadian children, the National Child Benefit system is a step forward in Canadian federalism, with the federal, provincial and territorial governments seizing on a good idea, setting common objectives and working as partners to secure better lives for our children.
• To accomplish this, an enriched federal benefit along the lines set out above will require provinces and territories to make offsetting reinvestments of provincial funds to assist children in low-income working families.

Appendix C

A Comparison - National Child Benefit and Employability Assistance for People with Disabilities

Similarities

• No specific legislative act governs either initiative; the parameters are an outcome of federal-provincial agreement.
• First Ministers agreed on the program goals.
• The program is not aimed at all members of the target group, but is primarily focussed on helping those who wish to enter or remain in the work force.
• Program design responsibilities are delegated from First Ministers to Ministers Responsible for Social Services to their officials.
• Provinces are responsible for program delivery (although many have arranged for Revenue Canada to deliver provincial and territorial tax benefits on their behalf).
• The federal government's role is to provide a substantial amount of the funding, to help establish an overall accountability framework, and to work with provinces to ensure that the program meets the basic overall objectives agreed to by the partners.

Differences

• In EAPD, the federal government's financial contribution goes to provinces whose programs meet the mutually agreed criteria. In the National Child Benefit, federal funds go directly to individuals.
• A formal, written multilateral framework for EAPD forms the basis for comprehensive bilateral agreements with provinces. For the NCB, the only bilateral agreements are operational agreements between Revenue Canada and provinces and territories for the transfer of information on recipients of NCB supplements, and administrative agreements to deliver certain tax benefit programs. NCB agreements on matters related to goals, accountability and governance are the public announcements and brochures that all participants approved.
• For EAPD, each province has agreed to deliver its own annual report. HRDC is responsible for undertaking overall evaluations and assisting with individual evaluations, in collaboration with provinces. Federal and provincial officials are already meeting to establish the parameters for this work. There is no commitment yet to a periodic accountability report for the overall program like the joint annual report on the NCB.