This Web page has been archived on the Web.

1997 October Report of the Auditor General of Canada

Main Points

17.2 Certain steps remain to be taken to complete this transition. The Department needs to strike a suitable balance among the effectiveness, cost and quality of its services; at present, it does not analyze costs along with key performance indicators. It also needs to link performance to the consequences of having met or not met expectations.

17.3 The support of the employees working in the Human Resource Centres of Canada is essential if the Department is to successfully implement the Framework. The credibility of data on certain programs needs to be resolved within a reasonable timeframe. Employees need to have confidence in the reliability of the information, or the entire initiative may be jeopardized.

17.4 The information that Parliament receives on actual results compared with stated objectives is incomplete. We believe that, given the importance and the special nature of the Employment Insurance Account, it would be beneficial for Parliament if information were all in one place. There is no distinct report containing financial information on performance. Such a report could include, among other things, performance information, results achieved and actuarial analyses, which at present are not tabled in Parliament. Tabling actuarial analyses would improve the transparency and the availability of the data necessary for establishing an adequate reserve in the Account.

17.5 Our audit of two activities, management of accounts receivable and protection of program integrity, confirmed that progress has been made in results-based management. The Department has made major efforts since 1994 to integrate and harmonize the management of accounts receivable for all its programs. However, the quality of the portfolio of accounts receivable has deteriorated over the last five years. The performance of accounts receivable could be better measured, targeted and analyzed in order to assess the cost effectiveness of the Department's collection activities.

17.6 In activities for the protection of program integrity, results-based management is most advanced in the Employment Insurance Program. Since 1992, the Investigation and Control Directorate has been using information on results to negotiate resource levels and to set the national savings expectation. However, the other activities for the protection of public funds could be better measured and targeted in order to manage for results.

Introduction

The Department is going through a transition

17.8 Human Resources Development Canada (HRDC) provides services in various areas such as employment insurance, income security and student loans. In 1996-97 the Department made disbursements totalling $56 billion, including payments for the Employment Insurance Account and the Canada Pension Plan. Some seven million Canadians received benefits or other forms of assistance.

17.9 The Department has begun a transition toward results-based management, whose underlying philosophy involves a change from the traditional emphasis on process to one that focusses on results. In practical terms, this means that managers will be more and more responsible because of increased authority and more flexible rules. This is expected to produce more effective accountability with an emphasis on results achieved.

17.10 To complete this transition, HRDC is working on a number of fronts simultaneously: the Service Delivery Network, the Income Security Programs Redesign Project, the Employment Insurance Act , the review of the Canada Labour Code , agreements on alternative service delivery, and Labour Market Development Agreements with the provinces.

17.11 In October 1996 the Department published its Results-Based Accountability Framework , setting out the main elements of its new management philosophy. Exhibit 17.1 provides a schematic summary of one of the major elements of the Framework - results-based management. The Department has also published a document entitled Redefining Our Roles and Responsibilities in Support of Departmental Priorities, which defines in general terms the roles and responsibilities of the Department's Corporate Service. The document Q uality Services Journey deals with the Department's approach to the government's Quality Service initiative. All of these documents are interrelated.

Focus of the audit

17.13 We also examined the information the Department provides to Parliament, including information on results. For this, we selected the Department's two largest programs: the Canada Pension Plan and the Employment Insurance Account. Further details on the audit objectives, scope and criteria are presented at the end of the chapter, in the section entitled About the Audit .

Observations and Recommendations

Progress toward Results-Based Management

Employee support and senior management participation are essential

17.15 A precondition for the success of an initiative aimed at changing a management approach is that it be clearly formulated and accurately reflect the Department's mission and operating principles. We found that the Framework does both. In examining the strategic planning documents (HRDC Business Plan Summary, Estimates, Quality Service Strategy), we noted that they are interrelated and are consistent in terms of their objectives and the methods to achieve them.

17.16 Another requirement for successfully implementing a new management framework is employee support. The Framework stresses the importance of consulting with employees, and the Department is making efforts to do so. It sent a questionnaire to employees in certain Human Resource Centres of Canada and held a series of discussion groups in five regions. These consultations revealed that employees were reticent about some aspects of the initiative. Given the importance of securing their participation, the Department is continuing discussions to address employees' concerns.

17.17 The active participation of senior management is also a decisive factor in the success of results-based management. We noted that HRDC established a steering committee on quality of service, made up of assistant deputy ministers, which has met regularly since September 1995 to discuss progress in implementing the service quality and results-based management initiatives. It has made various decisions aimed at ensuring that the pace of progress is satisfactory. The Department's Management Board has also monitored progress and discussed the implementation of the new management approach and the related performance reports that have been issued.

17.18 The Department has increased its delegation of authority and has shown more flexibility at the local level. It has begun implementing the Service Delivery Network, aimed at making it easier for clients to access various HRDC services while achieving savings in the use of resources. One of the objectives of this initiative is to increase the authority, empowerment and accountability of line staff. Income Security Programs is also introducing a new client service delivery network, which will broaden the responsibilities of employees at the local level.

17.19 The Department publishes reports on its efforts to reduce obstacles, that is, practices that burden and slow down management unnecessarily and deprive employees of the needed flexibility to resolve problems quickly and effectively. The Department points to a number of administrative changes that simplify the approval process and increase the authority delegated to the local level.

Roles and responsibilities of partners are generally well defined

17.21 In our opinion, the roles and responsibilities of the various federal partners who administer the Employment Insurance program (see Exhibit 17.2 ) are, overall, well defined. There is a Memorandum of Understanding between HRDC and Revenue Canada setting out their respective roles and responsibilities, which was revised in June 1994. While the role of the Canada Employment Insurance Commission is clear, it is somewhat limited and should not be compared with the traditional role of a board of directors.

17.22 The CPP is also administered and operated by various federal partners (see Exhibit 17.3 ), making it more difficult to plan and manage the program and to define the respective responsibilities of the various departments. Memorandums of understanding have been negotiated and signed by the various parties involved, clarifying the roles and responsibilities of each. We nevertheless have some concerns about some aspects of the Chief Actuary's role and responsibilities.

17.23 In accordance with the CPP Act, the Chief Actuary of the Office of the Superintendent of Financial Institutions prepares actuarial reports on the projected status of the CPP Account for the next 75 years. These reports are transmitted to the Minister of Finance who tables them before Parliament. In connection with our audit of CPP Disability benefits last year, we examined the assumptions made for financial forecasts by the Chief Actuary in his Fifteenth Actuarial Report as at December 31, 1993 (the most recent one published) and found them to be adequate and appropriate. However, we believe the Chief Actuary's role and responsibilities in relation to the provinces need to be clarified.

17.24 The federal and provincial governments, as partners in the decision-making process, can call upon the Chief Actuary's services to analyze the potential impact of any proposed changes to the Plan. We expected that his analyses would be available to all partners on a timely basis.

17.25 The Chief Actuary informs the Department of Finance of any requests he receives from the provinces, without disclosing their content. He has assured us that the results of his analyses go directly to the requesting province and that all results remain confidential. The Chief Actuary does not report to the provinces similar requests by the federal government. We believe that all analyses produced should be made available to all the partners. It is important that the Chief Actuary's role and responsibilities be clarified to ensure that all partners have access to the same information. This would add transparency to the decision-making process.

A system for measuring results has been put in place

17.27 By concentrating on a limited number of performance indicators, the Department is better able to ensure that managers and employees focus their attention and efforts on priority aspects of performance. HRDC produces regular performance reports, which are the subject of discussion and follow-up by departmental senior management. It has also begun setting up a Corporate Management System (CMS), an automated system to integrate and streamline the processing of information. This system will make it easier to produce the performance reports that are essential to results-based management.

17.28 In a context of increased co-operation among various levels of government, HRDC has not limited this new approach to its own activities but has extended it to joint activities with its partners. For example, the Department has signed Labour Market Development Agreements with a number of provinces. These agreements stem from the federal proposal to establish a new partnership with provincial and territorial governments in connection with labour market activities, based on the new Employment Insurance Act. These new arrangements aim to help put unemployed Canadians back to work, to encourage innovation and adoption of tested practices, and to strengthen federal-provincial-territorial partnerships.

17.29 The agreements generally include the elements essential to good accountability between partners. More specifically, they specify indicators for measuring results as well as targets for the results that are expected. The Department will use these same indicators to fulfil its obligations for monitoring and assessment. The indicators are: priority access for Employment Insurance claimants; the number of Employment Insurance clients who have returned to work; and savings in the Employment Insurance Account achieved as a result of reduced dependency on Employment Insurance benefits.

17.30 However, it is important that this performance information not be simply filed away but be put to good use by program managers. Hence, the Department's Investigation and Control Directorate, whose objective is to prevent, detect and deter fraud and abuse in the Employment Insurance Account, has decided to use the total savings generated by its activities as a key performance indicator. Adoption of this indicator has had a marked impact on the nature and scope of investigation and control activities, by encouraging managers to seek an appropriate balance between prevention and detection. The indicator has become an element used to determine the level of resources to be allocated to the program and to facilitate accountability. More details on this subject can be found in Chapter 11, Moving toward Managing for Results.

Certain Steps Remain to Achieve Stated Objectives

Balance in indicators is important

17.32 An important factor in choosing key indicators is the reasonable expectations of clients since, as the Framework notes, they are "best equipped to comment on what constitutes good service". Thus, Employment Insurance based its choice of some indicators on client surveys that showed which aspects of service were most important to clients. Income Security Programs also plans to eventually include client satisfaction in its indicators.

17.33 The choice of the key indicators is of paramount importance, as they will serve as the guideposts for determining the quality of program and service management. Choosing performance indicators entails some risk, since the focus on highly specific indicators can affect other aspects of performance not considered as high a priority. Exhibit 17.4 shows the key performance measures chosen by the Department, based primarily on performance issues. For example, issues of telephone centre performance are much more pronounced in Income Security Programs than in Employment Insurance. Therefore, the percentage of telephone calls answered is considered a key indicator for Income Security, but only a secondary indicator for Employment Insurance.

17.34 The Framework specifies, "Results measurement should strike an appropriate balance between efficiency, effectiveness, cost and quality of service." The Department's key indicators do not take cost and efficiency into account; only the secondary indicators do so. However, unlike key indicators, secondary indicators are not targeted in all cases and are not subject to reporting to Parliament.

17.35 Costs and results are closely linked. For example, it is important to know how improving service quality impacts on costs. Although the Department has established a number of measures to improve service, it does not always have adequate information to assess the link between the effectiveness and the cost of services. Since its operating costs are expected to be reduced in the coming years (see Exhibit 17.5 ), the Department will need to consider costs in analyzing the results it achieves.

17.36 The Department has indicated that it intends to review and refine its performance indicators on an ongoing basis. Indeed, as program performance issues evolve, so do the performance indicators. A key indicator may become a secondary indicator and vice versa. The Department is currently revisiting its key indicators for 1998-99. While some adjustments to indicators may be necessary to reflect what has been learned, it is important that they be used for a long enough period to identify trends in order to determine whether performance is improving or deteriorating.

17.37 Furthermore, as the Framework indicates, an essential element of results-based management is the evaluation of overall program effectiveness. Instead, however, the current indicators stress day-to-day management and program outputs. Program results must also deal with outcomes, that is, the effects and impacts of those outputs. This information is generally produced in connection with periodic program evaluations by the Department.

17.38 For example, the Evaluation and Data Development Branch monitors the impacts of the new Employment Insurance legislation. Its activities will include conducting a survey on this program's impact on individuals, performing a qualitative analysis at the community level, and asking specialists to perform impact analyses. This information needs to be examined together with the key indicators in order to report on performance and obtain an overall picture of the results achieved.

Further progress is needed to ensure credibility of data

17.40 Parliament is also a major user of these data. Information on results allows the Minister to answer to Parliament for the effectiveness of HRDC programs. However, as we note in paragraphs 17.47 to 17.66, Parliament currently does not receive all the information it requires. It is important that information provided to Parliament be credible, so that Parliament can use it in its decisions to authorize funding.

17.41 An audit by the Department's Internal Audit Bureau revealed certain problems among employees with the credibility of some program data. In the case of the Labour Program, the negative perception could be because the indicators are relatively new. For Income Security Programs, preliminary data are gathered manually, which increases the risk of reduced overall reliability. The Department contends that with the ISP Program Redesign currently under way, its data collection programs will eventually be automated. For the Human Resources Investment Fund, a study by the Evaluation and Data Development Branch is seeking to resolve some of the methodological problems associated with measuring the additional savings generated by the program.

17.42 For a number of years now, the Employment Insurance Program has had a system for the capture, analysis and reporting of performance data. The internal audit report notes that, while improvements are possible, all the features of an effective and useful system are present. Employees are therefore not as concerned about the credibility of data from the Employment Insurance systems.

17.43 Nonetheless, in various consultations held by the Department, employees have expressed skepticism about the credibility of data. This situation constitutes a major challenge for the Department, since the support of employees, especially in the rank and file and at the local level, is a crucial element that will determine the Framework's success. The internal audit also noted that the monitoring and analysis of performance information has largely been delegated to regional offices and Human Resource Centres of Canada, which may not have the resources and skills needed for the task.

17.44 It will be difficult for employees to accept the need for management on the basis of performance targets if they do not feel that the targets are based on reliable data. The internal audit report also notes the need to link performance to the consequences of having met or not met expectations as a step that the Department still has to take. A connection also needs to be made between resource allocation and expected results. It is clear that the Department needs to obtain more solid support from employees if it is to take these steps successfully.

17.45 Since acceptance at the local level is vital, the Department needs to resolve, within a reasonable timeframe, the ambiguity surrounding data credibility. Certain quality controls could strengthen confidence in the data and provide a reasonable level of assurance that they are reliable. In its report on this subject, the Internal Audit Bureau stressed the need to test data to ensure that they are relevant and, above all, reliable.

17.46 To build on the progress made and to maintain the pace of its implementation of results-based management, the Department should:

• ensure that the choice of key performance indicators takes costs into account;
• ensure the quality of performance data to increase their credibility among employees and managers who use the data; and
• ensure that the selected key indicators remain in place long enough for performance to be assessed over time.

We are developing a mechanism to maintain data integrity that will allow service delivery partners to transmit data directly to HRDC's systems. Some of the data integrity problems are due to systems problems and data capture methods, which are in the process of being fixed.

Information to Parliament Is Incomplete

17.48 We examined the information on results communicated to Parliament in HRDC's Estimates Part III, its Business Plan Summary and the Annual Report of the Canada Pension Plan .

17.49 The 1997-98 Estimates and the 1996-97 Business Plan Summary do not yet present all the results achieved in relation to stated objectives for the key performance indicators. It is still too soon to provide this type of information. We encourage the Department to communicate regularly to its employees, partners, clients and Parliament the results achieved as soon as the information is available and credible.

The Annual Report of the Canada Pension Plan is moving in the right direction

17.51 The annual report of the CPP for 1995-96 provides an excellent overview of the responsibilities of the various departments involved. It also presents a good summary of the services offered, the clientele served and the benefits paid. This constitutes a major improvement over previous years.

17.52 In order to make the report complete, we suggest that it include the following elements:

• a discussion of the mandate, objectives and strategies adopted by management, and an explanation of how the activities and services provided support the objectives and the mandate of the CPP;
• a comparison of actual performance with targets or objectives, to complete the information on achievement of results, when systems allow;
• adequate future-oriented information, for example, the impact of the aging of the population;
• information on the interrelationship between economic and social issues; and
• complete and relevant information on the major activities related to disability benefits.

Department's response: We agree with the recommendation and will strive to incorporate any relevant strategies, information on future direction of the Canada Pension Plan (CPP), and information on the performance of the CPP against the expected results, as it becomes available. It should also be noted that information on the future direction and projected financial status of the CPP is currently already available in other reporting vehicles, namely Part III of the Estimates and the CPP Actuarial Report.

There is no distinct report on the Employment Insurance Account

17.55 Neither the Department of Human Resources Development Act nor the Employment Insurance Act require that a separate report be prepared and submitted to Parliament, apart from a report on the impact of changes brought about by the new legislation, which must be submitted no later than December 31 for the next four years.

17.56 We are concerned that financial information on performance is diffused and that it is difficult to know the results achieved in the operations of the EIA compared with the budget and the Business Plan Summary. Moreover, there is no performance information on the collection of premiums, which is the responsibility of Revenue Canada.

17.57 We believe that the importance and the special nature of the Employment Insurance Account, which is wholly funded by employers and workers, warrants a distinct report that would bring information together in one place, including information on performance, results and the audited financial statements. This report could be linked to the new Performance Report that the Department is to table to Parliament in the fall of each year.

17.58 The Department should produce, on a timely basis, a report providing complete, relevant and timely information on the major activities related to the Employment Insurance Account in order to report on the results achieved and on the management of resources allocated.

Department's response: The requirement to produce an annual report on the Employment Insurance Account (EIA) was eliminated by Parliament through legislation. Information on the major activities related to the EIA reporting on the results achieved and on the management of resources allocated can be found in the Departmental Performance Report and the Report on Plans and Priorities. However, HRDC will explore the possibility of producing a complete report on the Account.

Actuarial analyses of the Employment Insurance Account are not tabled in Parliament

17.60 In our 1994 Study of Key Federal Social Programs, we noted that a reasonable reserve in the Unemployment Insurance Account would be desirable. It would alleviate the need for borrowing, reduce interest costs thereof, and provide some relative stability in the premium rate as well as protection against unexpected cost increases. At that time, we raised questions we thought deserved consideration, such as what constitutes a reasonable reserve and how much time is required to build it.

17.61 We note that the Commission has not yet come to a decision on what constitutes an adequate reserve and how much time is required to build it to ensure that there will be enough revenue over a business cycle and to maintain relatively stable rate levels throughout the business cycle.

17.62 On the basis of the report of the Chief Actuary of HRDC, the Commission set the 1997 premium rate for workers at $2.90 for each $100 of insurable earnings (the 1996 rate was $2.95). A reserve of some $12 billion at the end of 1997 is forecast, representing approximately 1.2 times the forecast average annual amount of regular benefits.

17.63 In this internal report, the Chief Actuary states:

...the amount of reserves needed to ensure `relative premium rate stability' is not a fixed amount. Reserves will be at a low point towards the end of recessions, and would be rebuilt during an economic recovery. There does not seem to be any need for exceeding an absolute level of about $15 billion. However, stabilization also involves dealing with the risks of setting up the reserve too quickly or at too high a level - perhaps starting when they would reach or exceed $10 billion.

17.65 Since the actuarial analyses are not public information, the setting of premium rates lacks transparency and may be susceptible to influence by factors unrelated to the objectives set out in the Act, namely to ensure that there will be enough revenue and to maintain relatively stable rate levels. Indeed, setting premium rates could be influenced by the fact that any annual surplus or deficit in the Account has an impact on the government's deficit. For example, if the Account had an annual surplus of $5 billion, the federal deficit for that year would be reduced by the same amount.

17.66 The Department should ensure that the actuarial analyses necessary for setting the premium rate of the Employment Insurance Account, including the level of a reasonable reserve and the time required to build it, are tabled in Parliament to make the setting of premium rates more transparent.

Department's response: All the relevant information with respect to the EI Account is already in the public domain such as in the government's annual economic update, the Budget, Main Estimates and the background material regularly provided with the HRDC press release announcing the annual premium rate decision.

Case 1 - Management of Accounts Receivable

17.68 To better assess the progress made in the transition toward results-based management, we examined the management of accounts receivable, which is shared between program staff and Financial and Administrative Services staff. Where management of an activity is shared, accountability for results is even more necessary to focus in the same strategic directions the efforts of those involved.

17.69 The active and inactive accounts receivable in the Employment Insurance Program and the Income Security Programs stem essentially from overpayments that have been detected and recorded as amounts to be recovered. Active accounts receivable are collected by program staff from benefit payments to beneficiaries before they are issued. Inactive accounts receivable are those owed by beneficiaries who no longer receive benefit payments; they are recovered by specialized collection agents of the 12 collection centres administered by Financial and Administrative Services. The number of employees working in the collection centres increased from 251 to 310 between 1992-93 and 1996-97.

17.70 The Department also collects from students who have failed to meet their loan repayment obligations. Responsibility for collecting newly defaulted student loans was transferred to financial institutions in August 1995; hence, the Department's current portfolio of student loans will decrease in the future. At present, the Department sends the defaulted student loans portfolio to collection agencies.

Major efforts are being made to improve performance

• develop, communicate and implement a single departmental policy on accounts receivable and collection;
• implement a single Departmental Accounts Receivable System; and
• implement various collection pilot projects to facilitate a decision on the collection strategy and the optimal organizational structure to be adopted.

17.73 Implementing the strategy requires co-ordination and co-operation between program staff and Financial and Administrative Services staff. Despite progress made for certain elements of the strategy, we noted delays in implementing a number of initiatives, and several pilot projects have not yet been finalized. For example, the Department only very recently began developing a single policy on accounts receivable. Furthermore, not all accounts receivable have been transferred yet to the new departmental system.

17.74 We believe an action plan setting out roles and responsibilities, priorities, resource requirements and timetables for bringing the strategy to completion would be beneficial.

17.75 The Department should complete the implementation of the strategy to integrate and harmonize the management of accounts receivable in order to improve performance results.

Department's response: We concur with this recommendation and will continue to work toward full implementation of the strategy as quickly as possible so that performance results for all HRDC receivables will be optimized and reflect the benefits of an integrated and harmonized approach to debt management and recovery.

Performance could be better measured and targeted

17.77 It is important that performance information cover key elements so that managers can make the most appropriate decisions, particularly in selecting collection methods. The Department uses various collection methods to recover amounts owing as quickly as possible. A comparison by collection method of the amounts recovered with the collection costs would enable the Department to select the most cost-effective method. However, management systems currently cannot provide information about the cost of each collection method. It is therefore difficult for managers to determine the appropriate mix of collection methods to improve performance.

17.78 Exhibit 17.8 shows that performance targets have not been set for a significant portion of the portfolio of accounts receivable. For example, active accounts receivable in the Employment Insurance Program, which represent a substantial portion of that program's total as of 31 March 1997, have no performance targets. With respect to the portion of the student loans portfolio collected by private collection agencies, the Department's view is that performance is a product of sound competition among the agencies, given that new defaulted student loans are allocated to them based on performance achieved according to a set of criteria.

17.79 We believe that setting performance targets for the overall portfolio of accounts receivable would enable the Department to better focus on results and to assess performance. Without performance targets, it is difficult for senior management to determine whether or not strategic directions have been achieved.

17.80 The Department should ensure that performance indicators and targets cover all key aspects of the management of the portfolio of accounts receivable. It should also evaluate the cost effectiveness of various collection methods.

Department's response: We agree with the recommendation that performance indicators for all of the portfolio of accounts receivable be utilized and that collection methods be evaluated for cost effectiveness.

As the Department's Accounts Receivable Strategy evolves and the initiative to harmonize various HRDC program recovery policies is realized, it will enhance our ability to set realistic target levels for all aspects of the management of the portfolio.

A more thorough analysis of performance is needed

17.82 The analysis in Exhibit 17.9 shows that the annual combined rate of recovery of accounts receivable for the Student Loans and Employment Insurance programs has consistently declined over the last five years, from 24 to 17 cents per dollar available for collection, a decrease of close to 30 percent. This decrease is due, in part, to the decline over the same period in the rate of recovery of accounts receivable for the Employment Insurance Program, from 46 to 37 cents per dollar available for collection.

17.83 Our analyses show that the quality of the portfolio of accounts receivable has deteriorated over the last five years, in part because of the increase in the age of accounts receivable. In other words, the older the accounts, the more difficult it is to recover the amounts due. As Exhibit 17.10 shows, accounts receivable in the Employment Insurance Program are becoming older; and the amounts in student loans older than four years expressed as a percentage of the outstanding total increased from 39 percent to 50 percent during the last five years (see Exhibit 17.11 ).

17.84 With respect to the Employment Insurance Program, the lower rate of recovery and older age of the accounts receivable are due to a decrease over the last five years in amounts recovered before benefit payments are issued to beneficiaries who owe outstanding amounts. Active accounts receivable are managed by program staff who are not specialized collection agents. Since this program has only one performance indicator and no performance target for managing active accounts receivable (see Exhibit 17.8 ), it is difficult to determine whether they are managed properly and whether performance results are appropriate.

17.85 We believe that the use of performance indicators and targets for the overall portfolio of accounts receivable would enable the Department to better assess whether the overall performance of the portfolio has improved or deteriorated. The Department has informed us that the Accounts Receivable System now being developed will allow it to compile more performance information and will therefore facilitate analysis of changes in the cost effectiveness of collection activities.

17.86 The Department should analyze thoroughly the causes of changes in the cost effectiveness of collection activities and in the quality of the portfolio of accounts receivable, in order to identify and implement in a timely manner any needed corrective action.

Department's response: We concur with this recommendation, and the Accounts Receivable Strategy initiatives, once fully implemented, will provide a means by which the effectiveness of recovery operations generally, and recovery tools and techniques specifically, may be more accurately measured and evaluated, both in terms of efficiency and cost effectiveness.

Certain collection practices can be improved

17.88 We noted some areas where collection practices need to be strengthened to improve performance and make management more results-based:

• Collection priorities are not based on debtor risks and profiles. It is essential that collection efforts be properly prioritized in order to recover the amounts owing as quickly as possible.
• Amounts to be recovered from benefit payments destined to debtors are predetermined by the Department. It does not seek to immediately recover the full amount or a large portion of the debt, even if the financial situation of the debtor so permits.
• The debtor's assets are not considered in assessing the capacity to pay as required. The Department's system is not designed to record information on assets.
• Errors and major delays were observed in the recording and recovery of overpayments, as well as errors in accounts receivable balances recorded in the Departmental Accounts Receivable System for the Income Security Programs.

Department's response: We concur with this recommendation. The HRDC Accounts Receivable Strategy was developed with a view to ensuring that all HRDC debtors are treated in a fair, consistent and equitable manner, while at the same time maximizing recoveries through utilization of the most cost-efficient recovery tools and techniques, targeted to specific categories of accounts. With the full implementation, over the next three years, of all elements of the Accounts Receivable Strategy, the Department will have much improved collection tools and an enhanced capacity to measure the effectiveness of specific recovery methods and practices.

Case 2 - Protection of Program Integrity

Management of the integrity of the Employment Insurance Program is results-based

17.92 The Investigation and Control Directorate of the Employment Insurance Program has, since 1992, managed activities for the protection of public funds in accordance with expected results. It operates at the national, regional and local levels. The differentiation of roles and responsibilities among these three levels is well defined. For example, directors of Human Resource Centres of Canada (HRCCs) are responsible for results; that is, they must make every effort to ensure that the savings expectation expressed in terms of total savings is met.

17.93 Information on results, measured in terms of total savings, is used to negotiate the level of resources required and to set the national savings expectation. Total savings comprise direct and indirect savings. Direct savings measure the value of overpayments detected and penalties imposed. Indirect savings represent the estimated value of improper payments not made as a result of investigations.

17.94 Exhibit 17.13 shows that total savings have grown since 1992, reaching a peak in 1995-96 and then declining in 1996-97. Although the Directorate failed to reach its savings expectation in the last two years, performance measured in terms of total savings is up over the last five years. The increase in total savings is due mainly to indirect savings, which nearly quadrupled from $68 million to $249 million during that period.

17.95 The adoption of total savings as a key indicator of performance has had a marked impact on the nature and scope of investigation and control activities, by encouraging managers to emphasize not only detection (measured by direct savings) but also prevention and deterrence (measured by indirect savings). Indirect savings as a proportion of total savings more than doubled over the last five years, climbing from 17 percent to 44 percent.

17.96 Efforts in prevention and deterrence have helped to improve the return on investment over the last five years (see Exhibit 17.14 ). The Directorate told us that a recent decrease in total savings and return on investment was due in part to the introduction of the new Employment Insurance Act in June 1996 and to the overall reduction in the size of the program in terms of benefits paid. The new Employment Insurance Act has required a major effort to train investigators, which has contributed to reducing the number of investigations each one conducts.

17.97 The use of performance indicators and expectations enables results-based management of the protection of the integrity of the Employment Insurance Program. However, this is not the case for Income Security Programs and The Atlantic Groundfish Strategy, which lack performance indicators and expectations.

17.98 The Department should ensure that the results of all activities for the protection of public funds against fraud and abuse in Income Security Programs and The Atlantic Groundfish Strategy are properly measured and that expectations are set in order to manage for results.

Department's response: We concur with this recommendation.

Performance could be analyzed more thoroughly

17.100 The national savings expectation is distributed among the regions. However, some regions do not allocate savings expectations among Human Resource Centres of Canada (HRCCs), or do so informally. Uniform methods of allocating savings expectations would make it easier for Centre directors to accept the expectation and would facilitate comparison of performance among Centres in the various regions.

17.101 Comparing results obtained with costs incurred makes it possible to assess the cost effectiveness of activities to protect public funds. The Department calculates return on investment at the national level and for each region and HRCC, in order to compare results with costs. Since no expectation is set for this performance indicator, however, it is difficult to determine whether or not the activities for protecting public funds are cost-effective.

17.102 The Investigation and Control Directorate uses a variety of investigative methods to reach the national savings expectation. Although it measures the total savings generated by each investigative method, it does not assess the cost effectiveness of each method because its information on costs is not detailed enough. It is hence difficult for managers to determine an appropriate mix of investigative methods to improve the cost effectiveness of activities.

17.103 The Directorate regularly prepares and communicates information on performance indicators. Performance varies significantly among regions and among HRCCs. A more thorough analysis of the causes of performance differences would be beneficial; by examining actual performance compared with expected results, the Directorate could identify any needed corrective actions. For example, the Business Plans of the Centres could clearly indicate the causes of discrepancies between actual and expected results and the appropriate corrective actions to be taken. We note that the Directorate recently set up, at the national level, a special unit whose mandate includes analyzing more thoroughly the causes of performance variances among regions.

17.104 The Investigation and Control Directorate of the Employment Insurance Program should conduct a thorough analysis of the causes of discrepancies between actual and expected results and of changes in cost effectiveness in order to report properly on results achieved and to identify the need for corrective action.

Department's response: We agree with this recommendation, and this process is already under way. We have a three-pronged approach: 1) an evaluation study by HRDC's Strategic Policy Branch; 2) a survey of front-line staff; 3) more in-depth analysis of administrative and performance data. All of this will be discussed by senior managers, at the director-general level, at Management Board and with the Commissioners.

Conclusion

17.106 Nevertheless, the Department still needs to take certain key steps that are critical to the success of an initiative of this magnitude. It needs to ensure that it has chosen its key performance indicators carefully and that they are relevant over time. It also needs to set expectations for all its programs and to ensure the credibility of the data that employees consider most important.

17.107 Our examination of the management of accounts receivable and the protection of program integrity confirms our conclusion that major progress has been made in results-based management but that certain steps are still needed in the area of performance measurement and analysis.

17.108 In addition, at the present time, very little information on results is communicated to Parliament. The information that is provided on the Employment Insurance Account can be greatly improved. With the implementation of the Results-Based Accountability Framework , the Department has positioned itself to start providing Parliament with better information on results.

17.109 Measurement of performance results, and follow-up, will require a firm commitment by senior management and the co-operation of all members of the organization, at the local, regional and national levels. The recent experience acquired by the Investigation and Control Directorate in the Employment Insurance Program could provide senior management with guidance and encouragement. Decisions and adjustments will be necessary if results measurement is to strike an appropriate balance among the effectiveness, cost and quality of services.

About the Audit

Objectives

• examine progress achieved toward results-based management;
• determine whether the Department is appropriately fulfilling its responsibilities and reporting on performance achieved on the basis of performance expectations for two selected activities - management of accounts receivable and protection of program integrity; and
• examine information to Parliament on the Department's two largest programs (the Canada Pension Plan and the Employment Insurance Account), including information on results.

Scope

We examined the information on results communicated to Parliament. Our audit efforts here were focussed on the Department's two largest programs, the Canada Pension Plan and the Employment Insurance Account.

We interviewed officials at headquarters and some partners of the Department such as the Chief Actuary of the Office of the Superintendent of Financial Institutions. We visited three collection centres, two Human Resource Centres of Canada and two regional offices, located in Ontario and Quebec. We analyzed planning documents, minutes, performance reports and other related documents. As appropriate, we used the work and the related reports of the Internal Audit Bureau.

Criteria

• A management framework for implementing results-based management should be established to ensure, among other things, that initiatives and related activities are closely linked to the Department's mission and operating principles.
• Roles and responsibilities should be clearly defined and accepted in order to promote accountability and achievement of performance and results.
• Reasonable expectations of performance and results should be clearly set out and accepted in order to fulfil responsibilities and achieve objectives.
• Sound management practices and appropriate management systems should be used in order to suitably fulfil responsibilities.
• Information on performance and results should be credible, analyzed and communicated in a timely manner to senior management and Parliament for decision making and accountability.
  

Quantitative information

Audit Team

For information, please contact Louis Lalonde, the responsible auditor.