2007 May Report of the Auditor General of Canada
Chapter 4—Canadian Agricultural Income Stabilization—Agriculture and Agri-Food Canada
What we examined
The Canadian Agricultural Income Stabilization (CAIS) program is a federal-provincial-territorial program whose objective is to protect agricultural producers from drops in income due to circumstances beyond their control. The program is administered by Agriculture and Agri-Food Canada for all of Canada except Alberta, Ontario, Quebec, and Prince Edward Island, where the provincial governments administer the program. We did not audit the provinces' delivery of the CAIS program.
We conducted this audit at the request of the House of Commons Standing Committee on Agriculture and Agri-Food. We examined how the Department processes applications for income support, ensures that all parties are respecting the various monitoring provisions set out in the federal-provincial-territorial agreements, and measures and reports its performance to Parliament.
Why it's important
The federal government has provided income support programs to farmers for decades. Agriculture and Agri-Food Canada receives about 55,000 applications to CAIS each year. In the 2005–06 fiscal year, the Department spent about $1.1 billion Canada-wide, making the CAIS program the Department's principal ongoing program of farm income support.
What we found
- The CAIS program is very complex, and the Department does not make it transparent enough to producers; some have complained that they do not understand how the Department calculates their benefits. For example, when the Department changes financial or other information that producers submit, it does not explain the impact on the benefit amount. Consequently, producers do not receive, in an easy to understand manner, all the information they need to decide whether to ask for a payment adjustment or appeal an unfavourable decision. Further, there are long delays before producers are told whether they will receive a benefit and in what amount.
- The Department focuses on preventing and detecting overpayments made to producers, and the onus is on producers to question incorrect figures in the Calculation of Program Benefits statement sent by the Department. Improving transparency would make program delivery more consistent with the program's objective, as set out in the Farm Income Protection Act—to protect the income of producers.
- Payment errors have been high, although they are beginning to decline. Staff who calculate payments work to standards that emphasize the number of applications processed and adherence to procedures more than accuracy of payments. While the Department collects data on the amounts and nature of the errors, it does not systematically use that information to improve its management of the program.
- Some employees who processed applications also worked as private consultants to help producers prepare their applications. We are concerned that such conflicts of interest could provide an unfair financial advantage to some applicants.
- Some of these problems are not new; our past audits of previous farm income support programs have found similar problems.
The Department has responded. In its response to each recommendation throughout the chapter, Agriculture and Agri-Food Canada has indicated the actions it has taken, will take, or will consider taking.
Agriculture and Agri-Food Canada's administration of farm income support programs
4.1 Agriculture and Agri-Food Canada has administered various farm income support programs since the late 1930s. The Canadian Agricultural Income Stabilization (CAIS) program is the latest in a number of programs that Agriculture and Agri-Food Canada developed to provide income support to agricultural producers.
4.2 In 1991, the Net Income Stabilization Account (NISA) program was established, under the Farm Income Protection Act, to help producers stabilize their income. The program enabled producers to accumulate funds in good years and to use those funds in bad years. As a complement to NISA, the Department introduced the Agriculture Income Disaster Assistance (AIDA) program in 1998 and replaced it with the Canadian Farm Income Program (CFIP) two years later. Both AIDA and CFIP were intended to help producers who experienced sudden and severe declines in farming income for reasons beyond their control. The CAIS program was introduced in December 2003.
Problems in farm income support programs identified in our past audits
4.3 We audited farm income support programs in 1991 and 1994. In 1996, we assessed the Department's progress in implementing our recommendations. Our audits questioned whether the Department had adequately measured and reported the effectiveness of the farm safety net programs and whether it had implemented proper financial management and controls. In particular, we noted that the Department
- needed to strengthen program administration in areas such as payment accuracy verification, error correction, and on-farm audits;
- needed to define program objectives and develop performance indicators; and
- had not issued annual reports and financial statements in a timely manner.
The Canadian Agricultural Income Stabilization program
4.4 The Canadian Agricultural Income Stabilization (CAIS) program aims to help producers protect their operations from small and large drops in income due to circumstances beyond their control. Producers can apply for CAIS support if they have farmed for more than six months and have reported farming income for income tax purposes.
4.5 The Department delivers the CAIS program in all provinces and territories except Alberta, Ontario, Quebec, and Prince Edward Island, where the provincial governments deliver the program. In the 2005–06 fiscal year, the Department spent about $1.1 billion on the CAIS program Canada-wide. The federal government is responsible for 60 percent of the total amount spent on the CAIS program. The provincial and territorial governments are responsible for the remaining 40 percent. The federal government has signed bilateral implementation agreements with the provinces and territories that outline the cost-sharing and other terms and conditions for programs and activities, including the CAIS program.
4.6 Producers receive a payment from the CAIS program when the farm income from their current year is less than their average farm income from previous years.
4.7 The CAIS program is based on margins:
- The production margin is allowable income minus allowable expenses for a given year.
- The reference margin is the average production margin for three of the past five years; the years with the lowest and highest production margins are not included in the average.
4.8 Producers receive payments when their current year's production margin is less than their reference margin. The proportion of the government payment increases as a producer's loss deepens—the greater the loss, the larger the government contribution. The payment amounts that producers receive vary widely. For example, in the 2004 program year, where the Department delivered the CAIS program, it received 57,000 valid applications. Of the producers who submitted these applications
- about 25,000 received no payment,
- about 15,000 received $10,000 or less,
- about 50 received payments of $500,000 or more, and
- 12 received more than $1 million.
4.9 The Department estimates that accountants, and other third parties who charge a fee for the service, prepare 70 to 80 percent of the applications. The Department's records indicate that it received about 55,000 valid applications for the 2005 CAIS program year, compared to 57,000 and 55,000 applications for 2004 and 2003 program years, respectively. Producers can opt out of the CAIS program, but they are then ineligible to rejoin the program for two years.
Focus of the audit
4.10 In 2005, the House of Commons Standing Committee on Agriculture and Agri-Food asked the Auditor General to audit federal spending on the CAIS program.
4.11 The purpose of the audit was to assess how well the Department
- processes CAIS applications,
- meets the monitoring requirements of the federal-provincial-territorial agreements, and
- measures and reports its program performance to Parliament and other interested parties.
4.12 More details about the audit objectives, scope, approach, and criteria are in About the Audit at the end of this chapter.
Observations and Recommendations
Canadian Agricultural Income Stabilization program applications
4.13 The Financial Administration Act and the Treasury Board's policies govern departments' effective delivery and administration of transfer programs, such as the Canadian Agricultural Income Stabilization (CAIS) program.
4.14 We expected the Department to have a system in place to ensure that it processes claims accurately and reasonably quickly. We also expected it to have adopted a reasonable approach to verify the completeness and accuracy of source data, an approach that
- balances efficiency and effectiveness with the costs and risks to applicants and the federal government, and
- is consistent with government requirements.
How the Department calculates benefits is not transparent to producers
4.15 In their applications for income support, producers are required to submit income tax information. They are also required to submit complicated, detailed, and accurate information about farm expenses, revenues, and inventories each year.
4.16 The method of calculating CAIS program benefits is complicated. The Department provides information to producers on its CAIS program website. It also provides group information sessions to producers as well as individual sessions, during which CAIS representatives advise producers on how to complete their applications. Despite these efforts, some producers have complained that
- they do not understand how the Department calculates the benefits,
- the benefits are not predictable, and
- banks will not consider any potential payments from the CAIS program when deciding whether to lend them money.
4.17 Stakeholders told us that the CAIS program payments are neither transparent nor predictable. The application forms do not give producers an idea of the amount they will receive. Producers provide the requested information and must wait until the Department has finished processing the application before they find out whether they will receive a payment. The Department provides an online CAIS benefit calculator on its website but does not guarantee the accuracy of the results.
4.18 We expected the Department to provide producers with information that would help them to submit accurate information in their CAIS applications. To assess the Department's efforts to explain how staff members calculate payments, we reviewed the Department's external guidance materials, publications, notices, and presentations to applicants as well as the Calculation of Program Benefits statement. We also examined the Department's internal documents that are designed to guide staff members on how to communicate with producers.
4.19 Some producers find it challenging to provide all of the complex information that the Department needs to process applications. Processors sometimes change the information in producers' applications. For example, the manual requires processors to
- contact producers, for clarification, when producers use inaccurate codes to identify their commodities and/or inaccurately estimate the quantity of unsold inventory;
- change commodity prices when producers submit prices that do not match the Department's estimated market prices; and
- change the information producers submit to reconcile the current year's opening inventory with the previous year's closing inventory and to make it match the data from Production Insurance (another federal-provincial-territorial agricultural program).
4.20 We reviewed a sample of 2004 program-year applications to determine whether the Department informed producers of the changes that the processors had made to the applications. We also reviewed the Department's guidance materials, which describe procedures for contacting producers.
4.21 We found that, in some cases, the manual that is given to processors, for verifying the accuracy of the information producers submit, instructs staff to contact producers for additional information needed to process applications. However, we found that the Department's guidance materials did not instruct staff on how to let producers know
- how the revised information was used to calculate the benefits,
- what impact the changes would have on the benefits,
- what other changes they made to producers' applications, or
- when these changes occurred.
4.22 Changes not reflected in statements. The Calculation of Program Benefits statement lists, line-by-line, the figures that the Department used to calculate CAIS benefits, and is generally eight to ten pages long. However, it does not explain which figures the Department changed or why.
4.23 By not explaining, in the Calculation of Program Benefits statement, the changes it makes to the figures in producers' applications, the Department is missing an opportunity to build a well-informed and knowledgeable community of applicants.
4.24 Onus on producers to question figures. Once they receive their Calculation of Program Benefits statement, producers have 90 days to ask the Department to adjust the figures it used to process their applications or to request an appeal. Producers can also call the Department's CAIS toll-free line to discuss the figures the Department changed and the reasons for the changes.
4.25 In our opinion, because the statement does not indicate which figures were changed, producers may not notice that changes have been made. The onus is on producers to compare the numerous figures in the final eight- to ten-page statement with the figures they submitted in their original applications.
4.26 If the Department clearly indicated the changes, explained the reasons for those changes, and explained how those changes affected the benefit amounts, the process would be more clear, transparent, and fair for the producers. It would also improve the producers' ability
- to correct figures that the Department had changed incorrectly,
- decide whether to ask the Department to adjust the figures used to calculate the benefit,
- appeal the Department's decision, and
- provide more accurate information on future CAIS applications.
4.27 Transparency needs to improve. During our review of a sample of the 2004 program-year applications, we found one payment made to a producer that increased by more than $90,000 due to a change in processing procedures. This change occurred after almost 20 percent of the applications for the 2004 program year had already been processed. The Department told us that it did not specifically inform producers, whose applications had already been processed of this change, nor did it inform them of the effect it may have had on their program benefits. However, the Department did tell us that it would investigate this situation to determine if proper processes were followed and that it would address any issues it identifies. Nevertheless, we are concerned that these producers may not have known of the changed procedures and, therefore, would not have had appropriate information to decide whether to request an adjustment or launch an appeal.
4.28 We also noted that in its guidance documents for producers the Department does not mention that it will accept re-assessments by the Canada Revenue Agency. If these re-assessments affect a producer's CAIS reference margin, the Department will accept the new information for up to three years after the end of the program year affected by the adjustment. The Department said it would mention this practice in the next update of the CAIS handbook for producers.
4.29 The Department plans to improve the CAIS application forms and tools. By the time this report is tabled, it intends to simplify the application form for the 2006 CAIS program year, by reducing the amount of information required, and to provide a web-based electronic CAIS application tool and online access to CAIS account information for individual producers.
4.30 As we have already noted, the CAIS program is complicated. The producers are responsible for providing accurate information in their CAIS applications; the onus is also on the producers to question figures and identify incorrect payments. Expanding the information in the final Calculation of Program Benefits statements would improve producers' ability to detect incorrect figures and decide whether to ask the Department to adjust the calculations or to appeal the Department's decision. Expanding the information would also be more consistent with the Department's objective of client service excellence and with the primary objective of the CAIS program, as set out in the Farm Income Protection Act, which is to protect producers' incomes.
4.31 Recommendation. Agriculture and Agri-Food Canada should improve the transparency of the final statement that it sends to producers about their farm income support program benefits. The Department should clearly and fully explain how it calculates benefits, including what information it changes and why it makes the changes.
Agriculture and Agri-Food Canada's response. The Department agrees with the recommendation and has already taken steps to improve the transparency of the information it provides to producers. More specifically, it intends to add explanations of benefit calculations and adjustments and what to do if producers have questions or disagree with the Department's decisions to the Calculation of Program Benefits statements for the 2007 year. Implementing this recommendation will also add to improvements that the Department has already made to its communications with producers as well as to changes to make the program simpler and more accessible, such as making program parameter changes, harmonizing its forms with the Canada Revenue Agency, reducing its supplementary form, and offering on-line My Account and calculator features.
4.32 Controls to ensure accurate payments. We expected the Department to have implemented controls to ensure the accuracy of CAIS payments, and we reviewed the controls.
4.33 The Farm Income Programs Directorate is responsible for processing CAIS applications. It is also initially responsible for approving payments. The Financial Operations Division independently reviews the Directorate's recommended payments. If the payments are accurate, the Division approves them. This review is the last step taken to verify that payments are accurate, before the government sends cheques and the Calculation of Program Benefits statements to producers.
4.34 The Department uses a risk-based verification strategy to process CAIS applications. Based on the risk assessment results, the Farm Income Programs Directorate processes about two thirds of the applications manually. It processes the rest electronically, with no human intervention after the initial data assessment stage. After payments have been made, the Financial Operations Division also reviews samples of payments to estimate absolute errors.
4.35 Payment errors declining. The Department's goal is to ensure that the total estimated absolute errors to producers amounts to three percent. For the 2003 and 2004 program years, this would have represented about $20 million per year.
4.36 However, it has so far been unable to meet this target:
- For the 2003 program year, the Financial Operations Division estimated that absolute errors in payments to producers totalled $70 million, or 10.2 percent.
- For the 2004 program year, the Division estimated that absolute errors in payments to producers totalled $43 million, or 6.5 percent.
4.37 When we completed our audit, the Division's preliminary calculations for the 2005 program year indicated that estimated absolute errors were lower than in previous years and were beginning to approach the internal target of three percent.
4.38 Pre-payment reviews. The Financial Operations Division reviews all applications for payments of more than a specified amount before approving them. It then returns applications with payment errors to the Directorate, so the errors can be corrected.
4.39 All payments that exceed the specified amount must be included in the Financial Operations Division's pre-payment reviews. The number of pre-payment reviews that the Division does depends on how much confidence it has in the accuracy of the Farm Income Programs Directorate's application processing. By the time we completed our audit, the Division's records showed that it had conducted pre-payment reviews of about 12,000, or more than 50 percent, of the 2005 program-year applications that resulted in payments to producers. It returned just over 4,000, or about 30 percent, of the applications it reviewed to the Directorate for correction. These applications represented a pre-payment error of 7.2 percent, or pre-payment errors totalling about $33 million.
4.40 Post-payment reviews. To assess the Department's overall processes and estimate errors in payments, the Financial Operations Division also reviews samples of payments after the cheques and Calculation of Program Benefits statements have been sent to producers. From these reviews, the Division estimates that 65 percent of the dollar value of errors in payments made to producers is due to overpayments, and 35 percent is due to underpayments. For the 2004 program year, this would have resulted in estimated overpayments of about $28 million and estimated underpayments of about $15 million.
4.41 The Financial Operations Division also uses information from the post-payment reviews to determine the specified amount for the pre-payment reviews. To reduce the errors in payments, the Division lowered the specified amount for its reviews each CAIS program year. Because more payments are now subject to review, the Division has had to increase its staff—from 36 in 2005 to 42 in 2006.
4.42 The Financial Operations Division's role is to assess whether the Farm Income Programs Directorate has adequate procedures and whether those procedures are working well. The Division's workload is directly dependent on how accurately the Directorate calculates program payments. The Division is detecting a large number of errors, representing a high dollar value, after applications have been processed by the Directorate. In our opinion, it would be more efficient for the Directorate to improve its procedures to prevent errors in processing, rather than relying on the Division to detect them during their reviews at the end of the process.
4.43 As mentioned earlier, the Financial Operations Division uses information it gathers from the post-payment reviews to estimate the absolute errors in payments to producers. We found that the Division uses a reasonable methodology for estimating errors. However, it does not have the statistical support to determine the accuracy of the estimate.
4.44 Recommendation. Agriculture and Agri-Food Canada should use a statistically valid methodology to estimate errors in farm income support program payments.
Agriculture and Agri-Food Canada's response. The Department agrees with the recommendation and is investigating the cost and feasibility of implementing additional accuracy determination methodologies to supplement approaches, which it will implement if they are found to be cost-effective and feasible. Added to the significant progress the Department has already made to reduce errors, implementing this recommendation could further enhance its current approach to internal reviews and verification.
4.45 Dollar amounts of errors not publicly reported. The Department's target for errors in payments to producers is three percent. The Department does not report this target, or how well it meets it, to Parliament, stakeholders, or other interested parties, even though this information would be helpful in assessing the Department's performance.
4.46 Recommendation. Agriculture and Agri-Food Canada should establish and publicize a target for the accuracy of payments for its farm income support programs, and should report to Parliament, and other interested parties, how well it meets this target.
Agriculture and Agri-Food Canada's response. The Department agrees with establishing a target for the accuracy of payments and, as noted, has in fact established an internal target. The Department will also take into consideration the recommendation to report to Parliament, and other interested parties, on its achievements in meeting payment accuracy targets.
The Department focuses on minimizing overpayments
4.47 The primary objective of the CAIS program is to help producers protect their farm income from risks, such as adverse weather, diseases, and low market prices. It is a complicated program that requires producers to provide accurate and detailed information in their applications about expenses, revenues, and inventories. For example, when completing their applications, producers have to select the correct codes from a list of more than 1,500 different codes.
4.48 Because this complexity can lead to errors in the information, we expected the Department to have processes to equitably manage the risk of underpayments and overpayments to producers. The Department's approach for two previous farm income support programs, the Agriculture Income Disaster Assistance (AIDA) program and the Canadian Farm Income Program (CFIP), focused on both overpayments and underpayments, with full verification and review of all payments. Therefore, we assessed how the Department processes applications and ensures that applicants receive the benefits they are eligible for.
4.49 CAIS payment errors can also affect payments from other programs, because the Department sometimes uses producers' CAIS data to calculate their benefits for other programs.
4.50 In March 2006, officials told the Department's senior management that one of the important elements of the Department's approach to the CAIS program is the "focus on overpayment risk." The Department told us that it detects and corrects a large number of errors before payments are sent to producers and that its approach reflects producers' interests and the overall integrity of the CAIS program.
4.51 The Department also told us that it takes a balanced approach to overpayments and underpayments. Nevertheless, we found a number of instances where the Department's processes and controls place more focus on identifying and correcting overpayments compared to underpayments.
4.52 We found that one of the principal ways the Department focuses on overpayments is through the risk assessment tests. The Farm Income Programs Directorate reviews applications in detail when the major risk assessment tests indicate a higher than expected payment and, therefore, a risk of overpayment to the producer. However, the risk of an underpayment to the producer does not trigger a similarly detailed review. The Directorate then sends the payment files to the Financial Operations Division for independent review before the cheques are sent to producers.
4.53 The Department does not dispute the fact that the Farm Income Programs Directorate's major risk assessment tests focus on the overpayment risk. Nor does it dispute the fact that the Financial Operations Division's reviews represent the last step in the process to verify that payments are accurate.
4.54 However, according to its pre-payment review criteria, the Financial Operations Division does not review applications that have payments that are below the specified amount. It also does not review "zero payments," which represent about 45 percent of all applications. Consequently, about 70 percent of the applications filed each program year are not reviewed by the Division before the Department sends the Calculation of Program Benefits statements to producers. If a payment is an underpayment and is below the specified amount, the application would not be reviewed by the Division before the producer receives the Calculation of Program Benefits statement. In our opinion, the payment of any amount is important to producers. In addition to paying a fee for participating in the CAIS program, 70 to 80 percent of producers also pay fees to have their applications prepared.
4.55 We are concerned that the Department's pre-payment processes do not control the risk to producers of underpayments for files below the specified amount that are not reviewed by the Financial Operations Division. The Department puts the onus on producers to question the figures and identify incorrect payments. However, the Department does not believe that what we consider to be a focus on overpayments has a significant impact on producers.
4.56 The Department estimates the absolute errors in payments to producers, which include both underpayments and overpayments. However, we believe two additional steps need to be taken to confirm this estimate. As we recommend earlier, the Department should use a statistically valid methodology to estimate errors in farm income support payments.
4.57 The Department also needs to include zero payment applications in its samples for post-payment reviews. By their nature, zero payment applications only have a risk of underpayment, not overpayment. About 45 percent of all applications result in zero payments. If these zero payments are excluded from the post-payment samples to detect errors, there is a risk that the Department's estimate that only 35 percent of the errors are underpayments is inaccurate.
4.58 Recommendation. When estimating errors in payments from farm income support programs, Agriculture and Agri-Food Canada should include zero payment applications in its post-payment samples.
Agriculture and Agri-Food Canada's response. The Department agrees with the recommendation and is investigating the cost and feasibility of implementing additional accuracy determination methodologies, including sampling zero payment files, which it will implement if they are found to be feasible and cost-effective.
4.59 Another example of the Department's focus on overpayments is the processing instructions for cases where producers provide a plausible reason for the Department to accept new information after the 90-day deadline for requesting changes or launching appeals. These instructions direct processors to "determine the correct values, update the . . . information, and reprocess . . . only where the payment will decrease." There is no equivalent instruction for underpayments. However, the Department informs us that the federal-provincial-territorial agreements limit adjustments that affect payments to 90 days from the date of the Calculation of Program Benefits statement. The federal-provincial-territorial agreements also require the Department to recover overpayments whenever they are discovered.
4.60 Recommendation. When processing applications for benefits, under farm income support programs, the emphasis that Agriculture and Agri-Food Canada places on efforts to prevent, detect, and correct both under-payments and over-payments to producers should be proportional to the relative risks of underpayments compared with overpayments—as determined through the Department's revised post-payment sampling results—as recommended elsewhere in this chapter.
Agriculture and Agri-Food Canada's response. The Department agrees with the recommendation. It will undertake, as indicated in its responses to other recommendations elsewhere in the chapter, to adjust its review and verification processes; to ensure a balance between overpayments and underpayments; and to ensure that, by the time payments reach producers, both have been identified and corrected to the same degree. Efforts in this regard will be proportional to the relative risks of underpayments compared with overpayments and will reflect both producers' interests and the overall integrity of the program.
The Department needs to strengthen its focus on continuous improvement
4.61 We expected the Department to have developed and implemented procedures to monitor and improve the delivery of the CAIS program. To determine whether it maximizes opportunities to improve processing accuracy, we reviewed the way the Department analyzes and uses data.
4.62 Data not used well. We found that a number of groups in the Department collect a lot of useful information, but the Farm Income Programs Directorate is not systematically using that information to improve processing accuracy.
4.63 The Directorate's system for processing applications includes three quality assurance reviews, at different stages, for which reports are sent to the Directorate's processing managers. These reports focus on the number of applications containing errors that were returned for processors to correct and the number of applications that are still outstanding. However, we found that the Directorate did not analyze the data to identify trends or target specific areas for monitoring.
4.64 As we indicated earlier, by the time we completed our audit, the Financial Operations Division had reviewed more than 50 percent of the 2005 payment applications and had returned about 30 percent to the Farm Income Programs Directorate for correction. Through these reviews, the Division gathers important data about processing errors as well as information about the errors producers make in their applications. However, the Division and the Directorate sometimes disagree about what constitutes an error and which errors require correction. We also found that the Directorate does not systematically use or analyze the Division's data to improve the accuracy of its application processing.
4.65 Furthermore, the Farm Income Programs Directorate's audit services unit conducts on-farm audits of producers' records to verify the accuracy of payments made to producers. At the time of our audit, the unit was completing the on-farm audits for the 2003 program year and was beginning the audits for the 2004 and 2005 program years.
4.66 We found some inaccuracies in the database the Department uses to record the results of the on-farm audits. Nevertheless, the Department's analyses of the information in the database indicate that
- about 90 percent of the audits of 2003 program-year payments resulted in changes to the original payment amount, and
- about 30 percent of the audits changed the payments by $5,000 or more.
4.67 We also found that the auditors do not analyze the causes of the errors—whether the Department made errors processing the application or whether producers submitted inaccurate information. In our opinion, these audits could provide valuable information about the accuracy of the data submitted by producers and about the accuracy and reliability of the Department's processing procedures.
4.68 Despite the numerous sources of relevant information, the Directorate does not systematically gather or analyze the data. It is therefore missing an opportunity to continuously improve the way it processes CAIS applications.
Some improvements are needed in monitoring and training processing staff
4.69 Although it has automated some steps, the Farm Income Programs Directorate still relies heavily on its staff to manually process the applications that fail the major risk assessment tests. Therefore, the employees that process applications need appropriate guidance and training to do their work accurately. We reviewed the Department's training materials and the way it monitors its processors' performance.
4.70 We found that the Department appoints senior processors as resource people to offer guidance to other processors. However, we also found that processors do not have to have successfully completed training before they begin processing applications, and the Department has not established a grading system for the training quizzes that are supposed to test how well employees understand the material. Some employees do not take the quizzes, some do poorly on the quizzes, and some are absent for significant portions of the courses.
4.71 Processing managers use quality assurance reports to monitor processors' performance and to determine whether they follow procedures. The processing managers ask the quality assurance team to increase the monitoring if processors make too many procedural errors.
4.72 The quality assurance reviews could also be used to provide more prompt and useful feedback on individual processors' errors, so managers can better decide what additional training employees need.
4.73 In addition, while we found that processors' individual performance objectives include targets for production, timeliness, and adherence to procedures, there are no targets for payment accuracy. By the time we completed our audit, from its pre-payment reviews, the Financial Operations Division had returned about 30 percent of the applications to the Farm Income Programs Directorate for correction. These applications represented a pre-payment error of 7.2 percent, or pre-payment errors totalling about $33 million.
4.74 Recommendation. Agriculture and Agri-Food Canada should strengthen its focus on continuous improvement in the administration of its farm income support programs by systematically
- collecting relevant information about the nature, extent, and timing of errors;
- analyzing the information; and
- continually applying the results of the analyses to improve all phases of application processing.
Agriculture and Agri-Food Canada's response. The Department agrees with the recommendation to strengthen its focus on continuous improvement. Specifically, the Department agrees that using all the information it has available more systematically will ensure that it builds on past improvements, while fostering a stronger continuous improvement culture.
Service time standards are not being met
4.75 The Department aims to deliver efficient, effective, client-focused programs. However, we found that one of the common complaints from producers is that CAIS payments are not timely.
4.76 We reviewed a sample of the 2004 program applications and calculated how long it took the Department to process them—from the date the Department received an eligible application to the date it sent the Calculation of Program Benefits statement to producers. We found that, on average, the Department took 120 days to process the applications.
4.77 During the first two years of the CAIS program, the Department measured its service time standard from the date the application was entered in the database to the date the application for payment was approved. The Department aimed to process applications received within 60 days during non-peak times and within 75 days during peak periods. The Department's records indicate that it met its service standard 65 percent of the time in 2003 but only 26 percent of the time in 2004.
4.78 For the 2005 program year, the Department began to deduct the time that processors spend waiting for producers to provide additional data from the time taken to process an application. As a result, the Department expects its performance against its service standard to improve. During our review of a sample of the 2004 program applications, we found that, most of the time, processors requested additional information by telephone and usually received it within 24 hours.
4.79 The Department also increased its service time standard to 90 days during peak periods. In addition, it now counts the days from the date it receives an eligible application to the date it approves a claim for payment (and considers processing complete). At the end of our audit, the Department's records indicated that it met its service time standard for the 2005 program-year applications, about 38 percent of the time.
4.80 We agree that using the date the Department receives the application from a producer instead of the date the information is loaded into its database is an appropriate way to measure how long it takes to process applications. However, a more client-focused approach is to use the date the Department sends Calculation of Program Benefit statements to producers—when it informs them of the outcome of their application—instead of the date an application is approved for payment.
4.81 The Department instructs processors to keep diligent records of all contact with producers in the electronic database, including the dates they requested and received additional information. We agree that accurately measuring how long it takes to process applications depends on processors recording accurate and complete information. We also agree with the Department's opinion that there may be a trade-off between accuracy and timeliness. While taking more time to review applications can reduce errors, it can also increase the time producers have to wait for their benefits.
4.82 Recommendation. Agriculture and Agri-Food Canada should set realistic service time standards for processing farm income support program applications and should accurately measure its performance against those standards.
Agriculture and Agri-Food Canada's response. The Department agrees with the recommendation and will continue working with provincial partners to establish national service standards for the CAIS program to improve the predictability of CAIS processing.
Provisions of the federal-provincial-territorial agreements
The Department is not fully meeting the audit and reporting requirements of the agreements
4.83 The federal-provincial-territorial agreements include important monitoring provisions that enable each of the parties to confirm that the Canadian Agricultural Income Stabilization (CAIS) program is being administered according to the agreements' terms and conditions. Alberta, Ontario, Quebec, and Prince Edward Island deliver the CAIS program to their own provinces; the federal government delivers it to the rest of Canada.
4.84 Given that the federal government pays 60 percent of the CAIS program-delivery costs for the provinces and territories and that the provinces and territories pay the other 40 percent, both levels of government need to know whether funds are being properly managed and whether the program is being delivered according to the federal-provincial-territorial agreements.
4.85 We expected Agriculture and Agri-Food Canada to have mechanisms in place to ensure that all parties are meeting the monitoring provisions—the auditing and reporting requirements—of the agreements. We assessed how well the Department ensures compliance with the audit and reporting provisions in the CAIS agreements. We did not audit the provinces' delivery of the CAIS program.
4.86 The agreements that govern the CAIS program include various audit provisions. One audit requirement in the agreements addresses financial reconciliations and states that all parties must
- reconcile their claims for program payments and administrative expenses with the amounts reported in their audited financial statements,
- conduct audits of those reconciliations, and
- provide the audits to those who made the payments within nine months of the end of the fiscal year.
4.87 We found that the Department had not prepared its reconciliation audits by the deadlines, but had completed them by the time we completed our audit. We also found that the Department did not obtain any of the audited reconciliations of the provincial agencies' claims within the deadlines. However, by the time we completed our audit, all but two of the four audits were complete. The Department took appropriate action to obtain the required information from the other two provincial agencies, eventually withholding federal funding from one of them.
4.88 Another audit provision in the agreements requires the Department to periodically conduct compliance audits, at its discretion, to determine whether the federal and provincial agencies administer the CAIS program according to the agreements' terms and conditions. Compliance audits also enable the Department to assess national consistency in program delivery. We found that the Department had not completed any compliance audits, although it was planning one for the 2004 program year that would include all provincial agencies as well as the Department.
4.89 The agreements also require each administration to prepare an annual report on their CAIS operations, revenues, and expenditures as soon as possible following the end of each fiscal year. We found that neither the Department nor the provincial administrations had completed any of the required reports.
4.90 Recommendation. Agriculture and Agri-Food Canada should ensure that it meets the audit and reporting requirements of the federal-provincial-territorial agreements.
Agriculture and Agri-Food Canada's response. The Department agrees with the need to meet the audit and reporting requirements of the federal-provincial-territorial agreements. In this regard, it has already been working with provinces on finalizing audit parameters and carrying out audits and will accelerate this work to ensure that audits are conducted in a more timely manner.
Performance monitoring and reporting
Performance monitoring and reporting needs to be improved
4.91 According to the Treasury Board's policies and guidelines (the Policy on Transfer Payments and its guide, Preparing and Using Results-based Management and Accountability Frameworks), departments need to establish mechanisms that enable managers to measure and report program results and to assess program effectiveness. We expected Agriculture and Agri-Food Canada to have developed a framework for measuring the effectiveness of the CAIS program and for reporting to Parliament and other interested parties.
4.92 We found that the Department has developed a results-based management and accountability framework that contains performance indicators for all its Business Risk Management programs—including the CAIS program—and established high-level expected results for those programs. However, for the CAIS program, it has not established specific and measurable targets to assess program performance or set deadlines to develop these targets. In our view, the lack of specific and measurable targets reduces management's ability to assess whether the CAIS program is achieving its objectives and to report to Parliament on the performance of the CAIS program.
4.93 Furthermore, while it has established performance indicators in the Results-Based Management and Accountability Framework, the Department is not preparing timely, comprehensive reports on all of them. For example, the data the Department routinely gathers focuses on operational information, such as the number of CAIS applications it received and processed and the amounts it paid producers. When we completed our audit, the Department had only prepared a comprehensive report for the 2003 CAIS program year.
4.94 While it continually updates this report as new information becomes available, the Department told us that it needs to wait until it has processed all 2004 CAIS applications, before it can prepare a similar report and measure its performance against the CAIS program's performance indicators for that year.
4.95 By the end of October 2006, the Department was still processing 830, or 1.5 percent, of the 57,000 applications it had received for the 2004 program year. In our view, monitoring program performance against the key indicators need not wait until the Department has processed all the applications. Even if all of the necessary information is not yet available, the Department can prepare comprehensive reports for each CAIS program year, which would be useful for monitoring and improving the delivery of the CAIS program and for reporting CAIS program results to Parliament.
4.96 Recommendation. Agriculture and Agri-Food Canada should
- establish specific and measurable targets for all of the performance indicators in its Results-Based Management and Accountability Framework; and
- produce timely reports to Parliament on the performance of farm income support programs, measured against those targets.
Agriculture and Agri-Food Canada's response. The Department agrees with this recommendation. It will, in conjunction with its provincial counterparts, develop targets for each of the measures in the Results-Based Management and Accountability Framework and report progress and achievements, on these targets, in the annual departmental performance report to Parliament. This task will be completed by December 2007 and will include, for example, targets related to producer participation in CAIS and to timeliness of payments.
Conflict of interest
The Department needs to continue its efforts to avoid conflicts of interest
4.97 We reviewed potential conflict of interest cases, in which employees who process applications for farm income support payments help producers prepare their applications. In our opinion, processors who also work as private consultants to help prepare Canadian Agricultural Income Stabilization (CAIS) applications contravene conflict of interest provisions in the Values and Ethics Code for the Public Service. These employees also risk contravening provisions that deal with providing preferential treatment and benefiting from information that is not available to the public. When we brought these cases to Agriculture and Agri-Food Canada's attention, it began investigating them.
4.98 The Department assesses CAIS applications against risk assessment tests. Applications that pass the tests are deemed low risk and are less likely to be reviewed by the Farm Income Programs Directorate. Individuals with detailed knowledge of these tests could use that knowledge to manipulate the data in producers' CAIS applications and to trigger payments.
4.99 We are not aware of any cases where such manipulation actually occurred. However, it may occur if CAIS employees help to prepare CAIS applications (as private consultants).
4.100 The Department told us that only employees who develop and implement these tests have full access to the calculations and details on how the tests work. The Department also told us that the tests, and the way processors use them to assess applications, are so complex that they would not be useful to employees.
4.101 We found that the processors' manuals contain specific explanations for why data from producers fail the tests. The Department requires processors to understand how the test result was calculated and to explain any possible anomalies. If they conclude that the level of risk associated with the application is acceptable, they can send the application "straight to payment."
4.102 Department officials also told us that some CAIS processors prepare CAIS application forms for their own farms. Processors must indicate that they are Department employees to ensure that their applications, and those of their immediate colleagues, are processed by other teams.
4.103 The Department sends annual notices to employees reminding them of their responsibilities under the Values and Ethics Code for the Public Service. After we notified the Department of the potential conflict of interest cases, it sent an additional reminder to 18 of the almost 400 employees who process applications for farm income programs.
4.104 We reviewed the confidential reports that several employees filed as a result of this reminder as well as their past reports and reports previously filed by other employees. Some CAIS employees said they were providing private consulting services on preparing CAIS applications, and some said they were charging fees for their services.
4.105 Some of the employees had disclosed such activities to management before we raised the issues with the Department—of those, some had done so for a number of years. However, the Department told us that it did not have records of follow-up action for all of the confidential reports that had been filed. After we raised our concerns about potential conflicts of interest, the Department conducted investigations and told five employees to stop preparing CAIS applications for their clients.
4.106 We also found that the Department had not always properly protected documents containing information about the risk assessment tests, as required by the Government Security Policy. However, as the Department does currently identify recent versions of these documents, if any of the almost 400 employees of the farm income programs processing unit were to use this information for private purposes, they would contravene the Policy and the Values and Ethics Code for the Public Service.
4.107 Finally, we found that although applicants may voluntarily disclose the name of the individual or company that prepared the application, they are not required to do so.
4.108 The Department informed us that it was strengthening its values and ethics regime, before we notified them of potential conflicts of interest. Furthermore, in December 2006, it reminded all employees of their obligations, under the Values and Ethics Code for the Public Service. It specifically told employees to file a report if they
- owned a farm and were applying for income assistance from the Department, or
- were considering helping producers apply to the Department's grant and contribution or farm income support programs.
4.109 The Department also told us that it is implementing a process to address risks associated with employees preparing applications for their own farms, which would include on-farm audits if necessary.
4.110 Recommendation. Agriculture and Agri-Food Canada should strengthen measures to reduce the risks of potential conflicts of interest. At a minimum, the Department should strengthen its values and ethics program—making it clear to employees that they cannot prepare applications for others for a fee—and establish processes to
- identify processors who also prepare applications for farm income support programs;
- address risks associated with employees preparing applications for their own farms or their families' farms, including performing on-farm audits if necessary; and
- ensure that all applications prepared by farm income support program employees are approved manually rather than electronically.
Agriculture and Agri-Food Canada's response. The Department agrees. It has been strengthening, and will continue to strengthen, its values and ethics program, by continuing to advise all new staff of their obligations, under the Values and Ethics Code for Public Servants, and reminding them of their obligations on an annual basis. Recently, the Department strengthened its conflict of interest processes to proactively involve management in the resolution of potential or actual conflicts of interest. The Department has also recently instituted a formal training program and will be delivering this program across the country. Additionally, it has instituted a 1-800 hotline number where employees and managers can get immediate assistance.
CAIS processors are required to self identify if they are completing a CAIS application for themselves, for a family member, or for friends. They are also asked to identify that application, and these applications are then processed by someone other than themselves in a separate team designated for that function, effectively distancing the employee from that application. Finally, the Department has informed employees through a Deputy Minister letter that they cannot prepare CAIS applications for a fee. This will be repeated on a yearly basis.
4.111 Where Agriculture and Agri-Food Canada delivers the Canadian Agricultural Income Stabilization (CAIS) program, the Department does not have a system in place to assure itself that applications are processed accurately. The Department focuses on detecting and correcting overpayment errors; its method of estimating errors in payments lacks statistical support, and it omits the sampling of zero payment applications. Consequently, there is a risk that the impact of the focus on overpayment is greater than that estimated by the Department. However, the Department does not believe that there is a significant impact on producers in what we consider to be a focus on overpayments. The Department has improved its accuracy of processing 2005 applications that result in a payment to producer; however, while close to reaching its target for payment accuracy, the Department has not yet met it.
4.112 Agriculture and Agri-Food Canada needs to improve the efficiency of processing applications and the effectiveness of managing the risks to applicants and to the federal government. Specifically, we found the following:
- The way the Department calculates benefits is complex and not transparent. The Department does not indicate or explain to producers how and why it changes the figures they originally submitted or how it uses the altered figures to calculate the producers' final CAIS benefits.
- The Department needs to improve its systematic use of available information to strengthen its continuous improvement efforts.
- The Department focuses on detecting and correcting payment errors at the end of the process instead of focusing on preventing them in the first place.
- There are long delays before producers are told whether they will receive a benefit. On average, the Department took 120 days to process 2004 program-year applications.
4.113 Given the amount of complex information producers are required to provide in their CAIS applications and the volume of applications it receives, the Department has implemented a reasonable approach to verify the completeness and accuracy of source data.
4.114 The Department is not meeting its service time standards for processing applications and, therefore, is not processing applications within a reasonable period of time.
4.115 The Department has not established an external service standard for payment accuracy, nor has it established specific and measurable targets for all of the performance indicators in its Results-Based Management and Accountability Framework. Establishing such targets would improve the Department's ability to assess the CAIS program, to determine whether it is achieving its intended objectives, and to report on its performance to Parliament. Furthermore, ongoing monitoring of the CAIS program's performance against the key indicators would be useful for improving the delivery of the program and reporting its results to Parliament.
4.116 With the exception of the reporting requirements, the Department substantively ensured that the monitoring requirements of the federal-provincial-territorial agreements were met. We did not audit the delivery of the CAIS program by the provincial administrations.
About the Audit
Our first objective was to determine whether Agriculture and Agri-Food Canada has reasonable approaches to verify the completeness and accuracy of the source data for Canadian Agricultural Income Stabilization (CAIS) applications and to ensure that claims are processed accurately and reasonably quickly.
Our second objective was to determine whether the Department has developed a performance measurement framework and is using it to
- assess how well the CAIS program meets its objectives;
- monitor and improve CAIS program delivery; and
- report to Parliament and other interested parties.
Our third objective was to determine whether the Department ensures that parties to the federal-provincial-territorial agreements that govern the delivery of CAIS are meeting monitoring requirements.
Scope and approach
The CAIS program provides income support to producers. It is administered by the Department for all provinces and territories except Alberta, Ontario, Quebec, and Prince Edward Island. Our audit focused on the processes and procedures that the Department uses to manage and deliver the CAIS program.
Specifically, we examined how the Department
- processes applications for income support,
- measures and reports on its performance to Parliament, and
- meets the monitoring requirements of the federal-provincial-territorial agreements.
We did not audit the provinces' delivery of the CAIS program.
The stakeholders that we interviewed included officials from
- several non-governmental organizations that represent more than 200,000 producers,
- the major banks, and
- provincial governments where the CAIS program is and is not delivered by the Department.
We interviewed officials and reviewed documents in Winnipeg (where the CAIS program is administered) and at the Department's national headquarters in Ottawa. We also reviewed files from a representative sample of 137 CAIS applications from the 2004 program year. The eligible 56,515 applications processed on or before 12 April 2006 were divided into low, medium, and high dollar amounts and zero-payment applications. Sample sizes from each of the four groups were selected, so confidence intervals for deviation rates would be less than 10 percent at a 90-percent confidence level. All estimates reported in the chapter have been weighted to accurately reflect the population.
We expected the Department to have
- established policies and procedures to ensure efficient and effective delivery and administration of the CAIS program,
- developed a results-based management and accountability framework that provides for appropriate measurement and reporting of results to the Department's managers as well as to Parliament and other interested parties,
- developed appropriate evaluation criteria to assess the effectiveness of the CAIS program,
- developed and implemented processes to monitor and improve CAIS program delivery, and
- ensured that all parties to the federal-provincial-territorial agreements had fulfilled their respective audit and reporting responsibilities regarding the CAIS program.
Audit work completed
Audit work for this chapter was substantially completed on 31 October 2006.
Assistant Auditor General: Andrew Lennox
Principal: Frances Taylor
Director: Raymond Kunze
For information, please contact Communications at 613-995-3708 or
Agricultural producers—Includes those who produce
- livestock and poultry and related products (for example, eggs, milk, meat, and animal fibres) and ranch fur products (for example, mink, fox, chinchilla, and rabbit);
- crop and plant products (for example, wheat, barley, oats, canola, corn, fruit, nuts, mushrooms, wild rice, maple syrup, and herbs); and
- nursery products (for example, trees, shrubs, annual and perennial plants, cut flowers, and cultivated Christmas trees). (Return)
Absolute errors—The total dollar value of overpayment errors added to the total dollar value of underpayment errors. (Return)
A results-based management and accountability framework is used to identify a program's performance measures and develop strategies to report on outcomes. (Return)