This Web page has been archived on the Web.

2008 December Report of the Commissioner of the Environment and Sustainable Development

Main Points

What we examined

The Agricultural Policy Framework (APF) is a federal-provincial/territorial agreement that came into effect in 2003 to create a national approach to agriculture, with programming aimed at five areas ("chapters"): Business Risk Management, Food Safety and Quality, Science and Innovation, Environment, and Renewal. Although the agreement did not extend beyond 31 March 2008, agriculture ministers of the federal, provincial, and territorial governments agreed to continue existing programs for another year while new programs are developed under Growing Forward, which will replace the Agricultural Policy Framework.

Our audit examined whether Agriculture and Agri-Food Canada has managed the Environment Chapter of the Agricultural Policy Framework appropriately to achieve its objectives for environmentally sustainable agriculture. We also examined its reporting on the performance of the Environment Chapter. We looked at the development of the National Land and Water Information Service (NLWIS), a major information technology project that is important to the support of all programs within the Environment Chapter. Finally, we examined whether the Department used the lessons learned from the Agricultural Policy Framework in developing Growing Forward.

Why it's important

Canada's agri-food sector accounts for eight percent of the gross domestic product and generates $130 billion in sales, including $31 billion in exports. The sector employs one in eight Canadians and contributes to the economic base of many rural communities in Canada. The long-term prosperity of the agriculture industry depends on its ability to co-exist sustainably with the natural environment. However, agriculture has changed significantly in response to market demands, new production technologies, and larger, more intensive operations.

Recent studies show that stresses on the environment from agriculture have been increasing. At the same time, public awareness and concern about these effects is growing. Programs within the Environment Chapter of the Agricultural Policy Framework provide farmers with access to information and resources in order to help them adopt farm management practices that benefit the environment.

What we found

  • The Department does not know to what extent its environmental programs have improved the environment. Departmental reporting is limited because it does not monitor and report on program results beyond outputs, such as the number of completed water projects (for example, wells and pipelines). As a result, senior management cannot be certain whether programs are achieving their intended results and where improvements are needed. The Department has spent about $370 million on environmental projects, but lacks sufficient data to demonstrate that action at the farm level has led to positive environmental change. In addition, agri-environmental targets in agreements with the provinces could not be measured by the planned end of the Agricultural Policy Framework, in March 2008, as the Department intended. Nevertheless, the beneficial management practices funded under AAFC's environment programs—for example, providing water sources for cattle away from streams—are supported by science, indicating that they will likely lead to positive environmental change. The Department is in the process of developing a modelling system to better understand the effects of its programs on the environment.
  • The Department's allocation of operating resources among the Environment Chapter's contribution programs was not supported by adequate information. Nor could the Department provide us with complete information on the operating costs of each program. In addition, information the Department required delivery agents to submit was not consistent with the terms in the signed agreements. Nor was it clear how the Department used the information it did receive. As a result of slow progress in fixing problems in data from delivery agents, the Department prepared a disclaimer on the usability of the data—the primary source for reporting on the performance of its environment programs—until the data problems are fixed. Recently, the Department voluntarily joined a government initiative to improve program delivery that includes a plan to develop revised agreements and simplify program administration.
  • The development of the National Land and Water Information Service (NLWIS) was poorly managed. This $100-million, major Crown project fell behind schedule and had to be revised from original plans. Key decision makers and the Department's senior management were not involved to the degree expected for a project of this complexity. Participants had differing views of the project's goals. In addition, project management suffered from a lack of expertise and continuity; roles and responsibilities were unclear; and project requirements were changed without appropriate approvals. Senior management approved an internal audit and an independent review of NLWIS. During our audit, steps were being taken to address problems identified by the audit and review.
  • In preparing for the next generation of the APF, the Department carried out formal "lessons learned" reviews. It also consulted extensively with stakeholders, who provided valuable information for Growing Forward. In developing the APF, the Department had experienced serious delays in negotiating and signing agreements with the provinces and territories. Based on that experience, the Department should have anticipated that negotiation and consultation would take longer than expected for Growing Forward and to plan accordingly. When the APF was to end on 31 March 2008, the APF environment programs were extended for up to one year, in order to provide more time for the development of Growing Forward.

The Department has responded. The Department agrees with all of our recommendations. Its detailed responses follow each recommendation throughout the chapter.

Introduction

3.1 The mandate of Agriculture and Agri-Food Canada is to provide policies, programs, and services to clients in the sector, and conduct research targeted to achieve security of the food system, the health of the environment, and innovation for growth.

3.2 Responsibility for agriculture is shared between the federal government and provincial and territorial governments and, as a result, these governments often share the costs of delivering programs. This creates a challenging set of circumstances in which to develop policy frameworks and environmental programs.

3.3 The Department supports Canadian agriculture through the Agricultural Policy Framework (APF), a five-year, $5.2-billion joint federal–provincial/territorial agreement to provide policies and programs to enhance the agricultural sector's long-term profitability, competitiveness, and sustainability. The APF, implemented in 2003, represented a significant change in the Department's approach to the agriculture sector. It was the first-ever attempt by federal, provincial, and territorial governments to develop a national approach to agricultural policies and programs. This represented a significant challenge for the Department, as agreements for governance mechanisms, funding arrangements, and program details had to be negotiated and implemented for all areas of the APF.

3.4 The APF is made up of five areas or "chapters":

  • Business Risk Management,
  • Food Safety and Quality,
  • Science and Innovation,
  • Environment, and
  • Renewal.

Programs under the APF are in place, in partnership with provincial and territorial governments, with some delivered by stakeholder groups.

3.5 The Environment Chapter, with a budget of approximately $600 million over five years, is designed to improve the health of the environment through increased adoption of sustainable development practices. According to the Department, in the context of Canadian agriculture and agri-food production, sustainable development means producing, processing, and distributing agricultural products in a manner that supports or enhances the high quality of life we enjoy in Canada, both today and into the future.

3.6 The Department asserts that environmentally sustainable agriculture will be achieved through accelerating efforts to reduce agricultural risks to the environment, which will benefit Canada's water resources, soil, air, and biodiversity. The APF Environment Chapter programs are designed to provide farmers with resources and information that encourage and facilitate practices to reduce agri-environmental risks. The APF uses incentives and information rather than regulations. This means that producers are encouraged to adopt environmentally friendly agricultural practices, by being made aware of the issues and possible solutions and by being partially reimbursed for their costs.

3.7 The Department's National Agri-Environmental Health Analysis and Reporting Program (NAHARP) measures changes in environmental conditions on agricultural landscapes across Canada. The most recent information from NAHARP, which precedes the APF, shows that changes in the agricultural landscape over a 20-year period (1981 to 2001) have resulted in increased environmental risks in some areas and decreased risks in others.

Focus of the audit

3.8 The objective of our audit was to determine whether Agriculture and Agri-Food Canada could demonstrate that it was managing the Environment Chapter of its Agricultural Policy Framework (APF) to achieve its environmentally sustainable agriculture objectives.

3.9 The APF is in transition as the Department develops its next generation of policies and programs, entitled Growing Forward. We looked at the overall development process for Growing Forward, as the process followed by the Department was the same for all of the chapters, including the Environment Chapter.

3.10 We then focused solely on the APF Environment Chapter. We examined the Department approach to managing the chapter's contribution programs. We did not examine the delivery of programs by provincial and non-provincial delivery agents, nor the payments made to these organizations.

3.11 To determine whether the performance measurement strategy for measuring and reporting of results was operating as intended, we examined the Department's Results-Based Management and Accountability Framework, departmental performance reports, and internal performance reporting.

3.12 Finally, we examined one key program within the Environment Chapter: the National Land and Water Information Service. We wanted to determine whether this $100-million information technology project had been appropriately managed.

3.13 More details about the audit objectives, scope, approach, and criteria are in About the Audit at the end of this chapter.

Observations and Recommendations

Agricultural Policy Framework and its successor

3.14 In 2005, Agriculture and Agri-Food Canada began a process to develop a policy framework that would follow the Agricultural Policy Framework (APF) when it was expected to end on 31 March 2008. As of April 2008, this new policy framework, now entitled Growing Forward, was still under development and the federal, provincial, and territorial ministers announced a continuation of APF programming for up to one year.

Analytical exercises were conducted

3.15 The Treasury Board of Canada Secretariat's Management Accountability Framework sets out its expectations of senior public service managers for good public service management. One of these expectations is to encourage continuous learning and improvement.

3.16 We expected the Department to have a lessons-learned process whereby it learned from the APF's strengths and weaknesses and applied those lessons to the development of Growing Forward. We found that the Department used lessons learned from the APF to identify opportunities to improve program management and administration.

3.17 To measure the APF's achievements, and to define the policy and program framework for Growing Forward, the Department undertook three formal analytical exercises. These were

  • a lessons-learned initiative,
  • an evaluation of the Environment Chapter, and
  • extensive stakeholder consultations.

3.18 The lessons-learned initiative was designed as a high-level assessment of the APF's various administrative, management, and implementation aspects. It was based on input from staff involved in the APF. The initiative identified what worked well and what required improvement. Documentation provided by the Department indicates that steps are being taken to implement identified improvements.

3.19 The evaluation of the APF Environment Chapter, conducted during the 2006–07 fiscal year, was intended to assess the programs' rationale and relevance, effectiveness of design and delivery processes, progress on results, and cost-effectiveness. The evaluation noted the slow uptake of the programs (the increase in the number of producers using the programs), and that the number of projects funded by the programs that were completed as at 31 March 2006 fell short of targets. However, by 31 March 2008, departmental records indicate that program use was close to the Department's targets, ranging from approximately 75 to 95 percent, for most of the programs.

3.20 In preparing for Growing Forward, the Department consulted with various stakeholders. The consultations focused on all aspects of agriculture and agri-food policy and included several phases: a first phase with invited stakeholders, a second phase of public consultations across the country, and a third phase of meetings with selected stakeholders to discuss the proposed future framework. A fourth phase of stakeholder consultation was planned for May 2008. In addition, approximately 100 submissions were received from associations and another 100 from individuals. Some provinces are also conducting consultation sessions with their stakeholders.

3.21 The Department recorded the consultation discussions and produced a report from each of its consultation sessions. Reports on what was heard are posted on the Department's website, by city in which the consultation occurred. As a result, the Department knows what participants said, and it communicated what it heard in subsequent consultation sessions. However, it did not formally gather stakeholder feedback on the accuracy of the published consultation information.

An overall analysis of stakeholder consultations is to be prepared

3.22 Given the time and effort invested by the participating stakeholders and by the Department, we expected that the output from the consultation process at each phase would be integrated into an overall analysis of key themes and issues and that this analysis would be made public. Since the consultation process was ongoing at the time of our audit, it was too early to conclude against this expectation. However, the Department has indicated that it will prepare such an analysis once the consultation process is complete.

3.23 Stakeholders informed us that the consultation process for Growing Forward was considerably better than that for the APF. In addition, a survey of participants was conducted after the second round of Growing Forward consultations. It showed that 63 percent of participants felt that the process for developing the next generation of agriculture and agri-food policy was clearly communicated, and that 69 percent believed that relevant issues were being addressed. However, only 49 percent understood how the consultation process would be connected to the decision-making process.

3.24 An internal presentation was made to senior the Department management in March 2007 on the survey results and the Growing Forward consultation process. The presentation indicated that there was skepticism about the final outcomes and that stakeholders were reserving their judgment until they saw how their input was incorporated.

3.25 Many factors will affect Growing Forward, such as discussions with the provinces and territories. However, by conducting its overall analysis, the Department will be in a better position to respond to queries from stakeholders as to how the consultations influenced Growing Forward decisions.

Development of a coherent policy framework is under way

3.26 We expected that the Department would have reviewed opportunities to fully integrate all the chapters and the environmental programs to demonstrate that Growing Forward is a coherent policy framework. We also expected that the Department would explain the policy and program changes from one framework to the next. However, because Growing Forward development was still under way, we could not yet arrive at a conclusion whether these expectations had been met.

3.27 Given that Growing Forward builds on the APF, it will be essential for the Department to clearly illustrate how the policy framework and programs have changed. This is essential to encourage maximum program "buy-in" by producers and to ensure that one program or chapter does not act as a disincentive for another program or chapter. the Department has indicated that such information will be available to producers and other stakeholders when development of Growing Forward programming is complete.

There was uncertainty for stakeholders

3.28 The first set of APF environment programs took much longer than expected to negotiate and sign agreements with the provinces and territories and develop agreements with delivery agents. Based on lessons learned, we would have expected the Department to have anticipated similar delays with the development of Growing Forward. In response to the April 2007 evaluation of the APF Environment Chapter, the Department management committed to developing "a clear transition strategy" from APF to Growing Forward programming to ensure that changes to programming would not affect service delivery and lead to implementation delays.

3.29 We recognize the challenges inherent in negotiating a comprehensive multilateral policy framework with all provincial and territorial governments that is appropriately informed by consultations with the sector. However, the Department could not provide us with evidence of a clear and timely transition plan to manage the risk of delays and to ensure consistent service delivery at the anticipated end of the APF.

3.30 In March 2007, the Department was aware of the risk that it might not be possible to conclude negotiations in time to implement Growing Forward programs in April 2008. In November 2007, federal, provincial, and territorial ministers announced their intention to seek the necessary authority to continue APF programming for up to an additional year, although the Department had initiated steps to obtain the necessary federal authority a few months earlier. Authority for a transition year was obtained, subject to approval of funding. In February 2008, the Department requested and received approval for funding to continue APF programs.

3.31 In March and April 2008, there were joint federal and provincial or territorial announcements of program continuation, by province and/or territory. Uncertainty regarding the programs continued until that time. Stakeholders, delivery agents, and the Department managers expressed concern about the environment programs' uncertainty, the extent of funding available, and their ability to plan under such conditions.

3.32 We recognize that a clear transition strategy could not be put in place until the appropriate authorities were obtained. This severely limited the Department's ability to have a clear and timely transition plan for the one-year extension.

3.33 A transition plan will also be needed when the extension year ends and Growing Forward begins. Such a transition plan would ensure that stakeholders are well informed in advance and would minimize program uptake delays and disruptions. A transition plan would include such things as a timeline for Growing Forward, changes made as well as those not made to programs, any administrative changes, and the amounts of funding available for projects under each program.

3.34 At the end of our audit, details on Growing Forward programming were not yet available. Therefore, it was too early for the Department to have prepared such a transition plan for Growing Forward.

3.35 Recommendation. In order to ensure that all stakeholders are well informed and to minimize program uptake delays, Agriculture and Agri-Food Canada should develop a clear and timely transition strategy well in advance of the end of its programs. Once Growing Forward is completed, Agriculture and Agri-Food Canada should prepare a schedule of review and consultations through the life of the new framework to ensure a smooth and timely transition to future policy frameworks.

The Department's response. Agriculture and Agri-Food Canada agrees. The Department is committed to working with provinces and territories to achieve smooth and timely transitions between policy frameworks and programs. These efforts involve regular stakeholder consultations and reviews, which include assessments of strengths and weaknesses of past approaches.

The Department will discuss the process of review and consultation with provinces/territories through the life of the framework, with a view to ensuring timely renewal. The Department recognizes the importance of beginning these processes early, because of the complexity of federal-provincial/territorial negotiations in this area of shared jurisdiction.

A strategic environmental assessment is to be completed

3.36 The Government of Canada's Cabinet Directive on the Environmental Assessment of Policy, Plan and Program Proposals calls for a strategic environmental assessment to be conducted for any policy, plan, or program proposal being provided to a minister or Cabinet for approval, and whose implementation may result in important environmental effects, either positive or negative. The Privy Council Office has indicated that departments are responsible for complying with the Cabinet directive and are accountable for the quality of their analyses.

3.37 At the time of our audit, Agriculture and Agri-Food Canada had not completed a strategic environmental assessment of Growing Forward. However, Department officials indicated that the Department intended to prepare one. Such assessments allow decision makers to anticipate, prevent, or reduce potential environmental harm and to enhance any environmental benefits.

Environment Chapter—contribution programs

3.38 By making the environment one of the key Agricultural Policy Framework (APF) elements, Agriculture and Agri-Food Canada recognized that the sustainability of the agriculture sector was closely linked with the natural environment. The Department established national environmental programs to improve the quality of water, soil, air, and biodiversity.

3.39 The Department either delivers or coordinates the delivery of five contribution programs (Exhibit 3.1).

Exhibit 3.1—Environment Chapter's contribution programs and their actual expenditures over five years

Environmental Farm Planning ($70 million) is a voluntary and confidential process used by farmers to systematically identify environmental risks and benefits from their farming operation, and to develop an action plan to mitigate the risks. Producers are not compensated for completing environmental farm plans. In order to receive funding under the National Farm Stewardship and Greencover Canada programs (described below), producers must complete an environmental farm plan.

National Farm Stewardship Program ($176 million) supports the adoption of environmentally beneficial practices by agricultural producers in the management of land, water, air, and biodiversity, for example, by improving on-farm storage and handling of agricultural waste and building fencing to prevent livestock damage adjacent to waterways.

Greencover Canada Program ($58 million) helps producers improve their grassland-management practices, protect water quality, reduce greenhouse gas emissions, and enhance biodiversity and wildlife habitat, for example, by planting trees and shrubs on agricultural land and establishing cover crops to protect soil from erosion.

National Water Supply Expansion Program ($64 million) improves the ability of agricultural producers to deal with drought situations, for example, by constructing water supplies (such as wells and water dugouts) and conducting groundwater studies.

Environmental Technology Assessment for Agriculture ($3 million) is a research-based program that evaluates the performance of innovative technologies of food production to improve the long-term economic and environmental performance of the sector, for example, by developing a process to use farm wastes for heat and power.

Source: Agriculture and Agri-Food Canada

3.40 The agreements specify the program to be delivered, the dollar commitment of the signatories, and the responsibilities of each signatory. In the Prairie provinces, Agriculture and Agri-Food Canada delivers most of the environment programs. In the other provinces and territories, the Department, the province or territory, or a delivery agent could be responsible for program implementation. For example, the Department delivers all programs in Manitoba except the environmental farm plan program, which is delivered by a non-government agency. In New Brunswick, a provincial agriculture department delivers almost all the programs.

3.41 We expected that the Department would have mechanisms in place to fulfill key provisions in the agreements. The Department negotiated 25 separate agreements to deliver the four on-farm contribution programs across Canada. We reviewed the agreements' monitoring and reporting requirements. We did not examine the Department payments to the provinces/territories and delivery agents.

The process for developing contribution agreements was lengthy

3.42 In March 2004, the Department created a team to coordinate the development of all agreements under the APF. It was responsible for approving the use of agreement templates and any subsequent changes made by environment program officials. The process for developing, negotiating, and signing the agreements was lengthy and complex. The internal evaluation of the APF Environment Chapter identified that agreements were being negotiated with provinces and territories while the Department was still changing their content, resulting in some variations across the country. The combination of drafting agreement templates and the internal approval process added up to one year to the signing of agreements. Given that the APF is a five-year framework, a one-year delay is significant. While the use of templates for agreements is a good idea, there was inadequate advanced planning regarding the template development process and what requirements the Department should include in the agreements' terms and conditions.

Consistency and clarity are lacking in the requirement and use of information

3.43 The Department requires the preparation of work planning documents, even though this requirement was not specified in all agreements. These documents are a means to identify the activities and results that delivery agents will pursue for the coming year.

3.44 Most work planning documents were available for the 2007–08 fiscal year but not for all years. Many of them did not have a clear indication of review by Department officials as only about half of them were signed and dated. Of those that were signed, they were dated late in the fiscal year—even as late as February 2008 for the 2007–08 fiscal year. Because annual work planning documents were completed at or close to the start of the program year's activities, their usefulness and purpose as a planning document was undermined.

3.45 Financial reporting requirements in the agreements often gave delivery agents the option of providing audited financial statements and/or statements of expenditure. We found that these documents were seldom submitted. In addition, there was no indication that the Department reviewed or used the submitted documents for any particular purpose. The agreements also require that the delivery agents submit various performance information reports. At times, the Department accepted substitute documents that only partially met the requirements specified in the agreements. As well, documents were not always submitted with the frequency required. We found that there was a lack of consistency and clarity with respect to what, exactly, delivery agents had to submit and that the Department did not always collect the specific information that was identified in the agreements.

3.46 The lack of consistency and clarity in what information the Department requires and actually uses undermines the importance of identifying reporting requirements in the agreements and adds to the administrative burden of delivery agents.

3.47 Recommendation. When developing agreement terms and conditions, Agriculture and Agri-Food Canada should determine what information it wants, why it wants the information, and how it intends to use it, and develop and communicate a plan for using the information. This information should be clearly and consistently expressed in the agreements' terms and conditions. The Department should monitor and ensure it obtains the requested information.

The Department's response. Agriculture and Agri-Food Canada agrees. Consistent with the Treasury Board of Canada Secretariat's Policy on Transfer Payments and recommendations of the Independent Blue Ribbon Panel on Grant and Contribution Programs, the Department will make its information requirements more clear in new environment program terms and conditions and contribution agreements with recipients, and it will provide information in these documents about why the information is required and how it will be used.

The Department will also ensure consistency of information requirements between different parts of the country and different recipients. The Department will ensure it receives and reviews the required information, and it will limit requirements to information that is needed. As well, it will regularly analyze the information it receives and use this analysis to report on progress toward objectives.

The Department will have most new terms and conditions in place by April 1, 2009; the others will be put in place, in consultation with provincial and territorial partners, by March 31, 2010.

The Department has identified some needed improvements to program delivery

3.48 Following the Treasury Board of Canada Secretariat's release of the Report of the Independent Blue Ribbon Panel on Grant and Contribution Programs in December 2006, Agriculture and Agri-Food Canada took the initiative to participate in working group meetings and developed a high-level action plan to address some of the Panel's recommendations aimed at simplifying program administration, improving risk management, and focusing on clients.

3.49 The Department is in the early stages of implementing its action plan, which includes developing revised contribution agreement templates and implementing a working group to help streamline the use of administrative forms. Several action items will coincide with the development of Growing Forward agreements and programs. Furthermore, the Department will need to develop a means to determine whether changes to the agreements result in the desired improvements to program delivery.

3.50 We also note that the Department created a separate organizational branch on 28 January 2008 in order to improve policy and program coherence and bring more effective management to the environment portfolio.

There has been slow progress in fixing data problems

3.51 The agreements require delivery agents to submit quarterly and annual operational information. We found that the Department has had several challenges with data collection and analysis.

3.52 After receiving the first submission of annual data for the 2005–06 fiscal year in June 2006 and comparing them with the quarterly data already received, the Department found several problems, including the fact that

  • data did not match,
  • mandatory data fields were missing,
  • codes for projects were unidentifiable, and
  • data provided by the same delivery agent could not be reconciled.

3.53 The Department recognized these problems and has made efforts to correct them; however, progress has been slow due to limited staffing availability and the complexity of dealing with multiple delivery agents. The data serve as the primary source for reporting on the performance of the Environment Chapter. The challenges in data collection and analysis raise concerns regarding the accuracy of information presented in the Department's departmental performance reports. To date, the reports contain no notice of data accuracy limitations.

3.54 Due to concerns regarding data quality, the Department prepared a legal disclaimer in February 2008 stating that it

"... cannot guarantee [the data's] accuracy or fitness for use by anyone else, and therefore the user assumes all liability and risk with respect to the completeness, accuracy, fitness for use, or other matter relating to the information or data records... ."

The disclaimer is attached to internal reports. The Department intends to remove it after the data quality problems are resolved.

3.55 We noted that in the 2003–04 fiscal year, department officials had identified the need for an integrated information system to assist with direct delivery in the Prairie provinces and a national information system to collect results and aggregate data from delivery agents for information management and performance reporting. The Department allocated $1.2 million for this purpose.

3.56 However, the Department developed a system for direct delivery only in the Prairie provinces. Despite the national information system being identified as a priority for environment program delivery, it was never developed, which has had a direct impact on the Department's ability to collect accurate and reliable data for performance reporting.

3.57 Recommendation. Agriculture and Agri-Food Canada should develop and implement a standardized process for collecting national data to ensure that it can report accurate, consistent, and timely information on program results.

The Department's response. Agriculture and Agri-Food Canada agrees. It will take steps, by December 2009, to strengthen its performance measurement practices for environment programs. These will include improving the systematic collection and analysis of program performance information to support accurate performance reporting, and meeting with delivery agents to clarify program data standards in order to improve data reliability and reporting requirements.

An interim database has been created by the Department to electronically store, analyze, and report program information received from delivery agents.

Operational resources were not tracked by program

3.58 The Department was allocated $126 million as the operational budget to deliver environment programs (excluding the National Land and Water Information Service; paragraphs 3.84 to 3.105); and $400 million for the programs' contribution portion over the five-year life of the APF.

3.59 We expected the Department to have plans that identified objectives and expected results to be achieved and that operational budgets would be allocated according to these plans. We requested financial information concerning amounts allocated, budgets requested by program managers to deliver the programs, budgets approved, and amounts allocated to environment programs for each year.

3.60 However, the Department was unable to provide a complete set of financial information to demonstrate that this type of planning had occurred. When the APF was introduced, the Department underwent an organizational restructuring. Cross-functional teams were developed and budgets were allocated based on team outputs, rather than on programs. A team output could include more than one program.

3.61 Without an analysis between planned operating expenses, actual budget allocation, and actual operating expenses by program, the Department could not demonstrate that a program's operational requirements were considered when resources were allocated.

3.62 In addition, the Department did not separately track the source of operational resources; therefore, the APF operational funds were amalgamated with other departmental funds. This approach resulted in resource allocations being based on short-term departmental priorities and led to significant financial confusion within the Department. This approach was followed until April 2007.

3.63 We noted that in the 2007–08 fiscal year, operating funds were reduced across the Department. Operating funds to deliver the environment programs were reduced by 10 percent. Analysis was not provided showing which programs should be reduced, and why, to determine priorities for the operational reductions.

3.64 The Department recognized that more accountability was needed. In the 2007–08 fiscal year, it set up its financial systems so that it could monitor the operational expenditures by program beginning April 2008.

3.65 The contribution portion of the APF Environment Chapter funding was kept separate from the operational funding and was monitored by program in the Department's financial system. Due to program implementation delays, the contribution funds were not used in the early years of the APF as planned. the Department used special provisions in place at the time to carry funds forward from one fiscal year to the next to avoid losing the funds that were not spent. Of the $400 million originally allocated to the contribution environment programs, Department records indicate that about $370 million was spent on these programs.

3.66 Recommendation. In order to ensure proper financial management, Agriculture and Agri-Food Canada should make certain that

  • it can demonstrate how operational resource allocations considered program objectives and results to be achieved, and
  • it can base operational budget reductions on analysis that considers program impacts.

The Department's response. Agriculture and Agri-Food Canada agrees. The Department has taken several steps in recent years to strengthen and improve its financial management, as reflected in the 2007–08 Treasury Board of Canada Secretariat Management Accountability Framework (MAF) assessment.

In the 2007–08 fiscal year, an improved business planning process was implemented that aligns information about departmental resources, priorities, and authorities from across different areas, resulting in complete and accurate data in support of financial and non-financial decision making and overall operational performance.

In the 2008–09 fiscal year, further steps were undertaken to ensure resource allocation and use—both grant and contribution and operational—are better tracked for each program.

Improvements have also been made in the area of financial budgeting and operational resource allocation. Multi-year, integrated planning and budgeting is under way, which considers program objectives and expected results and supports more longer-term, strategic decision making.

In addition, changes in governance have led to improved financial management accountability for the Department's branch heads over their respective financial resources.

Collectively, senior management is working to ensure that resources are directed to departmental priorities and that budget reductions and reallocations, if any, take into account program impacts.

Performance measurement and reporting

3.67 According to the Treasury Board of Canada Secretariat's 2000 policy and guidelines regarding transfer payments, departments must develop a Results-Based Management and Accountability Framework that provides for appropriate measuring and reporting of results.

3.68 We found that Agriculture and Agri-Food Canada's performance measurement and reporting framework is not fully developed. Improvements are required in the structure of its Results-Based Management and Accountability Framework, the type of information it reports, and the measurement of environmental change resulting from its programs.

The Department did not clarify how its programs improve the environment

3.69 The Department developed a performance measurement and reporting framework for the contribution programs of the Agricultural Policy Framework (APF) Environment Chapter. The performance framework's purpose was to identify how the programs' activities contributed to the achievement of the Environment Chapter's end outcomes using results-chain logic, a management principle that links program activities to desired results.

3.70 A key factor in an effective results chain is the clarity of the cause and effect relationships between each level. It should show how and which specific program activities link to specific deliverables (goods and services delivered), which in turn link to specific immediate and then intermediate outcomes (results), which link to a specific end outcome. In the Environment Chapter results chain, the Department grouped together all program activities. They lead to a group of deliverables, which lead to a group of immediate and intermediate outcomes. These in turn lead to the end outcome of improved stewardship by farmers of the soil, water, air, and biodiversity. The cause and effect relationship between each element of the results chain and how each program's activities relate to the end outcome is not clear.

3.71 We also found that the Department did not develop targets for each level. In particular, there were no targets at the intermediate level. This is relevant because this is the first level where amalgamated operational information can demonstrate what results the program has achieved.

3.72 We also noted that the Department did not monitor and report on program results beyond providing operational information, such as the number of completed water infrastructure projects (for example, wells and pipelines). Without more comprehensive information on what the programs are achieving, senior management cannot be certain whether programs are realizing their intended results or know where to focus program improvements. Furthermore, it limits the Department's ability to communicate a complete picture about the environment programs' performance to Parliament and stakeholders.

Impacts at the farm level need to be better identified

3.73 The Department's evaluation of the Environment Chapter indicated that there was insufficient data to link action occurring at the farm level through APF programs to positive environmental change. It identified the need to better understand the environmental and economic impacts of farming practices funded under the Environment Chapter, and the need to better understand the impacts of environmental farm plans (EFPs).

3.74 According to the Department records, as of March 2008, approximately 57,000 EFPs were completed and reviewed. This represents about 30 percent of the total farm population of over 200,000, and is close to the Department's target of 60,445 EFPs.

3.75 The evaluation also noted that, despite funding the EFP program, the Department does not have access to information contained in the plans. To encourage producer participation in the EFP program, the Department agreed to allow producers to keep the information confidential in response to address producers' concerns that disclosing their environmental risks could pose a liability to them, particularly if they did not act on those risks. However, the confidentiality of EFPs means that when a producer requests funding under the Environment Chapter, there is no way for the Department to know to what extent it will mitigate an on-farm environmental harm or whether it addresses one of the farm's priority risks. The confidentiality of EFPs also means that the Department cannot measure change in the implementation of the on-farm environmental action plan over time based on its EFP program. We note that other organizations, such as Statistics Canada, have developed a means to collect and report on statistics while maintaining confidentiality.

Environmental targets were set but the Department cannot yet measure their achievement

3.76 The Department measures the Environment Chapter's end outcome using six of the indicators in its National Agri-Environmental Health Analysis and Reporting Program (NAHARP). The six indicators are the following:

  • The percentage of farmland with various levels of residual nitrogen in the soil. Surplus nitrogen in the soil, such as from the use of fertilizer, poses a risk to the environment.
  • The percentage of farmland with various levels of risk of water contamination by nitrogen. Nitrogen in water contributes to algae growth and may pose a human health hazard.
  • The percentage of farmland at various levels of risk of water erosion. Water erosion contributes to large soil losses from farm fields and can affect water quality.
  • The levels of agricultural emissions of greenhouse gases. Greenhouse gases represent atmospheric pollution and contribute to climate change.
  • The change in levels of organic carbon in agricultural soil. Farming practices that remove carbon dioxide from the atmosphere and add it to the soil contribute to soil health and help reduce greenhouse gas emissions.
  • The availability of wildlife habitat on farmlands. Conservation of biodiversity depends on producers sustaining or enhancing the quality of wildlife habitats.

3.77 The APF implementation agreements between the federal and provincial/territorial governments include specific targets for each of the six NAHARP indicators. The targets were to be attained by 31 March 2008, the expected end of the APF. Example targets from the agreements include

  • a 21 percent reduction in the estimated average rate of water erosion on cropland, and
  • an 8 percent reduction in agricultural greenhouse gas emissions.

The commitment by the federal, provincial, and territorial governments to set goals and use NAHARP indicators to monitor and report on agri-environmental progress is a significant accomplishment.

3.78 However, the six NAHARP indicators do not measure all agricultural impacts. A number of negative agricultural impacts are not measured, such as the effects of pesticides, air pollution other than greenhouse gases, and wind erosion. The indicators do measure some benefits of agriculture on the environment, such as the sequestering of carbon in agricultural soils (removing carbon dioxide from the atmosphere), which helps reduce greenhouse gas emissions. But other benefits may not be measured.

3.79 Other than the six indicators identified by the Department to measure the Environment Chapter end outcomes, a broader set of 24 indicators is being developed under NAHARP to assess additional agricultural impacts.

3.80 We also noted that NAHARP indicators are reported only every five years, with a three- to four-year gap between the year of measurement and the year of reporting. The most recent information available is for 2001, which represents baseline data before the APF was established. For this reason, at the expected end of the APF, it was not possible to demonstrate the extent of environmental benefit of implementing the Environment Chapter, or the extent to which targets in the implementation agreements had been met. Departmental management is confident that the report on 2006 data, which will be available internally in 2008 and published in 2009, will show the impact of the APF programs.

Environmental benefits of programming are not measured

3.81 As previously mentioned, there are weaknesses in the Department's performance measurement strategy. There is a lack of clarity in the cause and effect relationships between each level of the Results-Based Management and Accountability Framework. As well, targets to assess progress have not been developed for each level. Departmental reporting focuses on immediate results rather than on intermediate and end outcomes. There is insufficient data to demonstrate that action at the farm level has led to positive environmental change. In addition, there is delayed reporting of environmental change under the NAHARP program; therefore, the impact of the APF environment programs on the environment is not yet known.

3.82 Despite the above, the beneficial management practices funded under the Department's environment programs are supported by science that indicates that these activities will likely lead to positive environmental change. For example, providing water to cattle away from streams will likely improve water quality. As well, a 2006 survey of producers indicated that those with an environmental farm plan were more likely to adopt certain farming practices, such as testing the soil to determine proper fertilizer application rates. We recognize the challenges in attributing environmental change to programming and the long-term nature of environmental change. The Department is in the process of developing a computer modelling system to better understand the effects of its programs on the environment.

3.83 Recommendation. In an effort to know and demonstrate whether the environment programs are achieving the intended results and to continuously improve, Agriculture and Agri-Food Canada should ensure that its Results-Based Management and Accountability Framework for the environment programs under Growing Forward is designed and fully implemented to monitor and improve program effectiveness. The framework should

  • provide a reasonable connection between each element of the results chain,
  • identify specific and measurable targets at all levels, and
  • collect and report outcome information as well as operational information.

In addition, the Department should identify and implement options for collecting more specific information on environmental farm plans to facilitate performance reporting.

The Department's response. Agriculture and Agri-Food Canada agrees. A new logic model for the Department's environment programs will be in place by 1 April 2009 to support program delivery. It will better define specific and measurable outcome targets at each outcome level.

The Department will also strengthen its practices relating to performance measurement in this area. It will link National Agri-Environmental Health Analysis and Reporting Program agri-environmental information to Canadian Regional Agriculture Model information to measure the impacts of certain beneficial management practices.

The Department will use environment program and other data to better understand the impacts of beneficial management practices and their linkages to environmental farm plans. By the end of the 2008–09 fiscal year, taking into consideration concerns about producer confidentiality, the Department will explore options to collect more specific information on EFPs.

The logic model and strengthened performance measurement strategy will be in place by April 2009.

National land and water information service

3.84 The National Land and Water Information Service (NLWIS) was to provide integrated, up-to-date, and consistent land use, soil, water, climate, and biodiversity resource information to Agriculture and Agri-Food Canada and land-use managers to enable them to make environmentally responsible land-use decisions. This type of information is needed to support program and policy decisions about the environment.

3.85 Because the total estimated cost of the NLWIS information technology (IT) project is $100 million over five and a half years, NLWIS was identified as a major Crown project. The APF contributed $70 million and the Department reallocated $30 million from existing ongoing activities. As of March 2008, $72 million had been spent on NLWIS on improving access to geographical information, maximizing the use of data collected by others, and improving data collection.

3.86 The project is being developed and implemented in four phases. By the end of phase two in October 2007, the project was about one year behind schedule. As a result of the delay, the timing of project phases was adjusted and the scope of the project has been scaled back.

3.87 We expected that NLWIS would be appropriately managed and developed in accordance with best IT practices.

3.88 We found that the project was poorly managed. Senior management recognized that corrective action needed to be taken and approved an internal audit and an independent review, both of which concluded in early 2008. The Department has taken steps that it expects will address the problems identified. However, it is too early to determine if the corrective measures will resolve the issues.

Project governance has been ineffective

3.89 The original NLWIS project charter clearly identified the roles and responsibilities of all the key players that were to provide oversight. Two senior Department committees were to take on this oversight role: the Environment Board and the Senior Project Committee.

3.90 We found that the Department's Environment Board was not actively involved in the project from the start. As well, the Senior Project Committee was not actively involved until late in 2007. A plan did exist to involve other external stakeholders on key committees and external advisory groups; however, it was never formally put in place.

3.91 This has led to a lack of a shared vision for the project. The NLWIS project team, the Department environment program officials, and the key external stakeholders all had different views of what the project was going to accomplish. This also led to a lack of clarity regarding roles and responsibilities.

3.92 Agriculture and Agri-Food Canada officials acknowledged that these issues needed to be addressed. They recognized that it was important to revisit the purpose of NLWIS and re-establish a vision and governance structure that would be shared among all key stakeholders.

3.93 In January 2008, a new governance structure was adopted, and at the end of March 2008, the vision was significantly restated in more general terms and some key aspects were removed. The Department expects that these changes will help to address the problems that existed during the first two project phases.

3.94 Recommendation. Agriculture and Agri-Food Canada should ensure that appropriate governance arrangements are established and function as intended for the remainder of the National Land and Water Information Service (NLWIS) project.

The Department's response. Agriculture and Agri-Food Canada agrees, and has already taken action to significantly strengthen NLWIS governance.

The conduct of an internal "Systems Under Development Audit" and a management-initiated review in early 2008 resulted in two significant improvements to NLWIS program governance: the naming of an Assistant Deputy Minister, Environment, who would work with the Chief Information Officer to oversee completion of the NLWIS project, and support from an Assistant Deputy Minister-level Steering Committee, which includes representatives from Treasury Board of Canada Secretariat.

Sound project management has been lacking

3.95 Insufficient details in business case. A business case explains the rationale for the project and the project results needed to meet an organization's business needs.

3.96 The NLWIS business case does not outline measurable and quantifiable key intended results to ensure project success. The service delivery model remains general and has not yet been approved. While the first project phase has specific deliverables, the remaining three phases were still described at a very broad level and were highly dependent upon consultations with key internal and external stakeholders, which have not materialized. Because of this it is not possible to assess whether the NLWIS project objectives were met and to determine the value and impact of what was not delivered.

3.97 Recommendation. Given the recent review and resulting refinements to the National Land and Water Information Service (NLWIS) project, Agriculture and Agri-Food Canada should ensure that the expected benefits of NLWIS are specific enough to be measured.

The Department's response. Agriculture and Agri-Food Canada agrees. In May 2008, the NLWIS Steering Committee approved the refined project scope plus specific deliverables for all phases, to a level of detail sufficient to measure outcomes achievement.

3.98 Inadequate organizational capacity for the project. In general, we found that the Department lacked the organizational capacity to manage a project of this size and complexity.

3.99 Although staffing gaps had been reported to senior management, up until recently the project was reported as on time and on budget. However, this was not the case. The latest quarterly report (covering October to December 2007) indicates that the inability to obtain sufficient skilled resources has led to delays and has affected scoping and the project priorities.

3.100 The Department expects that this area of risk will continue to affect the delivery of the remaining project deliverables, and the project management is significantly revising the project deliverables to minimize the impact.

3.101 Recommendation. Agriculture and Agri-Food Canada should be able to demonstrate that it has the capability to successfully complete the remainder of the National Land and Water Information Service (NLWIS) project in accordance with revised specifications.

The Department's response. Agriculture and Agri-Food Canada agrees and has already taken action as a result of an internal review and a systems-under-development audit, both concluded in early 2008.

A multi-disciplinary team with the skills and competencies for a project of this nature, supported by expert consultants, has been in place since early 2008 to ensure successful project completion.

Detailed resource planning has also been completed for the refined scope, which includes resource allocation and redirection as required to deliver on each scope element.

3.102 Weak monitoring and control processes. An organization uses project management to control and coordinate its activities, resources, time, and costs. We found that NLWIS had weak project management practices, which led to long delays, and a significant refining of the project scope.

3.103 Project management processes have varied over the last two and a half years because there have been five different project managers on this project. Most project managers lacked major Crown project experience. The frequent change in project managers has led to deliverables being late and project requirements being changed without the appropriate approvals. The Department has recently taken action to address this.

3.104 Department measures and reports the actual expenses against budgeted dollars. It does not measure what was delivered for the resources used. Also, the impact of deferred work is not reported.

3.105 Recommendation. Agriculture and Agri-Food Canada should ensure that the monitoring and reporting of the National Land and Water Information Service (NLWIS) project's progress includes a comparison of actual deliverables to expected deliverables and the impact of work not completed.

The Department's response. Agriculture and Agri-Food Canada agrees. In early 2008, as a result of a management-initiated review and a systems under development audit, project oversight was strengthened by the adoption of an earned value approach for tracking results and expenditures against plans and a risk monitoring risk mitigation strategy to ensure that expected outcomes and benefits are met.

An earned value approach report is reviewed, discussed, and approved at every NLWIS Steering Committee meeting.

Conclusion

3.106 The overall objective of the audit was to determine whether Agriculture and Agri-Food Canada could demonstrate that it was managing the Environment Chapter of the Agricultural Policy Framework (APF) to achieve its environmentally sustainable agriculture objectives. We examined four components of the APF Environment Chapter: the process for Growing Forward development, management of the contribution programs, the performance measurement and reporting framework, and project management of the National Land and Water Information Service. We did not examine program effectiveness or program payments made to delivery agents. We used four sub-objectives to assess the overall objective.

3.107 First, we looked at how the Department is developing the next generation of the APF, Growing Forward, to see if a lessons-learned approach was being used. We noted that the Department had carried out three formal reviews of APF, including an extensive stakeholder consultation process. These reviews produced valuable information for Growing Forward. Because the Department had experienced serious delays in negotiating and signing agreements with the provinces and territories when it developed the APF, based on "lessons learned," it should have anticipated that negotiation and consultation would have taken longer than expected for Growing Forward, and planned accordingly. We did not find evidence of a clear and timely transition plan to manage the risk of delays and to ensure consistent service delivery at the anticipated end of the APF. Many elements of Growing Forward policy development were not yet complete at the time of our audit. As a result, the extent to which our expectations regarding the development of the policy framework and associated programs were fulfilled could not be assessed.

3.108 Second, we examined whether the Department is effectively managing the contribution programs of the Environment Chapter. We found that despite a lengthy process to develop contribution agreements, the reporting requirements were unclear and there was limited evidence that information received was well used. The Department identified problems with data quality; however, progress to resolve the problems has been slow. In addition, we found that information about the operational resources used to deliver each program was not tracked until recently. However, contribution funds paid to provinces, delivery agents, and producers were appropriately tracked by program. While the Department recognized the problems it had in managing the contribution programs, and was taking corrective action, this action was not advanced enough to demonstrate that the Department was effectively managing these activities.

3.109 Third, we examined whether the Department has appropriate performance information to inform decision makers and to assess and report on the Environment Chapter's success and improve program delivery. We found that there are weaknesses in the Department's performance measurement and reporting strategy. There is a lack of clarity in the cause and effect relationships between each level of its Results-Based Management and Accountability Framework. As well, targets to assess progress have not yet been developed for each level of the Framework. Departmental reporting focuses on outputs rather than on intermediate and end outcomes. There is insufficient data to demonstrate that action at the farm level has led to positive environmental change. In addition, there is delayed reporting of environmental change under the NAHARP program, such that the impact of the APF environment programs on the environment is not yet known. Despite the above, the beneficial management practices funded under the Department's environment programs are supported by science that indicates that these activities will likely lead to positive environmental change. We also recognize the challenges in measuring environmental change, such as the long time frame required to measure trends and the difficulty attributing environmental change to programming. However, these weaknesses mean that the performance information is inadequate to assess and improve environmental programming, and the impact of the APF environment programs on the environment is not yet known.

3.110 Finally, we reviewed the development of the National Land and Water Information Service (NLWIS) to determine whether the information technology project was appropriately managed. The NLWIS was designed to support environmentally responsible land-based decisions. We found that this major Crown project was poorly managed. It fell behind schedule and had to be revised from original plans. Key decision makers were not involved to the degree expected for a project of this complexity. Participants had differing views of the project's goals, and project management suffered from a lack of expertise and continuity. Roles and responsibilities were unclear, deliverables were late, and the project requirements were changed without appropriate approvals. In recognition of these problems, senior management approved an internal audit and an independent review of NLWIS. During our audit, steps were being taken to address problems identified by the audit and review; however, it was too soon to determine if those actions would correct the identified problems.

3.111 Therefore, it is too soon to determine whether the Department is managing the Environment Chapter of the Agricultural Policy Framework to achieve its environmentally sustainable agriculture objectives. In summary, we found the following:

  • The Growing Forward policy and program development is ongoing. Many of the activities that we expected to review during the course of our audit were not yet complete.
  • The Department's performance measurement and reporting framework was not fully developed. As well, the impact of the APF environment programs on the environment has not yet been measured because National Agri-Environmental Health Analysis and Reporting Program indicators are reported every five years, with a three- to four-year gap between the year of measurement and the year of reporting.
  • The Department is taking action to address a number of deficiencies. For example, action is underway to improve the quality of data from delivery agents and to improve the terms and conditions in contribution agreements. The Department is also taking corrective measures with the NLWIS major Crown project. However, it is too early to say if these measures will resolve the problems.

About the Audit

Objectives

Our overall audit objective was to determine whether Agriculture and Agri-Food Canada (AAFC) could demonstrate that it was managing the Environment Chapter of the Agricultural Policy Framework to achieve its environmentally sustainable agriculture objectives.

There were four sub-objectives, which were to determine whether Agriculture and Agri-Food Canada

  • used a lessons-learned approach to develop the next generation of the Agricultural Policy Framework Growing Forward,
  • effectively managed the contribution programs of the Environment Chapter,
  • had appropriate performance information to inform decision makers and to assess and improve Environment Chapter program delivery, and
  • had appropriately managed the National Land and Water Information Service.

Scope and approach

The Agricultural Policy Framework (APF) represented AAFC's policy and program framework for a five-year period, which ended on 31 March 2008. The APF included the Environment Chapter plus four other chapters: Business Risk Management, Food Safety and Quality, Renewal, and Science and Innovation. We examined the development of the next generation of the APF, entitled Growing Forward, to ensure that a lessons-learned approach had been used.

The remainder of our audit focused solely on the APF's Environment Chapter. We looked at whether its five contribution programs were effectively managed. This included the management of resources for those programs and the mechanisms in place to ensure that delivery agents fulfilled the programs' key requirements. We also examined the performance indicators and standards put in place to measure the achievement of the environment programs' objectives, and to monitor and improve on program effectiveness.

In addition, we examined one specific program within the Environment Chapter: the National Land and Water Information Service (NLWIS). This key information system was intended to support the environmental programs by providing environmental information; decision tools; and improvements in national data collection, analysis, and reporting. Given the importance of this system, and the fact that it was a $100-million, major Crown project, we wanted to ensure that it was developed and managed in accordance with best practices.

In conducting this work, we did not audit the delivery of services by the provinces/territories and the delivery agents. However, we did examine the documents that they provided to AAFC to account for their performance.

We interviewed officials and reviewed documents at AAFC's headquarters in Ottawa and at its office in Regina (where most of the contribution programs are administered). We reviewed all 25 agreements that the Department has in place for the delivery of those programs, and we examined all associated files to determine whether key terms and conditions in the agreements were met.

Criteria

Listed below are the criteria that were used to conduct this audit and their sources.

Criteria Sources
Agricultural Policy Framework and its Successor

We expected the Department to have applied a lessons-learned approach in the renewal of the Agricultural Policy Framework.

Treasury Board of Canada Secretariat, Management Accountability Framework (MAF), Areas of Management (2005), Learning, Innovation and Change Management

We expected the Department to have reviewed opportunities to maximize integration and coherence across the chapters, in preparing for the next generation of the Agricultural Policy Framework.

Treasury Board of Canada Secretariat, Management Accountability Framework (MAF), Areas of Management (2005), Governance and Strategic Directions

We expected the Department to have reviewed opportunities to maximize integration and coherence among the programs within the Environment Chapter, in preparing for the next generation of the Agricultural Policy Framework.

Treasury Board of Canada Secretariat, Management Accountability Framework (MAF), Areas of Management (2005), Governance and Strategic Directions

Environment Chapter—Contribution Programs

We expected the Department to have based its management of financial resources and staff on sound planning, to contribute to effective program delivery of the Environment Chapter programs.

Treasury Board of Canada Secretariat, Management Accountability Framework (MAF), Areas of Management (2005), Governance and Strategic Directions

We expected the Department to have effectively managed the contribution programs of the Environment Chapter.

Treasury Board of Canada Secretariat, Policy on Transfer Payments (June 2000), Section 7

Performance Measurement and Reporting

We expected the Department to have identified performance indicators and standards to measure the achievement of the environment programs' objectives, and to have systems in place to monitor, report on, and improve program effectiveness.

Treasury Board of Canada Secretariat, Preparing and Using Results-Based Management and Accountability Frameworks (February 2005), Section 4

National Land and Water Information Services

We expected the Department to have developed and managed the National Land and Water Information Service in accordance with best information technology practices.

  • Project Management Institute, A Guide to the Project Management Body of Knowledge and Government Extension to the PMBOK Guide (2004), sections 4–12
  • Information Systems Audit and Control Association, Control Objectives for Information and Related Technology (2004), Section A

Audit work completed

Audit work for this chapter was substantially completed on 30 April 2008.

Audit team

Assistant Auditor General: Neil Maxwell
Principals: Katherine Rossetti, Frances Taylor
Director: Raymond Kunze

Bernard Battistin
Joanne Butler
Alina Dan
Melissa Miller

For information, please contact Communications at 613-995-3708 or 1-888-761-5953 (toll-free).

Appendix—List of recommendations

The following is a list of recommendations found in Chapter 3. The number in front of the recommendation indicates the paragraph number where it appears in the Chapter. The numbers in parentheses indicate the paragraph numbers where the topic is discussed.

Recommendation Response
Agricultural Policy Framework and its Successor

3.35 In order to ensure that all stakeholders are well informed and to minimize program uptake delays, Agriculture and Agri-Food Canada should develop a clear and timely transition strategy well in advance of the end of its programs. Once Growing Forward is completed, Agriculture and Agri-Food Canada should prepare a schedule of review and consultations through the life of the new framework to ensure a smooth and timely transition to future policy frameworks. (3.28–3.34)

Agriculture and Agri-Food Canada agrees. The Department is committed to working with provinces and territories to achieve smooth and timely transitions between policy frameworks and programs. These efforts involve regular stakeholder consultations and reviews, which include assessments of strengths and weaknesses of past approaches.

The Department will discuss the process of review and consultation with provinces/territories through the life of the framework, with a view to ensuring timely renewal. The Department recognizes the importance of beginning these processes early, because of the complexity of federal-provincial/territorial negotiations in this area of shared jurisdiction.

Environment Chapter—Contribution Programs

3.47 When developing agreement terms and conditions, Agriculture and Agri-Food Canada should determine what information it wants, why it wants the information, and how it intends to use it, and develop and communicate a plan for using the information. This information should be clearly and consistently expressed in the agreements' terms and conditions. The Department should monitor and ensure it obtains the requested information. (3.43–3.46)

Agriculture and Agri-Food Canada agrees. Consistent with the Treasury Board of Canada Secretariat's Policy on Transfer Payments and recommendations of the Independent Blue Ribbon Panel on Grant and Contribution Programs, the Department will make its information requirements more clear in new environment program terms and conditions and contribution agreements with recipients, and it will provide information in these documents about why the information is required and how it will be used.

The Department will also ensure consistency of information requirements between different parts of the country and different recipients. The Department will ensure it receives and reviews the required information, and it will limit requirements to information that is needed. As well, it will regularly analyze the information it receives and use this analysis to report on progress toward objectives.

The Department will have most new terms and conditions in place by April 1, 2009; the others will be put in place, in consultation with provincial and territorial partners, by March 31, 2010.

3.57 Agriculture and Agri-Food Canada should develop and implement a standardized process for collecting national data to ensure that it can report accurate, consistent, and timely information on program results. (3.51–3.56)

Agriculture and Agri-Food Canada agrees. It will take steps, by December 2009, to strengthen its performance measurement practices for environment programs. These will include improving the systematic collection and analysis of program performance information to support accurate performance reporting, and meeting with delivery agents to clarify program data standards in order to improve data reliability and reporting requirements.

An interim database has been created by the Department to electronically store, analyze, and report program information received from delivery agents.

3.66 In order to ensure proper financial management, Agriculture and Agri-Food Canada should make certain that

  • it can demonstrate how operational resource allocations considered program objectives and results to be achieved, and
  • it can base operational budget reductions on analysis that considers program impacts. (3.58–3.65)

Agriculture and Agri-Food Canada agrees. The Department has taken several steps in recent years to strengthen and improve its financial management, as reflected in the 2007–08 Treasury Board of Canada Secretariat Management Accountability Framework (MAF) assessment.

In the 2007–08 fiscal year, an improved business planning process was implemented that aligns information about departmental resources, priorities, and authorities from across different areas, resulting in complete and accurate data in support of financial and non-financial decision making and overall operational performance.

In the 2008–09 fiscal year, further steps were undertaken to ensure resource allocation and use—both grant and contribution and operational—are better tracked for each program.

Improvements have also been made in the area of financial budgeting and operational resource allocation. Multi-year, integrated planning and budgeting is under way, which considers program objectives and expected results and supports more longer-term, strategic decision making.

In addition, changes in governance have led to improved financial management accountability for the Department's branch heads over their respective financial resources.

Collectively, senior management is working to ensure that resources are directed to departmental priorities and that budget reductions and reallocations, if any, take into account program impacts.

Performance Measurement and Reporting

3.83 In an effort to know and demonstrate whether the environment programs are achieving the intended results and to continuously improve, Agriculture and Agri-Food Canada should ensure that its Results-Based Management and Accountability Framework for the environment programs under Growing Forward is designed and fully implemented to monitor and improve program effectiveness. The framework should

  • provide a reasonable connection between each element of the results chain,
  • identify specific and measurable targets at all levels, and
  • collect and report outcome information as well as operational information.

In addition, the Department should identify and implement options for collecting more specific information on environmental farm plans to facilitate performance reporting. (3.67–3.82)

Agriculture and Agri-Food Canada agrees. A new logic model for the Department's environment programs will be in place by 1 April 2009 to support program delivery. It will better define specific and measurable outcome targets at each outcome level.

The Department will also strengthen its practices relating to performance measurement in this area. It will link National Agri-Environmental Health Analysis and Reporting Program agri-environmental information to Canadian Regional Agriculture Model information to measure the impacts of certain beneficial management practices.

The Department will use environment program and other data to better understand the impacts of beneficial management practices and their linkages to environmental farm plans. By the end of the 2008–09 fiscal year, taking into consideration concerns about producer confidentiality, the Department will explore options to collect more specific information on EFPs.

The logic model and strengthened performance measurement strategy will be in place by April 2009.

National Land and Water Information Service

3.94 Agriculture and Agri-Food Canada should ensure that appropriate governance arrangements are established and function as intended for the remainder of the National Land and Water Information Service (NLWIS) project. (3.89–3.93)

Agriculture and Agri-Food Canada agrees, and has already taken action to significantly strengthen NLWIS governance.

The conduct of an internal "Systems Under Development Audit" and a management-initiated review in early 2008 resulted in two significant improvements to NLWIS program governance: the naming of an Assistant Deputy Minister, Environment, who would work with the Chief Information Officer to oversee completion of the NLWIS project, and support from an Assistant Deputy Minister-level Steering Committee, which includes representatives from Treasury Board of Canada Secretariat.

3.97 Given the recent review and resulting refinements to the National Land and Water Information Service (NLWIS) project, Agriculture and Agri-Food Canada should ensure that the expected benefits of NLWIS are specific enough to be measured. (3.95–3.96)

Agriculture and Agri-Food Canada agrees. In May 2008, the NLWIS Steering Committee approved the refined project scope plus specific deliverables for all phases, to a level of detail sufficient to measure outcomes achievement.

3.101 Agriculture and Agri-Food Canada should be able to demonstrate that it has the capability to successfully complete the remainder of the National Land and Water Information Service (NLWIS) project in accordance with revised specifications. (3.98–3.100)

Agriculture and Agri-Food Canada agrees and has already taken action as a result of an internal review and a systems-under-development audit, both concluded in early 2008.

A multi-disciplinary team with the skills and competencies for a project of this nature, supported by expert consultants, has been in place since early 2008 to ensure successful project completion.

Detailed resource planning has also been completed for the refined scope, which includes resource allocation and redirection as required to deliver on each scope element.

3.105 Agriculture and Agri-Food Canada should ensure that the monitoring and reporting of the National Land and Water Information Service (NLWIS) project's progress includes a comparison of actual deliverables to expected deliverables and the impact of work not completed. (3.102–3.104)

Agriculture and Agri-Food Canada agrees. In early 2008, as a result of a management-initiated review and a systems under development audit, project oversight was strengthened by the adoption of an earned value approach for tracking results and expenditures against plans and a risk monitoring risk mitigation strategy to ensure that expected outcomes and benefits are met.

An earned value approach report is reviewed, discussed, and approved at every NLWIS Steering Committee meeting.

 

PDF Versions

To access the Portable Document Format (PDF) version you must have a PDF reader installed. If you do not already have such a reader, there are numerous PDF readers available for free download or for purchase on the Internet: