This Web page has been archived on the Web.

2000 October Report of the Auditor General of Canada

Main Points

14.1 In CIDA's Geographic programs, contracts and contribution agreements with Canadian executing agencies are used to implement development projects. Where CIDA used a competitive process for selecting executing agencies, with some exceptions the process was properly conducted. However, we observed instances where contracts did not comply with the Treasury Board Contracting Policy or the Government Contracts Regulations.

14.2 An authority framework, similar to that for the contracting process, is not in place for CIDA's contribution agreements. The terms and conditions for grants and contributions related to the Geographic programs are very general and provide no direction on how and when to use contribution agreements. They include a provision that contributions are to be approved in accordance with regular departmental procedures and authorities; however, exceptions can be dealt with internally by CIDA. Consequently, CIDA's use of contribution agreements to select executing agencies often varied from its stated internal policies or practices. CIDA can select executing agencies by means of contribution agreements, which are, in effect, the same as sole-source contracts that would not be permitted under Government Contracts Regulations. This was the case in about half of the contribution agreements we examined.

14.3 CIDA project officers attach considerable importance to monitoring the agreements under their responsibility. They commonly use monitors under contract to review and report on progress, and they insist on receiving reports from Canadian executing agencies as required.

14.4 For the Voluntary Sector program, the Canadian Partnership Branch obtains reasonable information on the financial health of its Canada-based partners. However, limited information is received on projects that were funded, on the amounts spent on them, and on results obtained. CIDA bases its funding primarily on historical levels rather than on partners' performance. More meaningful and accurate information on the Canadian Partnership Branch is needed in CIDA's Performance Report to Parliament.

Background and other observations

14.5 CIDA is responsible for managing about $1.8 billion of Canada's international assistance. Approximately $700 million of that amount goes to Geographic Branches for programs aimed at countries in Asia, Africa and Latin America. Most of this amount is spent through contracts and contribution agreements with third parties, referred to as executing agencies, to deliver development assistance projects. Another $260 million goes to the Canadian Partnership Branch for grants and contributions to organizations to carry out their own aid programs.

14.6 In 1998, our Office published a report that commented on results-based management related to Geographic programs. We concluded that progress in managing for results was evident but uneven.

14.7 This audit focussed on how CIDA's Geographic Branches manage contracting and other types of agreements for goods and services, including the selection of Canadian executing agencies to deliver projects. It also examined the control framework for agreements in the Canadian Partnership Branch's Voluntary Sector program. The audit aimed to assess whether CIDA's contracting/contribution agreement processes respect Government Contracts Regulations, Treasury Board guidelines and its own policies; whether they are fair and transparent; and whether they meet operational requirements and development needs.

The Agency's responses to our recommendations are included in this chapter. The Agency accepts the recommendations and indicates the actions that it is taking or intends to take to address them.

Introduction

CIDA uses contracts, contribution agreements and grants to deliver its development assistance

14.8 Nearly all of CIDA's budget for 1999-2000 ($1.7 billion or about 93 percent) is allocated to development assistance programs. The planned spending by program for this period is shown in Exhibit 14.1. Operating and capital expenditures of $131 million (about seven percent) make up the rest of the budget. CIDA's program expenditures of $1.7 billion are made in the form of grants, contributions and other transfer payments. These payments are made to Canadian and developing country institutions, provincial governments and their organizations and agencies, and Canadian private-sector firms, for specific development projects, programs and activities. Exhibit 14.2 shows the distribution of agreements by dollar value in the Geographic Branches, excluding food aid.

14.9 The Geographic Branches carry out Geographic programs, also referred to as bilateral or country-to-country programs; these programs account for about 40 percent of CIDA' s program expenditures. The Geographic Branches also carry out development co-operation initiatives, mostly through Canadian executing agencies, directly with countries eligible for Canadian aid. The projects aim to reflect both the needs of developing countries and Canada's ability to meet those needs.

14.10 CIDA' s Canadian Partnership Branch promotes partnerships between developing countries and organizations based in Canada and internationally, to support sustainable development and reduce poverty in the developing world. The Branch provides grants and contributions to support partner organizations that are responsible for the design, planning and implementation of the development programs and projects.

Focus of the audit

14.11 The objectives of our audit were to assess whether CIDA's contracting/contribution processes:

  • respect Government Contracts Regulations, Treasury Board guidelines and its own policies;
  • are fair and transparent; and
  • meet operational requirements and development needs.

14.12 The audit focussed on how CIDA's Geographic Branches manage contracting and other types of agreements for goods and services, including the selection of Canadian executing agencies to deliver projects.

14.13 The audit also examined the control framework for program funding agreements in the Voluntary Sector program of the Canadian Partnership Branch. This program supports programs and projects planned, designed and implemented by the voluntary sector.

14.14 Sampling plan. In the Geographic Branches, agreements with a value of over $100,000 account for over 90 percent of the total value of all agreements. We selected a random sample of 33 agreements over $100,000, with a total value of $214 million, in six chosen countries in the three-year period 1996-97 to 1998-99. The selection of countries reflected the dollar importance of each region and our discussions with CIDA. The six countries chosen were Peru, China, India, Cameroon, Egypt and Mali. The sample included six agreements over $10 million and three food aid contracts. In order to ensure the currency of our findings, we also chose a small random sample of eight projects from the 1999-2000 year for a file review. In addition, we selected a random sample of seven partnership agreements, with at least one project in one of the six countries. Further details on the audit are in About the Audit at the end of the chapter.

Policy Framework

CIDA authorities for contracts and contribution agreements

14.15 The awarding of contracts for both aid and corporate services is subject to Government Contracts Regulations (GCRs) and Treasury Board Contracting Policy. CIDA's contribution agreements are not subject to the GCRs and are governed by the terms and conditions for grants and contributions related to the Geographic programs approved in March 1996, and by the terms and conditions for the Canadian Partnership Branch approved in March 1995.

14.16 The Geographic Branches' Geographic Programs Roadmap sets out the monetary approval authority limits for both contracts and contribution agreements. The level of approval required for a project depends on its dollar value. Projects between $500,000 and $5 million require approval by a vice-president; projects between $5 million and $15 million need the Minister's approval; and projects over $15 million must be approved by the Treasury Board. Partnership grants and contributions are based on cost sharing with partners. The limits for partnership agreements are $5 million per year up to $15 million under multi-year funding agreements.

CIDA's policy framework for contracts and contribution agreements in the Geographic Branches

14.17 Policies in effect before 1997 relating to competitive contracts. Prior to 1997, there were two phases to the competitive contracting process. The first phase involved an invitation to prequalify. CIDA put its request on the open bidding system and invited bidders to make simple, short proposals that allowed CIDA to identify those who met the requirements. The proposals were to include the bidder's experience and achievements, key personnel, and management/financial capability. There was no costing done at this phase of the process. CIDA then established a review committee to evaluate the proposals and retain those that met the standards. The committee submitted the list to the Minister, who selected three to five candidates for prequalification.

14.18 In the next phase of the process, the prequalified candidates were requested to submit detailed proposals. These proposals included a detailed costing and total bid price. The bid evaluation team examined the bids and prepared an evaluation report. An Evaluation Review Board reviewed the report and made a final recommendation to the Minister. Following this process, CIDA negotiated the contract terms and final amount with the successful bidder.

14.19 Policies and practices prior to 1997 relating to contribution agreements. Prior to 1997, CIDA's policy called for the use of contribution agreements for "country focus" projects delivered by not-for-profit organizations. One of the distinctive features of the country focus approach was that in most cases it was to be used for projects that were initiated, developed and approved by the not-for-profit organization implementing them. The not-for-profit organization was not hired to implement a project for CIDA but received support for a project of its own.

14.20 By 1996, it had become common practice for CIDA, when engaging a Canadian executing agency for a project, to use contribution agreements whenever dealing with not-for-profit organizations, even for CIDA-initiated projects. There was no requirement that contribution agreements be tendered competitively and they could provide for reimbursements of up to 100 percent of costs. Once awarded, contribution agreements are managed by CIDA in much the same way as contracts.

14.21 Changes as of 1997. On 10 December 1996, CIDA's Minister announced the implementation of an Open Competition Pilot Project for all agreements over $100,000 in the Geographic Branches. This initiative was designed to allow both for-profit and not-for-profit organizations equal access to service contracts over $100,000 as well as to the Bilateral Responsive Mechanism for unsolicited proposals. This meant that not-for-profit organizations could compete for service contracts and that for-profit organizations could enter into contribution agreements with CIDA. The Agency emphasized that a competitive contracting environment is to be the chief mechanism for delivering the assistance program. Under this initiative, contribution agreements would be used for unsolicited proposals through the Bilateral Responsive Mechanism, which does not require agreements to be open to competition.

14.22 CIDA issued guidance on the awarding of contribution agreements for unsolicited proposals. The guidance makes clear that, under the Bilateral Responsive Mechanism, contribution agreements are to be used as a vehicle for engaging Canadian executing agencies only when the project was initiated by the proposing entity. The guidance states that in order to help differentiate sole-source "responsive" projects from "directed" projects, it is important that responsive projects:

  • be unsolicited proposals from eligible development partners;
  • be implemented under a contribution agreement between the implementing organization and CIDA; and
  • require that project approval and the selection of the implementing organization be inseparable.

14.23 Once CIDA decides that a project qualifies as responsive, it can then determine whether and to what extent it wishes to contribute toward furtherance of the project. No profit is allowed to the project proponent.

14.24 For competitive contracts, the prequalification phase of the selection process was discontinued.

14.25 In December 1997, CIDA incorporated these changes into its Geographic Programs Roadmap, which provides the policy and regulatory context for the conduct of the Bilateral Aid Program. In November 1999, CIDA issued a draft Contracting Guide for Managers in CIDA. The final version of the Guide was issued officially in May 2000. This guide contains CIDA's key contracting policies and reinforces the direction taken by the Open Competition Pilot Project toward increasing competition.

CIDA's policy framework for grant and contribution agreements in the Canadian Partnership Branch

14.26 The 1998-2001 Branch Plan of the Canadian Partnership Branch (CPB) outlines the policy framework within which the Branch operates. Key features of this framework are the following:

  • Responsiveness. CPB's Canadian partners are responsible for the planning and implementation of their own policies, programs and projects in collaboration with their developing country partners. Responsibility for and "ownership" of the programs and projects supported through CPB belong to the Canadian or international partner and its partner overseas. These programs and projects are not CIDA programs or projects.
  • Cost-sharing. All Canadian and developing country partner organizations are required to contribute resources to their development initiatives.
  • Focus at the institutional level. The Canadian Partnership Branch ensures the eligibility of the organizations receiving support and assesses the extent to which their programs reflect CIDA's programming priorities and objectives. The Branch is also accountable for assessing the financial, managerial and technical capabilities of partner organizations, and for verifying the results achieved.

Observations and Recommendations

Planning and Design

Contract plans were incomplete

14.27 Under CIDA policy, a contract plan must be prepared early in the project design stage. This plan sets the stage for all contracts or other arrangements needed to support the project. Among other items, the plan is to identify the selection method (competitive or non-competitive) and the type of executing agency (not-for-profit, private sector or other). We found that of the 12 non-competitive agreements that we examined where a contract plan was required, 8 did not address either the selection method or the type of executing agency to be used.

Unclear or unrealistic expectations affect contract performance

14.28 CIDA's Policy Statement on Results-Based Management specifically calls for "defining realistic expected results, based on appropriate analyses." We found that 15 of the 33 agreements in our sample had unrealistic or unclear expectations of what was to be achieved. In all but one case, the projects had dropped one or more of the major expected results or were significantly behind schedule. In the one case, the project team redefined an unclear objective and created clearer expectations, which were being met. In the other cases where the projects were behind schedule, we found no sound analysis (for example, critical path) of how the time could be made up (see Exhibit 14.3).

14.29 For the contracts in our sample, the project approval documents, the statements of work and the requests for proposal were consistent with each other. The statements of work were generally clear enough to permit CIDA to include in contracts the outputs that Canadian executing agencies were expected to deliver, and to enable CIDA to monitor project progress, notwithstanding whether these expected outputs were realistic.

14.30 One of the perceived advantages of tendering contracts for competition is that the process should lead to a tighter and better-understood description of the job to be carried out. We looked for reasons why so many contracts let competitively still had unrealistic expectations of results. We believe the main reasons are the following:

  • Many projects are planned in considerable detail by CIDA staff, using outside consultants to assess feasibility. The consultants who assess the feasibility do not generally bid on the contract, and the time frames between the feasibility assessment and contract implementation can be measured in years. Competition for funding encourages CIDA staff, consultants and developing country officials to present anticipated project results as optimistically as possible, even if experience suggests that more modest expectations would be realistic.
  • Bidders are required to bid on the project as described in the request for proposal. They are given points in the evaluation for their understanding of the project. During our audit, bidders told us that they could not consider making a bid that would indicate that project expectations are unrealistic, as they would then stand no chance of winning.
  • CIDA's evaluation teams for tendered bids generally include the project team leader, who is the officer proposing the project and who has usually directed the design phase, and one or more technical specialists who understand the subject area. A contracting officer assists the team to ensure compliance with contracting policy. The evaluation team focusses on selecting the bidder with the best approach and experience, rather than on questioning whether the expected results are realistic.

14.31 As part of the project approval process, CIDA should review more rigorously whether the expected results that have been established for projects are realistic.

Agency's response: CIDA will place greater emphasis, as part of the project approval process, on ensuring that the established expected results for projects are realistic.

Provisions for sustainability need to be built into agreements more consistently

14.32 In our 1993 chapter on Bilateral (Geographic) Programs, we outlined certain conditions for sustainability of benefits. CIDA incorporated this thinking into its results-based management framework (which we published in our 1996 follow-up chapter) by setting out four conditions for ensuring the sustainability of benefits:

  • Stakeholders take charge of project activities.
  • There is commitment of sufficient financial resources to maintain project benefits, where applicable.
  • Institutional capacity and ongoing relevance are adequate to maintain project benefits.
  • The national and international environments are conducive to maintaining project benefits. (See photograph)

14.33 During the audit, we examined contracts and contribution agreements with Canadian executing agencies to determine if they included any provision relating to conditions that would make sustainability of benefits more likely. There were 25 agreements where provisions relating to sustainability could have been considered. Of these agreements, 12 called for one or more of the conditions for sustainability to be addressed, while the other 13 did not. The following two cases provide examples; the first addressed sustainability and the second did not.

14.34 An $18 million contract for an environment development project in Egypt required the Canadian executing agency to spend the last two years of the project putting in place a transition strategy to turn over management and administrative responsibilities to local organizations.

14.35 A micro-finance project in Peru was to transfer from the Canadian executing agency to members of the credit union system knowledge and skills in the areas of marketing, information systems, banking procedures, management and audit. The agreement contained a training module but did not require testing of the credit union staff's capability before handover of the project. The agreement could also have required the Canadian executing agency to identify and develop the leadership skills of an organization to ensure that the knowledge and skills acquired by the member credit unions are maintained and further developed. This would have supported stakeholders in taking charge of project activities. A revised implementation plan has proposed to introduce an institution-strengthening component. (See photograph)

14.36 Although clauses in agreements with executing agencies are not the only way to help ensure that conditions for sustainability will exist, CIDA needs to continue to look at this option in its planning of agreements.

CIDA did not always comply with the Canadian Environmental Assessment Act

14.37 Under the Canadian Environmental Assessment Act (CEAA), CIDA must examine proposed projects to determine the environmental effects of the project and make a decision on their significance. For each project, CIDA requires a preliminary environmental assessment form to be completed in the design stage, to establish whether the Act applies to the proposed project and whether an environmental assessment is required.

14.38 For projects that undergo an assessment, a completed environmental screening report must be filed in the public registry along with other project-related documents. The public registry is a public record of documentation related to any environmental assessments conducted for the projects. Neglecting to file the required documents in the public registry denies the public its legislated right to participate in the environmental assessment process.

14.39 An internal audit found that only about five percent of new CIDA initiatives require an environmental assessment under the Act. The number of applicable cases is relatively small, but in 41 percent of those cases, the audit concluded that CIDA was not complying with the requirement to conduct an environmental assessment prior to a funding decision.

14.40 Of the projects we examined, two required environmental assessments. Only one was found in the public registry and it had been filed late. The project included the construction of health centres in China. The original preliminary environmental assessment form indicated that the CEAA did not apply. However, in this instance, the Act did apply to the construction activities. CIDA discovered its oversight some time later, but the host country partner began construction before the required assessments were completed. After reviewing those assessments, CIDA required ongoing monitoring of the facilities for potential adverse environmental impacts. Although the monitoring was done, CIDA did not file the monitoring reports in the public registry as required. The assessment of the other project was not found in the public registry but did not raise any environmental concerns. (See photograph)

14.41 The Mali Mortgage Financing project (see case study 1) is a case where the original project plan correctly indicated that the CEAA did not apply. However, as the project progressed, the Canadian executing agency's activities were expanded to include co-ordinating housing development to stimulate demand for mortgages. This made necessary a new preliminary environmental assessment, as housing development typically triggers the application of the Act. Following the approval of the new activities, no review of the project was conducted to determine if the Act applied.

14.42 Environmental assessments serve to bring forward environmental considerations in the project planning and design stages. Conducting the assessment after the start-up of a project is almost equivalent to not conducting one at all. CIDA's Executive Committee approved new environmental assessment directives and agreed that the Agency strive for full compliance with the CEAA by 1 July 2000. An Environmental Assessment Co-ordinating Committee was established to assist with this effort.

Selection

Contribution agreements were used instead of competitive contracts

14.43 In our 1999 chapter on sole-source contracting and the use of advance contract award notices (ACANs), we stated:

Contracting is an essential tool for federal departments and agencies in delivering programs. Central to government contracting are the principles of best value and open access to contracting opportunities. "Best value" is the best combination of value and price that the government can obtain in acquiring goods or services for the Crown. "Open access" is a fair chance for all qualified vendors to do business with the Crown without political or bureaucratic favour. An open, competitive bidding process provides the best guarantee that both of these principles will be respected. The Government Contracts Regulations state that competition is the norm, and departments are to solicit bids before entering into a contract.

14.44 Over the years, CIDA has subscribed to this principle for contracts but not for contribution agreements. Until 1997, CIDA traditionally used the non-competitive vehicle of contribution agreements when choosing a not-for-profit organization, and competitive contracts when choosing a for-profit organization as a Canadian executing agency. Exhibit 14.4 shows the percentages, in terms of dollar value, of competitive and non-competitive agreements over $100,000 signed in the Geographic Branches from 1996-97 to 1999-2000. These include goods and services contracts and contribution agreements.

14.45 With the advent of the Open Competition Pilot Project (OCPP) that the Minister announced in early December 1996, CIDA indicated that contribution agreements would be used for unsolicited proposals. Not-for-profit organizations would henceforth be encouraged to compete with for-profit organizations for contracts to deliver CIDA aid, and for-profit organizations would be eligible to receive contributions, on a non-competitive basis.

14.46 The contracts and contribution agreements that we selected were from those signed in the years 1996-97 to 1998-99. The contribution agreements dated beyond 1997 should have followed the procedures of the Bilateral Responsive Mechanism unless the selection process had already begun under the previous policy framework. Those agreements whose selection process had begun before 1997 should have followed the country focus approach. In both time periods, contribution agreements were to be used for what CIDA now terms unsolicited proposals.

14.47 We concluded from our examination that 6 contribution agreements out of 11 that we examined were initiated by CIDA, rather than the not-for-profit organization. Consequently, a competitive process should have been considered for the selection of the executing agency. Case study 1 provides an example where competition should have been considered but was not. The following case is another example.

14.48 In May and December 1996, CIDA awarded two contribution agreements to a Canadian not-for-profit organization to design and implement a health project in China. The total value of the agreements was $5.4 million. The government of China had requested Canadian assistance in May 1995 to carry out this project. CIDA undertook a feasibility mission in February 1996. The consultants who carried out the mission identified the necessary experience and expertise that would be required of a Canadian executing agency. In making its selection, CIDA decided to sole-source the design and implementation of the project through contribution agreements, even though the project was initiated by CIDA and the selection approval memorandum noted that more than one supplier was available.

14.49 We noted that CIDA did hold a competition to select an executing agency for each of two contribution agreements in our sample. The requests for proposal were sent to not-for-profit organizations. Both projects were awarded before the Bilateral Responsive Mechanism was adopted in 1997. Although in accordance with the country focus approach, these were not unsolicited proposals, the process was in the spirit of greater fairness and transparency.

14.50 We noted another instance where two very similar projects in two countries were treated quite differently under similar circumstances (see Exhibit 14.5).

14.51 CIDA conducted a performance review of the Open Competition Pilot Project in early 1999. One of the main conclusions from the performance review was that the restriction to the use of contribution agreements for unsolicited proposals was not well understood or applied. Among the weaknesses that CIDA staff identified for the reviewers were the lack of clear distinction between directed sourcing and unsolicited proposals, the lack of clear directives on the extent of the involvement allowed for project officers to assist or give advice on proposals, and the need to discuss and resolve some remaining ambiguities associated with "solicited unsolicited proposals."

14.52 In our selected sample of eight contracts and contribution agreements from 1999-2000, we looked to see if the 1997 policy direction was being consistently followed. Of three contribution agreements classified as unsolicited proposals, in our view two should have been opened to competition because they did not meet the conditions of the Bilateral Responsive Mechanism. The "unsolicited" criterion for contributions is still not completely understood at CIDA.

14.53 Framework for contracts and contribution agreements. Contracting activities take place within a well-established management framework with underlying principles of best value and open access. Activities are governed by the Government Contracts Regulations, and competition is the norm. Exceptions are permitted in certain limited circumstances and must be fully justified on the contract file or in a submission to the Treasury Board. However, no similar framework at the level of GCRs exists for entering into contribution agreements for CIDA projects, even though these are virtually identical to contracts.

14.54 CIDA's authority to use contribution agreements to implement its projects is governed by the terms and conditions approved by the Treasury Board in March 1996. The terms and conditions are very general and provide no direction on how or when to use contribution agreements. Direction is provided in several documents referred to in CIDA's Framework Policy for Bilateral Aid. The terms and conditions also include a provision requiring that contributions are to be reviewed and approved in accordance with regular departmental procedures and authorities. However, exceptions can be dealt with internally. Consequently, when CIDA wishes to make an exception to its own policies on the use of contribution agreements, it is able to do so with no requirement to justify the exception. We observed several instances where, in effect, CIDA's use of a contribution agreement to select an executing agency was the same as using a sole-source contract - which would not have been permitted under the Government Contracts Regulations.

14.55 CIDA's terms and conditions for grants and contributions related to the Geographic programs are due for renewal in March 2001. This will provide an opportunity to establish a clearer authority framework for the use of contribution agreements by the Geographic Branches to fund projects. Projects that do not meet the description of unsolicited proposals as described in CIDA's current policy for the Bilateral Responsive Mechanism should be tendered for competition.

14.56 When CIDA seeks Treasury Board approval for renewal of its terms and conditions, the Agency should include a framework that makes it clear how and when contribution agreements will be used in the Geographic programs.

Agency's response: CIDA will continue to use contribution agreements under the terms and conditions for grants and contributions related to the Geographic programs approved by the Treasury Board. However, it will introduce a more structured framework to better guide the use of contribution agreements, especially those that are not for unsolicited proposals. CIDA will enhance its use of contracts to better reflect this as one of the key mechanisms for the delivery of its aid programs. The two mechanisms are intended to complement each other.

14.57 A separate group within CIDA should review all proposed non-competitive contracts and contribution agreements over $100,000 for compliance with authorities and CIDA policy.

Agency's response: CIDA will expand the mandate of its separate group in the branches to cover the review of contract plans over $100,000 for compliance with authorities and CIDA policy.

14.58 There were also two directed contracts out of our sample of 15 contracts that, according to Government Contracts Regulations, should have been competitively tendered but were not. Exhibit 14.6 provides one example.

14.59 Regional distribution of contracts and contribution agreements. We obtained data from CIDA on the distribution, by province, of service contracts and contribution agreements over $100,000 in the Geographic Branches for the years 1997-98 and 1998-99. Exhibit 14.7 shows the regional distribution of service contracts for those years. Exhibit 14.8 shows the regional distribution of contribution agreements in the Geographic Branches for the same period.

14.60 CIDA has made efforts to make its selection process for competitive contracts fairer and more transparent. A key step was the move from a two-step process of prequalification and tendering to a one-step tendering process for competitive agreements. This new tendering process introduced clear policies on the necessity for competition. Upcoming projects and competitions are advertised on the government open bidding system. CIDA put a regional officer in Industry Canada's Vancouver office to encourage more Western participation and increase the pool of qualified contractors. A similar position is planned for Atlantic Canada. CIDA contracting personnel have also given presentations across the country to potential contractors.

Some errors occurred in the selection process for competitive agreements

14.61 Out of the 15 agreements in our sample that had used a competitive process, we found two with errors that had occurred in the selection process.

14.62 In one case, a contribution agreement for $4.1 million to train judges in China was awarded to the bidder with the highest score, a consortium of not-for-profit organizations. Three months after the winning bidder was notified, but before the agreement was signed, CIDA discovered that the winning bidder had made a $600,000 error in its calculations. The bid evaluation team had not found this error. Had the error been discovered and added to the bid price as normally required, a different consortium of not-for-profit organizations would have won the bid. Because the agreement had not yet been signed, CIDA still had an opportunity to re-evaluate the bids. CIDA wrote to the selected consortium and advised that it had two options. It could confirm the revised amount as calculated by CIDA, in which case the Agency would reassess its scoring, and this might have an impact on the evaluation result; or the consortium could withdraw its bid. These two options followed the government's normal bid evaluation practice. However, neither option was chosen and CIDA proceeded with negotiations with the selected consortium. Although the consortium asked for increased funding and a reduced level of effort to cover the error, CIDA refused and an agreement was eventually signed at the bid price. In our view, CIDA's treatment of this selection process was unfair to the consortium that was not selected.

14.63 In another case, a bidder who should have been disqualified was chosen (see case study 2).

CIDA does not have a performance evaluation system for executing agencies

14.64 A performance evaluation system for contractors can serve to provide a record to be consulted during subsequent evaluations of tenders. While planning our audit, we looked at whether aid organizations in some other Western countries carried out performance evaluation of contractors. Our research showed that even where performance evaluation was mandatory, such as through legislation, project and program managers did not comply and only a low percentage of contractors were ever evaluated. The main reasons given were fear that public disclosure would lead to lawsuits against officials, and the workload required to carry out evaluations in a public context. For those reasons, CIDA has decided that it will not perform contractor evaluations and that each bid in response to a request for proposal will be evaluated on its own merits.

Execution

Agreements do not provide for management of assumptions deemed critical to project success

14.65 The development environment is one of considerable uncertainty. Every agreement is unique and every bilateral partner presents different challenges. As part of project management, CIDA managers of bilateral projects must identify assumptions at the planning stage that are critical to the project's success. The Canadian executing agencies are required to update the assumptions as part of their preparation of the project implementation plan. Some of these relate to external factors over which the Canadian executing agency would have no control (often the co-operation expected from the host country partner in terms of inputs); other assumptions, such as "equipment provided by the project is adequately maintained," could reasonably be addressed by the Canadian executing agency should the assumption be at risk.

14.66 We examined the agreements and their implementation to determine to what extent CIDA had identified critical assumptions. These can also be viewed as the areas of major risk that need to be managed for the project to be successful. We also examined whether the agreements assigned responsibility for tracking, reporting and resolving issues concerning those critical assumptions.

14.67 Our audit found that where reasonable, and as part of its development of a logical framework analysis, CIDA identified critical assumptions. This analysis is included in the project approval document and the request for proposal to suppliers. However, there was no specific requirement that suppliers describe in their proposals how critical assumptions were to be addressed. Nor did CIDA provide any guidance on how the responsibility for dealing with the critical assumptions would be allocated between CIDA and the Canadian executing agency. After initial identification of the critical assumptions, a framework would be useful to specify how they would be managed and who would be responsible for responding to situations where individual critical assumptions were not being met.

14.68 Only two of the agreements we examined provided for how the critical assumptions would be managed. We found that generally CIDA and its executing agencies have been slow or unable to take responsibility and react when critical assumptions fail to materialize and problems ensue. Case study 1 and case study 2 provide examples.

14.69 CIDA designed a project for Mali to improve its income tax system. An assumption it identified as critical to the project's success was that there would be political will and consistent support by the host government. It signed a contract with the Canadian executing agency in June 1997 for $13 million. When there were signs that the necessary conditions for tax reform were not in place and that local and Canadian personnel did not have the assumed expertise, neither the Canadian executing agency nor CIDA took any decisive action. Canadian executing agency personnel were kept in place and they worked on the activities that they could.

14.70 Some CIDA project officers expressed frustration that the Canadian executing agencies were "not doing their job." CIDA officers often rely on contracted monitors to provide them with independent assessments of executing agencies' progress. The monitors' reports we examined did not routinely address the continuing validity of the critical assumptions or identify new critical assumptions that would need to be realized for the project to succeed.

14.71 Some Canadian executing agencies told us that many of the assumptions (such as the provision of competent staff by the host country) are beyond their control. As well, their contracts do not require them to monitor and report on whether the critical assumptions continue to be valid.

14.72 Where appropriate, CIDA should incorporate into its agreements provisions that designate responsibility for monitoring, reporting and dealing with situations where critical assumptions are at increased risk of no longer being valid.

Agency's response: CIDA will update its Geographic Programs Roadmap to better provide guidance to managers and reflect their responsibility for monitoring, reporting and dealing with situations where critical assumptions are at risk.

Co-operation of the host country is key to successful performance and sustainability

14.73 In our 1998 audit, we described problems of control over the flow of money into counterpart funds. Our sample of 33 agreements for this audit included 6 agreements for items to be sold and the proceeds deposited in counterpart funds. The host country typically signs a memorandum of understanding (MOU) with Canada, guaranteeing the full deposit of the proceeds.

14.74 In two of the six agreements, the host country did not deposit the full amount owed to the counterpart fund, even though it had signed an MOU agreeing to do so.

14.75 In one case, in the fall of 1996 CIDA purchased $4 million of wheat for shipment to Egypt. The Egyptian government had agreed to deposit the full equivalent value in local currency of approximately 10 million Egyptian pounds into a special account to be used for development purposes. Our audit found that the Egyptian government deposited only 8 million Egyptian pounds (approximately $3.2 million) to the special account in July 1997. During our field visit, we observed that CIDA had never verified the amount deposited, as it was entitled to do. In the other case, which occurred in Mali, the problem was similar although the dollar amount was smaller.

14.76 CIDA should verify that agreed-upon amounts have been deposited to counterpart funds, and where they have not it should take appropriate action.

Agency's response: CIDA will more consistently verify that all agreed-upon amounts have been deposited to counterpart funds and take appropriate action where needed.

14.77 Projects in the Geographic Branches include the participation of the host government or one of its agencies in achieving the overall project results. This requires that CIDA's executing agency and the host country partner work closely together in harmony and that each fulfils its designed role in a responsible manner. The contribution of the host country partner to the project is defined in an MOU; however, not all countries have the same capacity or political will to fulfil their commitments in the MOU.

14.78 The Canadian executing agency normally cannot be held responsible for ensuring that the host country partner respects its obligations; nor can it be held responsible for overall project results where there are indications that the host country partner has not fulfilled its commitments. However, it can be made responsible for providing CIDA with information on the participation of the host country at the operational level.

14.79 About half the agreements we examined had difficulties related to the participation of the host country. In those cases, CIDA's options were limited. Either the project continued without all the planned participation of the host country, or CIDA and its executing agency absorbed extra responsibility (for example, see case study 1). In only one case was the project terminated due to unresolvable differences between CIDA and the host country agency. The decision was well founded and prevented considerable further expenditures on a project that was likely to be unsuccessful, but at a considerable cost of time, money and energy.

14.80 Although CIDA plans to spend a certain amount in a country each year, its officers must allow for the capacity of the host country partner when considering the size and complexity of projects being proposed. The difficulties we found relating to host country participation tended to be in large projects in least developed countries.

For projects in serious difficulty, CIDA does not normally cancel the agreements

14.81 Due to the nature of most development projects, as well as the complex logistics of getting Canadian staff on the ground, CIDA usually must pay contractors for the inputs they provide rather than for the outputs they achieve. At the same time, the agreements make the Canadian executing agency accountable for achieving the expected outputs. If the Canadian executing agency fails to do so, CIDA must either cancel the agreement or amend its scope.

14.82 To get a large project planned and approved, tendered, contracted and operationally planned can take anywhere from one to four or five years. It involves considerable expense and effort on the part of both CIDA and the host country partner. Given this large investment in resources and the short-term costs associated with cancellation, it is difficult for CIDA to decide to cancel the agreement when results are not forthcoming. CIDA does not build into its management of project agreements any formal requirement or mechanism for "off ramps" (decisions to proceed or withdraw).

14.83 In our sample of 33 agreements, 15 had a significant portion of deliverables that were not being met as planned. CIDA chose not to cancel 13 of these agreements, and cancelled 2 of them.

14.84 CIDA should build into its larger service agreements a provision for a formal review point, at which time a decision can be made on whether the project should be continued or cancelled or its scope changed. The procedures for doing this should be built into the agreements. (While the timing could vary, a logical point might be 12 to 18 months after CIDA's approval of the Project Implementation Plan.)

Agency's response: CIDA had already started to do this for some of its larger service agreements. CIDA will now more systematically include the provision of a formal review point as part of the professional services agreement.

Agreements are generally well monitored

14.85 We found that CIDA staff attach considerable importance to monitoring projects under their responsibility. They commonly use monitors under contract to review and report on the progress being made, and they insist on receiving periodic reports from the Canadian executing agencies as stipulated in the agreements. They also visit projects and attend on-site project steering committee meetings.

14.86 The quality of reporting by Canadian executing agencies continues to present difficulties for project officers. All the service agreements we examined contained clauses that required the Canadian executing agency to report on progress. Nevertheless, both CIDA and Canadian executing agencies expressed their frustration to us over expectations for these reports, and the time taken to produce them. Canadian executing agencies often produce the reports late, and it may take a few iterations to satisfy the CIDA project officer. Canadian executing agencies told us that CIDA does not specify clearly what it wants; however, CIDA officers told us they believe it is the Canadian executing agency's job to develop an appropriate report. In some cases, this problem resulted in projects being slowed because the CIDA officer did not have the necessary information to approve new work plans.

14.87 In May 1999, CIDA's Performance Review Branch issued a Guide to Project Performance Reporting for Canadian Partners and Executing Agencies. The guide, which is posted on CIDA's Web site, provides a template for results-based project reporting. It also explains how the monitoring of assumptions critical to a project's success should be integrated into the report and how financial and activity information should be presented. From the agreements we examined, this guide does not seem to be widely applied yet in the Geographic Branches.

The quality of annual project progress reports continues to be inconsistent

14.88 Each project officer in CIDA is required to prepare an annual project progress report (APPR) on each project. The report shows an overall assessment of the project's progress against plans, for both results achieved and amounts spent. This report has been the main internal method of accountability for results by project. It is also the basic document used for preparing CIDA's annual Performance Report to Parliament. In our 1998 audit of Geographic programs, we examined the quality of CIDA's APPRs and found that it varied widely.

14.89 We examined the APPRs in our sample for the year ended 1998-99. We found that in 6 of the 29 reports, some information did not realistically describe the actual status of the project. Most of the problems with the quality of the information related to the overall assessment of whether a project was reported as progressing satisfactorily, had "manageable problems" or had "serious problems". CIDA has not provided project officers or managers with guidance on how to interpret these terms.

Agreements need to better support results-based management

14.90 In order to support results-based management (RBM), agreements need to be managed with a high degree of professionalism:

  • Expectations for results need to be clear and realistic.
  • The capacity of the host country partner needs to be properly assessed.
  • The sustainability of project benefits needs to be addressed in agreements where appropriate.
  • Agreements need to be tendered competitively where applicable.
  • Assumptions critical to a project's success need to be managed.
  • Operational issues such as staff turnover and progress reporting need to be dealt with smoothly.

Our audit identified problems in the above-noted areas at the planning, selection and execution stages of the agreement process. When a combination of problems is present in a particular agreement, the effect is compounded and it becomes more difficult to obtain results.

14.91 When the proper elements of results-based management are in place, the chances are increased that an agreement will result in a successful project. One such example is the Women's Law project in China (see case study 3).

14.92 As CIDA works on improving its management of contracts and contribution agreements, we believe it would be useful to consider a number of questions, such as the following:

  • Should CIDA managers focus on identifying the development need and the results expected, letting bidders propose how they would design and manage a project to achieve the expected results?
  • How can formal review points or "off ramps" be built into agreements so that projects less likely to achieve desired results can be terminated without great additional cost or loss of goodwill?

Canadian Partnership Branch Agreements

CIDA often does not receive key information for monitoring performance

14.93 The objective of the Canadian Partnership Branch (CPB) is to promote mutually beneficial partnerships between Canadian and developing country organizations in order to support sustainable development and reduce poverty in the developing world. CPB partners include a wide range of organizations: community, development and environmental organizations, universities, churches, professional associations and co-operatives.

14.94 Canadian Partnership Branch operations are delivered through four programs: Voluntary Sector, Industrial Co-operation, Scholarships and Awards, and Youth Internship. Our examination of the Branch focussed on the largest of its programs - the Voluntary Sector program. Organizations in the Voluntary Sector program are funded through either program or project funding. Program funding constitutes financial assistance, which is renewable and thereby enables an organization to undertake development initiatives on a broader, more continuous basis. Program agreements normally extend up to three years. Each program usually comprises a number of projects, ranging from a few to several hundred. Project funding provides support for a single non-renewable initiative for one to three years.

14.95 The Canadian Partnership Branch ensures the eligibility of the partner organizations that receive support and assesses the extent to which their programs reflect the Agency's programming priorities and objectives. The Branch is also responsible for assessing the financial and technical capabilities of partner organizations and for assessing progress toward achieving the program results sought. CIDA does not get involved in the day-to-day management of the partners' program and project activities. The Agency focusses its efforts toward managing its relationship with its program partners at the institutional level, rather than at the project level.

14.96 As part of this audit, we examined a sample of seven program agreements in the non-governmental organization division of the Voluntary Sector program. We selected agreements that had one or more projects in the six countries we visited. All the agreements selected were with organizations that had been CIDA partners for a number of years.

14.97 Our audit concentrated on how well the control framework of the Canadian Partnership Branch is working. We identified a number of key elements in the CPB control framework, including:

  • grant and contribution agreements that call for CIDA's partners to provide interim financial reports, annual financial reports, and annual narrative reports;
  • review of multi-year program proposals;
  • annual review of financial health by the Financial Risk Assessment Unit;
  • periodic institutional assessments; and
  • ongoing dialogue and occasional site-visits with partners.

14.98 For the agreements we reviewed, financial reports as well as annual audited financial statements were generally submitted as required. The narrative reports submitted by partner organizations varied considerably in the quality of information provided. Two reports we reviewed clearly described the programs and projects that were funded and the amounts spent on them. The others provided little specific information in these key areas. In the absence of this information, the CPB cannot know or verify statements of results achieved. In addition, where results were indicated, they were not linked back to those expected. In an effort to improve reporting on results, in January 1999 CPB instructed its program officers that effective immediately all new proposals for funding renewal were to include a results-based management planning framework. In addition, a Guide to Performance Reporting for Canadian and Developing Country Partners was issued in February 2000.

14.99 We found that the annual financial risk assessment had been done in six of the seven cases we looked at. In the other case, the assessment done in the year previous to the agreement indicated a low risk. CIDA received audited financial statements for all seven program agreements. In our opinion, the financial risk assessments were effective in providing assurance to CIDA on the financial health of its partners.

14.100 The primary goal of the periodic institutional assessment is to provide an overview of the organization, specifically related to issues of management and administration. The terms of reference of the institutional assessments do not require an assessment of the organization's efficiency and effectiveness in obtaining development results. However, the assessments we reviewed noted that CIDA's program partners had difficulty measuring the outcomes of their program activities. In many cases, this inability resulted from an absence of baseline data. As a result, the institutional assessments provide CIDA with limited information on whether its partners are delivering effective programming.

14.101 Institutional assessments are essentially a review of the organization, which are agreed upon jointly and carried out by an independent third party. They are normally conducted when a program renewal is due. CPB managers are accountable under CIDA policy for assessing the capacity of recipients to deliver aid. However, institutional assessments were not carried out in two of the seven cases prior to their program renewal. For the five program agreements that had institutional assessments, there was evidence that recommendations were being followed up by the partner organizations.

14.102 We noted that CPB officers are required to review each program proposal before recommending a funding renewal. These reviews include the program officer's assessment of the performance of the institution over the past one to three years. However, in the agreements we reviewed, with one exception the funding approved by the Minister from 1996-97 to 1998-99 was based on historical levels less any across-the-board cuts as a result of government spending restraint. In the one exception, the partner's funding was reduced as a result of CIDA's performance assessment.

14.103 We also noted that for 1998-99, the Minister approved across-the-board cuts in funds to most CPB partners, but provided increases to 5 out of 27 organizations as a result of CIDA performance assessments.

14.104 For the agreements we examined, CIDA's control framework provides it with reasonable information on the financial health of its partners. However, CIDA receives only limited information on development results at the program and project level. As noted in paragraph 14.100, CIDA's institutional assessments commented on the limited ability of partners to provide performance information relating to the outcomes of their programs.

14.105 The Canadian Partnership Branch should continue its efforts to obtain better performance information from its partners on the use of funds contributed by CIDA and development results achieved.

Agency's response: The Canadian Partnership Branch has recently developed "A Guide to Reporting" for its partners, which provides guidance on the nature and scope of performance and financial information required by CIDA.

More meaningful and accurate information is needed on the Canadian Partnership Branch in the Performance Report to Parliament

14.106 As part of our audit, we reviewed the performance information on Canadian Partnership Branch activities provided to Parliament in CIDA's 1998-99 Performance Report. Our observations are similar to those we made in 1998 on the Geographic programs' information in CIDA's Performance Report. The CPB information in the Performance Report provides examples of achievements without any context on expected results or resources expended to achieve them. Where the examples span a number of years, the number of years is not indicated.

14.107 We also reviewed two specific items of information in the Performance Report to assess their accuracy.

  • The Report indicated that for every $1 contributed by the CPB to the Voluntary Sector program in 1998-99, an average of $1.19 in funds was contributed by partner organizations. Based on our review, a better presentation of CIDA's contribution relative to that of its partners would have been closer to $1 for every $0.57.
  • The Report also indicated that 85 percent of CPB's programs and projects were likely to meet or exceed planned results. CIDA based this estimate on results ratings provided by Voluntary Sector program officers. We noted some concerns about the quality of the information that officers relied on in their estimates. For example, four based their estimate of progress on the amount of budget spent. Based on our review of the documentation for the seven program agreements we audited, we could agree with only one of the results ratings provided by the program officers. This leads us to conclude that the figure of 85 percent was overstated.

14.108 CIDA should improve the completeness and accuracy of the information on the Canadian Partnership Branch in its Performance Report.

Agency's response: The Canadian Partnership Branch will continue to improve indicators and statistics given in the Performance Report to ensure that they are more meaningful and useful for the reader. The training offered to, and the experience gained by, Canadian Partnership Branch staff in results-based management will over time improve the quality of the reporting on results being achieved.

Gaps in CIDA's New Information System

14.109 As part of implementing its new Enterprise Resource Planning (ERP) integrated management information system, CIDA developed a module called "RBM tool". Project officers regularly input into this module the project information that currently goes into the Annual Project Progress Report. This module represents an improvement; although the information it contains is essentially narrative, it automatically integrates information from the financial module on total project spending to date. The new module also permits easier compilation of aggregate reports by data field - for example, a report listing all projects with serious problems and a narrative description of the problem. The module is useful for senior management review and for capturing a global view of a project's progress.

14.110 However, the introduction of the ERP system and the RBM tool module has so far not achieved the integration of financial, contract and performance data at the project management level. To properly manage agreements, project officers require information to monitor the details of outputs and activities against agreed-upon work plans. They also need to be able to match data on project spending with other data on outputs provided by executing agencies or monitors. An example of this type of project reporting was provided in CIDA's 1999 Guide to Project Performance Reporting: For Canadian Partners and Executing Agencies.

14.111 Out of its total information technology budget of $32 million for 2000-2001, CIDA has allocated $18 million to further develop its ERP system. Although there is reference in the planning documents to the RBM tool module, it has not yet been decided how the funding will be used. At this time, it is not clear whether the system improvements will provide project officers with detailed data on project activities and results relative to work plans, and be able to relate these data to spending at this level of detail. For instance, the software has the capacity to accommodate a reporting format under which Canadian executing agencies could submit performance information electronically to CIDA project managers for their input to the system. However, CIDA has no specific plans to develop this type of reporting format.

Some Compliance Problems in Contracts Under $100,000

14.112 For the years 1996-97 to 1998-99, CIDA entered into about 6,100 agreements; 5,300 of these were under $100,000 and reflected about 10 percent of the total dollar value of all agreements. Of the agreements under $100,000, about three quarters had an original value of under $25,000, which is the threshold above which a competition is normally required. Fewer than 200 of those were amended to more than $25,000.

14.113 Some contract splitting occurred. We focussed on the approximately 1,500 agreements between $25,000 and $100,000. We looked for possible cases of contract splitting. This occurs when an organization splits a larger contract into smaller contracts, thereby circumventing contracting regulations or policies. We analyzed a variety of possible indicators, such as more than one contract awarded to the same supplier with overlapping or concurrent dates, or several contracts awarded to the same supplier for the same amount. We then selected for further examination 13 service contracts issued to four companies from 1996-97 to 1998-99. We concluded that contract splitting avoided NAFTA tendering requirements in 9 of the 13 cases (see Exhibit 14.9 for an example).

14.114 Non-compliance with contracting policy regarding former public servants. In the course of our audit, we became aware that CIDA had entered into 10 non-competitive contracts with former public servants who were receiving a pension. These contracts contravened CIDA's delegated authority because the Agency did not obtain prior Treasury Board approval, as required for contracts with former public servants who were receiving a pension if the contract is non-competitive and its value exceeds $25,000.

14.115 In the fall of 1999, CIDA discovered two cases that contravened its delegated authority. To its credit, CIDA then examined its database of approximately 3,000 contracts and requisitions entered into with individuals since April 1997. CIDA's examination was made more difficult because there was no central or departmental registry that identified former public servants who were receiving a pension or had received a retirement incentive package. CIDA relied on the collective memory of its Human Resources and Corporate Services Branch staff to identify former such CIDA employees who had received a subsequent CIDA contract. However, this collective memory could not be extended to consistently identify former public servants from other departments who had received contracts from CIDA. From this examination, CIDA compiled a list of 114 of its former employees whose files it then included in preparing the information for submission to the Treasury Board. A total of 10 cases were identified. Three contracts dealing with a single Corporate Services (non-aid) issue are still active, with a total value of $886,000.

14.116 Because issues such as non-competitive contracting, compliance with authorities and due diligence were of particular interest to us in this audit, we examined how this situation arose in the first place. We also examined the adequacy of the actions taken to obtain retroactive Treasury Board approval of these contracts and to ensure that the problem does not happen again.

14.117 We paid particular attention to the three contracts that remain active. In addition to the compliance issue identified by CIDA, we have some further concerns:

  • We noted that one company had previously obtained four other non-competitive contracts for the services of the same individual. One of these, for $46,700, also exceeded CIDA's authority but was not included in the Treasury Board submission for authorization of retroactive contracts.
  • There was little information to support the fees arrived at. In two cases, CIDA accepted the statement that the fee amount matched what a private sector client would pay. However, there was no documentation on file indicating the fees that had been charged to others. In the third case, the program officer authorized more than recommended by CIDA's Contracting Management Division; this was done on the basis that had the former employee not retired, the employee likely would have had a promotion in the intervening period and should, therefore, be paid at a higher rate.
  • In our view, the documentation in these files did not adequately justify an exception from the Government Contracts Regulations to enter into non-competitive contracts.

14.118 CIDA has since taken measures to prevent a recurrence of the problem. Among other measures, it has issued a bulletin to all staff reminding them of the policy on contracting with former public servants. It has also reminded staff of the requirement for a declaration form to be completed by an individual or a firm, indicating whether the individual is a former public servant in receipt of a pension. This declaration is to be maintained in the contracts file and a copy kept centrally with Contracting Management Division.

14.119 We believe that CIDA also needs to seek Treasury Board approval for the overlooked contract that exceeded $25,000.

14.120 The actions taken by CIDA should serve to keep this situation from recurring - but only if policy requirements are respected.

Conclusion

14.121 CIDA generally complied with Government Contracts Regulations and Treasury Board Contracting Policy in its contracting processes. However, the terms and conditions governing the use of contribution agreements are very general and permit CIDA to make exceptions to its regular procedures. CIDA varied from its internal policies or guidelines by often using contribution agreements instead of competitive contracts for projects that were not unsolicited proposals but were initiated by CIDA. This also raises questions on the fairness and transparency of the process used by CIDA in selecting executing agencies for its larger projects.

14.122 CIDA's processes for entering into contracts and contribution agreements are not yet providing sufficient support for results-based management. In many instances, these agreements do not meet CIDA's operational requirements. They contain unclear or unrealistic expected results, or do not provide for monitoring of and action on changes to assumptions that are critical to project success.

14.123 The control framework for the non-governmental organization division of the Voluntary Sector program in the Canadian Partnership Branch provides reasonable information on the financial health of CIDA's partners. However, there was limited information on the projects funded and results achieved. The quality of information on the Canadian Partnership Branch in CIDA's Performance Report to Parliament needs to be improved.

About the Audit

Objectives

Our objectives in this audit were to assess whether CIDA's contracting/contribution processes:

  • respect Government Contracts Regulations, Treasury Board guidelines and its own policies;
  • are fair and transparent; and
  • meet operational requirements and development needs.

Scope and Approach

This audit focussed on how CIDA Geographic Branches (excluding the Central and Eastern Europe Branch) manage contracts, contribution agreements and other types of agreements for goods and services, including the selection of Canadian executing agencies to deliver projects.

The audit also examined the control framework for agreements in the Voluntary Sector program of the Canadian Partnership Branch.

We conducted our work at CIDA's headquarters and visited the selected projects in the field. We reviewed project documentation and held discussions with the local counterparts for the selected projects, and with Canadian executing agencies and CIDA officials.

Contracts and contribution agreements population

Since CIDA uses contracts, contributions and other types of agreements for the same purposes and manages them all in a similar fashion, we included all these types of agreements in our sampling plan.

In selecting a population of agreements to test from, we wanted to include both agreements that had taken place before the Open Competition Pilot Project as well as those made after. We included in our population contracts and contribution agreements signed in the Geographic Branches, as well as grants and contributions signed in the Canadian Partnership Branch, for the three-year period from 1 April 1996 to 31 March 1999.

Sampling plan

In the Geographic Branches, agreements with a value of over $100,000 account for over 90 percent of the total value of all agreements. To carry out our audit, we developed a dollar-unit statistical sampling plan that involved examining $1.8 billion of Geographic Branches agreements and $600 million of Canadian Partnership Branch agreements signed in the years 1996-97 to 1998-99. This resulted in the selection of 33 Geographic Branches agreements with a total value of $214 million, and 7 Canadian Partnership Branch agreements, in six chosen countries. The countries chosen were India and China for Asia, Peru for the Americas, and Egypt, Cameroon and Mali for Africa and the Middle East. The selection of countries reflected the dollar importance of each region and discussions with CIDA.

In order to test the continued relevance of our findings, we also randomly selected for examination a separate small sample of agreements signed in the 1999-2000 year, for which we did not perform field visits.

Criteria

Our main sources of criteria were previous audits of the Office of the Auditor General, Treasury Board directives and CIDA guidelines.

a. General criteria
  • Government Contracts Regulations, and Treasury Board and CIDA policies should be respected.
  • For contribution agreements and sole-source contracts, review and control of selection of suppliers, negotiation of arrangements, drafting of terms and monitoring should be at least as rigorous as for competitive contracts.
  • Where possible, contribution agreements and contracts should reflect results-based management principles, policy and framework in all possible respects.
  • Information from performance monitoring reports and reviews should be used as required for management decision making and taking corrective action on projects.
b. Criteria related to the contracting/contribution agreement process
Planning
  • The contracting plan should contain an analysis of different options for the particular project, with competition being the norm. For the non-competitive options, the choice should be well documented and in line with Treasury Board approvals and contracting policy.
Design
  • The statement of work should be clear enough to be translated into contract deliverables.
  • Recipient country and counterpart fund contract plans should respect government and CIDA contracting goals of probity, fairness and transparency.
Selection
  • The contract team should have sufficient technical knowledge to develop a detailed evaluation grid and to use the grid to evaluate the proposals.
  • The evaluation grid should be sufficiently detailed so that the weights assigned to each item being ranked are shown.
  • The evaluation grid should be consistent with the Terms of Reference.
  • The evaluation of the project proposal should address the contractor's approach to managing inherent and external project risks.
Execution and monitoring
  • Project implementation plans should be consistent with the statement of work in the contract and the project approval document and also show the steps and milestones to achieve results.
  • Contract performance should be monitored and evaluated against the terms and conditions of the contract. Contractual remedies for substandard performance should be applied.
  • Project and contract files should follow a standard format and be well maintained.
Performance reporting
  • Annual project progress reports should be an accurate representation of the project status and results.
Environment
  • Where there is an element (goods or physical work) of the project that could affect the environment, an environmental assessment should be completed to determine environmental effects.

Audit Team

Assistant Auditor General: David Rattray
Principal: John Hitchinson
Director: Paul Morse

Richard Brisebois
Lori Buck
Shannon Byrne
Derek Demonte
Bryan De Pape
Geneviève Fournier
Johane Garneau
Geneviève Hivon
Manfred Kuhnapfel
Raymond Kunce
Amipal Manchanda
Jacques Marquis
Barry Neilson
Barbara Vandernoot

For information, please contact John Hitchinson.