1993 Report of the Auditor General of Canada
Chapter 12—Canadian International Development Agency—Bilateral Economic and Social Development Programs
Our view: There is a need for CIDA to streamline the way it runs its business, including its existing controls, to be more efficient and effective
Our view: CIDA's approach to project management needs to be revised. More innovative project management skills and practices are needed
Our view: CIDA needs to be more transparent about what it is trying to achieve and how well it is doing
CIDA recognizes the need for reform and has outlined a sequential approach to meeting the challenges ahead
Assistant Auditor General: Richard B. Fadden
Responsible Auditor: Vinod Sahgal
12.1 Official Development Assistance is a significant aspect of Canada's foreign policy. For 25 years CIDA has been the main conduit for delivering development assistance. Annual expenditures for bilateral economic and social development assistance exceed one billion dollars.
12.2 We conducted this audit taking into account the increasing international awareness that continued investment in development projects that are not likely to be sustained beyond donors' financial assistance represents a questionable use of scarce resources.
12.3 In its ability to meet development needs of the nineties, the Agency has been losing ground. Current performance has not maximized the use of resources and has not led sufficiently to self-reliant development. There is considerable scope to improve CIDA's performance in promoting enduring benefits from its investments.
12.4 CIDA's bilateral programs need to concentrate more on those countries and activities where there is the greatest potential.
12.5 Lessons learned from 25 years' experience show a need to resolve conflicts among multiple objectives and to establish a more results-oriented, focussed, businesslike and accountable style of operation. They also call for dedication to the basic objectives - such as fighting poverty and helping people help themselves - that Canada has affirmed repeatedly over many years.
12.6 None of these changes will be easy to make, since they will require a change in mindset, skills and culture, and strong ministerial support. Their successful implementation will depend on the will to reform, not only among management and staff of the Agency but also among those who act as its development partners, in Canada and in the developing countries where CIDA operates.see Exhibit 12.1 )
12.8 Chapter 11 of our 1992 Report dealt with the way CIDA manages Canada's participation in the regional development banks. We audited certain other aspects of CIDA's operations in 1988, including contributions to non-governmental organizations.
12.9 The past decade has been particularly turbulent, with events that have challenged earlier assumptions about development aid and about what works. The link between development and the degradation of the environment has become more visible. The challenge ahead will be to articulate a clear strategic direction and priorities for Canada's overall co-operation in international development. This will have to be done during a period of large federal deficits, increased emphasis on domestic needs, demands for economic assistance from emerging nations in Eastern Europe and the related pressure for better fiscal planning and control. A new policy direction for Canada's overall international co-operation may become necessary.
12.10 CIDA provides assistance to about 115 countries; however, 70 percent of its bilateral expenditures are directed to a core group of 35 countries in Asia, Africa and the Americas.
12.11 Few Canadian organizations, private or public, attempt the complex and high-risk task that CIDA undertakes: making large annual expenditures in a wide variety of ambitious projects in a great many countries and in a multitude of economic and social sectors, often in uncertain political and economic conditions. CIDA in the field has a reputation for integrity and co-operation.
Official Development Assistance is a significant part of Canada's foreign policy12.12 In 1987, after a comprehensive review of Canada's Official Development Assistance, the House of Commons Standing Committee on External Affairs and International Trade published a report entitled "For Whose Benefit?" The Committee report and the government response to it led to a comprehensive strategy for development assistance. The document entitled Sharing Our Future became Canada's foreign policy on Official Development Assistance. It reaffirms earlier strategic policy directions - fighting poverty and promoting self-reliant development. CIDA has pointed out that there is no other official source of Canadian policy on Official Development Assistance. ( see Special Insert )
12.13 Thinking has evolved over the past two decades on how best to alleviate poverty. In the 1960s, general economic growth was seen as the key. As the "trickle down" theory proved unrealistic, emphasis in the seventies shifted to social and economic interventions intended to redistribute income. The results of that policy were mixed at best. In the 1980s, the focus changed to the interrelationship among population growth, poverty, environmental degradation and the policy environment. It became clear that translating economic growth into human development and reduction of poverty requires effective policy management. Today, CIDA points out that a combination of factors contributes to alleviating poverty: policy reform in key sectors, structural adjustment leading to improved macro-economic planning and emphasis on human resource development. Fighting global poverty through economic and social development is recognized as an investment in the future.
12.14 The importance of development aid to Canada's foreign policy has been reaffirmed in the recent document "Foreign Policy Themes and Priorities" (1991-92 update) issued by the Department of External Affairs.
CIDA's mandate12.15 There is no specific enabling legislation that sets out CIDA's mandate and accountability for Official Development Assistance, although the House of Commons Standing Committee on External Affairs and International Trade recommended such legislation in 1987. The ODA charter in the strategy document Sharing Our Future sets out the basis for its mandate. The Department of External Affairs Act makes the Secretary of State for External Affairs the Minister responsible for CIDA, thereby bringing the actions of the Agency under that Act's broad authority. Its spending is authorized by the annual appropriation Acts. ( see Special Insert )
12.16 Program objective. Part III of CIDA's 1993-94 Estimates states: "The objective of the CIDA program is to facilitate the effort of the peoples of developing countries to achieve self-sustainable economic and social development in accordance with their needs and environment by co-operating with them in development activities; and to provide humanitarian assistance." Two percent of total Official Development Assistance was to be allocated to international humanitarian assistance.
Audit Objective, Scope and Approach12.17 We planned this audit taking into account the increasing international awareness that many development projects are having problems, although reportedly fewer in some regions than others. There is cause for concern. The World Bank stated, in a 1992 report, that the number of its projects facing ``major problems" has gone up significantly. Management in this institution was reportedly more preoccupied with approving new loans than with pursuing the successful implementation of development projects. Now, however, as an indicator of effectiveness, successful implementation of approved projects is taking priority over new annual commitments. Similar difficulties have been voiced by other development agencies, such as US A.I.D. It has been pointed out that continued investment in development projects that are not likely to be sustained beyond donors' financial assistance represents a major waste of scarce resources.
12.18 Objective . We conducted this audit with a view to:
- providing objective information, advice and assurance to help Parliament scrutinize CIDA's use of resources and the management of its results;
- promoting accountability and good practices in CIDA in managing for results; and
- highlighting areas for improving the Agency's management of its country programs and projects.
- CIDA implements the principles and priorities set forth in the Official Development Assistance strategy Sharing Our Future , and whether there are other relevant considerations;
- CIDA defines, plans, implements and accounts for the results of its programs and activities;
- the Agency, in cases where it entrusts other parties with the disbursement of Canadian funds on its behalf, obtains independent assurance that moneys have been spent for the purposes intended, and have been accounted for fully; and
- management identifies lessons learned and applies them to future policy, programming and implementation at the Agency, country and activity levels.
12.21 We used a combined country- and project-based approach to conduct this audit. CIDA funds its bilateral programs and activities on a country basis. This allows it to consult with the Department of External Affairs on foreign policy and related funding considerations. It also allows for dialogue with the recipient country and with other donors. This helps in developing a framework for matching the needs of the recipient country with Canadian capabilities, while taking into account foreign policy and fiscal considerations. At the same time, CIDA's discrete activities are carried out on a project basis.
12.22 We focussed our attention on three country programs, selected in consultation with CIDA management, and examined in depth 18 development projects in the field. The countries - Bangladesh, Pakistan, and Sri Lanka - are situated in a region where a significant portion of the world's poorest people live and where CIDA has been operating for over 25 years. Understanding the political, economic and cultural environment and the development needs of these countries was a key consideration in planning this audit. We do not assume that these three countries and 18 projects reflect all the kinds of programs and projects CIDA undertakes around the world, but CIDA agrees with us that our findings on its management are representative. No one CIDA country is "typical", but CIDA's management of bilateral development aid projects is reasonably well represented by its approach in the three countries.
12.23 Emphasis . We paid special attention to understanding CIDA's accountability in relation to managing for results. Accountability at its simplest calls for holding public officials answerable for their actions, including the provision of appropriate information to Parliament. The objective is not so much to attach blame when something goes wrong as to find out why it went wrong, and what can be done to rectify it and guard against a recurrence in the future. As well, we explored what kinds of yardsticks could be used to monitor the progress and results of CIDA's development projects. ( see Special Insert )
12.24 The Minister without question is accountable to Parliament for measuring and reporting on the results of CIDA's programs. The Minister's accountability must take into account that, in practice, CIDA cannot be expected to be successful in every activity it undertakes. CIDA's operating environment is such that control over the ultimate result of its efforts is often limited by the fact that it works in partnership with other sovereign states.
12.25 At the time of our audit, CIDA had completed a major Strategic Management Review. This review was comprehensive: it covered the organizational status of the Agency, its major management processes and its various delivery methods. We took the review's findings into account in shaping our audit and developing our observations and conclusions. ( see Special Insert )
12.26 We consulted with CIDA's Audit and Evaluation Division. It had seldom carried out significant examinations of the Agency's overall management of the bilateral programs, in terms of results or the effectiveness of its programs in each country. Consequently, the Division had not systematically addressed country-related or other obstacles to successful implementation of the Agency's mandate. Accordingly, in our audit we could place only limited reliance on its work. A study the Division had commissioned on the quality of project evaluations by CIDA's geographic branches was helpful to our understanding of the Agency's operating environment. We used the results of the study in formulating our views on what improvements are needed in project evaluations. We discussed our audit approach with the Auditors General of each of the three countries covered in our examination and with members of their staffs who had received training in Canada through the Canadian Comprehensive Auditing Foundation.
Audit Observations12.27 Our observations are organized into several sections. First, we discuss some of the challenges that CIDA faces. Second, we outline the lessons that CIDA itself identified in its recent Strategic Management Review, and that we have confirmed in our audit. Third, we describe the actions CIDA is taking to respond to the observations of its Strategic Management Review. Fourth, we set out some additional findings that we believe are crucial to CIDA's achieving value in its bilateral country programs.
Our Findings on Lessons Identified by CIDA's Strategic Management Review12.28 CIDA has reflected recently on the lessons it has learned in two and a half decades of operation. The main vehicle for this reflection was the Strategic Management Review, one objective of which was to assess CIDA's ability to meet the challenges of the nineties and recommend ways to improve its management practices and philosophy. In our view, this was a timely and valuable re-examination of the fundamentals of the Agency's operations.
12.29 The consultants who conducted the main examination at CIDA's request identified many problems that call for early remedial action, and formulated three key recommendations:
- improve the Agency's "development" knowledge base;
- increase the relevance of the aid program through better dialogue with its stakeholders; and
- reduce its delivery costs.
CIDA's Lesson #1 There remains considerable confusion in the minds of many about the sometimes conflicting objectives of the Official Development Assistance program and about how to resolve these conflicts on a daily basis.
Our view: Conflicts among objectives need to be resolved12.31 The Agency, in practice, has been tasked with multiple objectives: while charged with pursuing one primary objective, CIDA has others that can often conflict with it. It is difficult for CIDA to concentrate on putting poverty first and encouraging self-reliance while, at the same time, it has commercial and political objectives that do not always lend themselves to dealing with poverty in a direct way and that encourage external dependency. We appreciate that government programs seldom operate in sole pursuit of a single overriding objective. Nevertheless, competing or conflicting objectives should be pursued only consciously and in full realization of their effect on the primary objective.
12.32 The way development is to be implemented is, at present, very broadly defined. Often the link between the activities undertaken and CIDA's primary purpose of "putting poverty first", as defined in the ODA Charter, appears to be indirect.
12.33 In Bangladesh, the site of CIDA's largest bilateral program, Canadian bilateral aid to date has been over $2 billion. Food and commodity aid accounted for two thirds of the almost $1.1 billion total aid between 1981 and 1990. Even now, food and commodity aid account for 40 percent of annual bilateral aid to this country. This is notwithstanding CIDA's stated priority for South Asia: to direct a considerable portion of the funds in the country programs toward improving the productive capacity of the poor. Although we recognize that humanitarian food aid to bridge a food gap is sometimes necessary, CIDA agrees that it should not act as a disincentive to local production. Aid in this form given over extended periods raises questions about the country's dependency on foreign aid. We do not expect that the Agency can or should discontinue food aid. However, it is recognized that the effects of such aid - however beneficial - are necessarily short-term if the agricultural productive capacity of the recipient country is not improved in a sustainable way in the long run.
12.34 CIDA points out that the many structural weaknesses in Bangladesh make self-reliant development very difficult. For example, the state institutions that CIDA must deal with are weak and there is a lack of local financial resources for operating and maintaining development projects. Given these shortcomings, it has been argued that it is preferable to continue funding economic and social development projects, with emphasis on improving the policy environment, even if the result is more external dependency. It was pointed out that the consequences could be political, social and economic chaos if donors withdrew their support. In such a case, the costs of peacekeeping and/or emergency humanitarian assistance could dwarf the present cost of bilateral aid. Although these arguments cannot be dismissed, we believe that they do not sufficiently emphasize CIDA's primary purpose for using funds earmarked for self-reliant development.
12.35 In Pakistan, where Canada has invested over $1.3 billion to date, Canadian commercial and other interests appear to have shaped CIDA's activities as much as the objective of helping the poorest people become more self-reliant. For example, in the 1980s a large share of project aid was devoted to funding state-owned infrastructure projects in rail transportation and energy, which were delivered by the Canadian private sector. These do not appear to be the most direct ways for helping Pakistan's poorest people improve their human development and earning capacity. Direct involvement in areas such as population control, education, health and the productive capacity of the poorest people was not prominent in the portfolio of projects we examined.
12.36 Many factors have contributed to the difficulty of producing with dispatch a coherent plan of work that would put approved development objectives first. CIDA is subject to many competing interests, and Pakistan, like many other developing countries, is plagued by chronic poverty and, reportedly, by administrative complexities and weak state institutions, which have traditionally delivered aid projects. It was explained to us that without a functioning system of financial accountability, government efficiency in the recipient country is reduced, and the probability of corruption increases. An effective network of non-governmental organizations as an alternative vehicle for aid has yet to evolve. This is the inherently difficult situation in which CIDA must work.
12.37 In Pakistan we encountered an extraordinary range of views among CIDA officials on the strategic thrust the Agency should be taking, and we observed a lack of coherence in programming. For example, opinions on what the country program should include range across such diverse approaches as working in small grass-roots projects (a small contribution to a girls' school; helping villagers build their own bridge over a stream); helping the country reform its judicial system to minimize administrative constraints and inefficiencies; encouraging Canadian private business to increase trade; and deregulating and opening the economy along the lines of a model that, to some, appears to have been successful in Indonesia. Moreover, recipient government officials have lobbied for megaprojects in the power and oil and gas sectors, and have been supported by some Canadian private interests. Each of these views may have merit, but a sense of coherent direction has not developed. Some argue that CIDA's ability to deal more directly with poverty may, in the future, require placing a higher priority on the social sector - which reportedly the Government of Pakistan has not supported sufficiently in the past. ( see photographs )
12.38 In Sri Lanka, we found CIDA's projects and programs driven primarily by Canada's political concerns about social justice, peace and human rights in a situation where civil order had broken down. CIDA's budget for this country has gone down from a peak of $43 million in 1982-83 to less than $10 million today. Some donors point out that, unless human rights are restored and the root causes of ethnic conflict satisfactorily resolved, traditional economic and social development objectives are bound to be thwarted. This may well be true, but should CIDA continue to provide Official Development Assistance funds in a country where it believes that its traditional development activities, emphasizing human resource development, have become impossible to implement? If CIDA believes that funding should continue, it should publicly explain the basis for the shift in emphasis, the criteria applied to make this decision, and the nature of the program's reorientation.
12.39 In summary , the issue of conflicting objectives raised in 1987 by the House of Commons Standing Committee on External Affairs and International Trade remains unresolved. At stake is CIDA's ability to pursue its development mandate effectively.
CIDA's Lesson #2 CIDA is an over-extended Agency. Activities are dispersed over numbers of countries, projects, delivery channels and funded institutions. A lack of strategic focus leads to reduced program impact. Furthermore, Canada's influence with recipient governments is diluted, given our relatively low ranking among donors in terms of volume of aid to any given country.
Our view: CIDA needs to concentrate its efforts12.40 CIDA would benefit from concentrating on its primary objectives and priorities and reducing the dispersion of its programs. There has been, in the past, little Agency-wide incentive or pressure to do so. CIDA needs to be more selective, by focussing on key countries and more productive activities where there is the greatest potential for benefit in terms of development.
12.41 Despite recent efforts by CIDA to reduce dispersion, particularly in Asia, we found that the mix of existing projects and programs covered a wide range of sectors, types of CIDA support, channels of delivery and funding mechanisms, as shown in Exhibit 12.2 . There was a general lack of focus, especially evident in two of the three country programs we examined.
12.42 Although in the last few years CIDA has reviewed its strategic objectives for many of its "core" countries, and although the plans (Country Policy Frameworks) do represent, in cases like Pakistan, some narrowing of focus, the objectives typically remain broad and the measures of success undefined. Over time, dispersion may be reduced, aided by shrinking budgets. In the meantime, however, CIDA in some respects has extended its scope over the past few years. In addition to servicing capital infrastructure projects and undertaking economic and social programs and projects, which have been the traditional focus of its attention, CIDA has greatly increased its emphasis on the broad themes of human rights, good governance and the environment. "Policy dialogue" with recipient governments has received more emphasis and CIDA has co-operated with other agencies, such as the International Monetary Fund and the World Bank, in "structural adjustment" negotiations and funding.
12.43 In Sri Lanka adversity had, in fact, led to a concentration and simplification of aid strategy. In light of serious ethnic violence and human rights abuses in that country, CIDA decided to focus primarily on aiding non-governmental organizations to research policy issues related to good governance and to assist small entrepreneurs, instead of the traditional mix of infrastructure and institutional development projects in a multitude of sectors. ( see photograph )
12.44 Undoubtedly, CIDA could do valuable work in many fields and in many countries. But adding to an already large and complex workload could detract from CIDA's ability to deliver a core program focussed on its primary objectives. The proliferation of programs, and the experience of many years that saw CIDA staff transferred too frequently from one country to another and from one program to another, have moulded the Agency in ways that might be difficult to change in the short term.
12.45 Lack of focus makes it difficult to achieve the desired impact or build enduring partnerships. It also increases the cost of program delivery. Lack of focus hinders an emphasis on a clear strategy for building the strength of institutions and for promoting equitable distribution of wealth. It increases the administrative burden on CIDA staff. It strains CIDA's monitoring, audit and evaluation capability and increases the difficulty of measuring results. Moreover, it limits in-depth learning and development of the "core competence" of CIDA staff.
12.46 In summary, this is an issue of productivity. Given increasing resource constraints, CIDA needs a clear and specific strategy for each country, focussed on the Agency's primary objectives. At stake is the quality of CIDA's programs and projects in relation to its mandate.
CIDA's Lesson #3 CIDA is an over-regulated Agency. Too much focus is placed, and time spent, on managing the bureaucratic processes and on minimizing risk, to the detriment of development content. This leads to higher administration costs, delays, and efficiency losses. Contracting processes are too complex and cumbersome.
Our view: There is a need for CIDA to streamline the way it runs its business, including its existing controls, to be more efficient and effective12.47 We share the view of CIDA's Strategic Management Review that CIDA staff have been devoting more attention to process than to the substance of development. On the one hand, CIDA had resisted being held accountable for managing results rather than process alone, but on the other, it objected to the proliferation of process-oriented controls. This presents a dilemma. It is unlikely that controls agreed to with the Treasury Board can be discontinued before CIDA demonstrates its ability to achieve value for money with development results. There is a need to incorporate flexibility while retaining essential controls. If CIDA could demonstrate that the benefits that accrued from the investments in developing countries were greater than the costs it incurred, one might expect that the pressure for controls on its processes would be lessened, and its ability to innovate would be less constrained.
12.48 One aspect of the dominance of desk administration is the limited contact that CIDA officers have in the field with the recipients of their projects. CIDA project team leaders' site visits, or other visits to the field, tend to be far too superficial to be effective. The Agency's general decentralization in the late 1980s was supposed to address this problem, but to CIDA that effort proved to be too costly and has since been reversed.
12.49 CIDA staff have few opportunities to develop, through direct experience, their "core competence" in social and economic development. The Strategic Management Review pointed this out, noting that management processes have grown in complexity over the years and that CIDA staff have devoted their time increasingly to managing the paper flow. "Hands off" management has become the order of the day, without adequate recognition of either the consequent limitations on learning or the increase in operating costs. For example, all six projects we examined in Pakistan contracted out all phases of the work: project design, implementation, monitoring and evaluation. In some cases, contractors were engaged to manage other contractors. There is nothing inherently wrong with contracting out, particularly where specialized expertise is needed and where it is cost-effective. However, CIDA needs to recognize that, like all government departments and agencies, it is answerable for the cost and performance of its contractors.
12.50 Another factor that limits the "core competence" of the Agency is the frequent rotation of staff without apparent concern for loss of expertise. CIDA's officers are shifted from one country to another frequently, and can find themselves managing projects in entirely unfamiliar economic sectors in a country where the culture and language are also unfamiliar. Excessive rotation can severely limit their contribution to the Agency's performance. Some way must be found to enable CIDA staff to remain working for longer periods in regions where they have developed in-depth knowledge and understanding over a number of years, and in types of projects where they have expertise as well. If CIDA's staff remain generalist administrators, who may be managing a women-in-development rural credit project in Indonesia one year and a geological survey project in Nepal the next, they inevitably will be focussed on desk administration and buried in paper rather than substantially involved in generating development results in a particular country.
12.51 Short posting tenures of two or three years also limit knowledge and lead to a disassociation of individual CIDA staff members from the results of projects and programs. The projects we examined had life cycles varying from three to fourteen years. Longer tenures, say four to six years, and greater delegation to local staff could be considered, as could the identification of milestones within projects to facilitate the staff's accountability. US A.I.D., for instance, posts its staff in countries like Pakistan and Sri Lanka for four years. The life of most projects we looked at was much longer than the typical tenure of a CIDA officer in a particular post. For example, CIDA's involvement in land reclamation projects generally requires at least a ten-year commitment. In a project we examined in Pakistan, there had been a succession of four different project team leaders, four different field officers, and three different monitors. When one considers that the project is often continued through a number of phases, it is clear that the total number of staff responsible at some time for some part of the project over its life is very large. A long-term commitment by a stable CIDA management team with particular knowledge in the project is rare. An approach that links staff assignment lengths with evaluable results is needed.
12.52 In summary , this lesson from CIDA's Strategic Management Review and our views thereon point toward the need for a new control structure for the Agency. There is a need to simplify its management process and make individual staff much more answerable for managing results of clearly identifiable activities assigned to them. To do this, CIDA needs to modify its administrative policies, systems and practices. It also needs to link changes in administrative practices to improvements in staff capability to manage for results, become more adaptive and assume more personal responsibility. At stake is the Agency's ability to manage efficiently and effectively.
CIDA's Lesson #4 The existing project management approach needs significant modification.
Our view: CIDA's approach to project management needs to be revised. More innovative project management skills and practices are needed12.53 CIDA's approach to project management was designed mainly to deliver replicable and highly structured capital projects, such as roads and bridges. This engineering "blueprint" for funding, staffing and managing CIDA's activities cannot be applied in a mechanical way to many of the newer types of CIDA projects, such as human resource development, policy advice, structural adjustment or balance-of-payment support. These are much "softer" in nature, where results can be demonstrated only over a longer period of time. The added complexity of such projects demands higher levels of project management skills and cost consciousness throughout the Agency. The nature of the risk associated with human resource development and policy influence calls for a much greater understanding of the cultural, political and social implications involved. A good understanding of local culture and values becomes critical to assessing the appropriateness of expertise and technology provided by Canada. The environment in many respects is more demanding. For example, professionals in recipient countries are more inclined to challenge the appropriateness of donors' advice and proposals. The CIDA officer now needs to have more expertise in development planning, as well as more expertise in the country and the nature of the project than was previously the case.
12.54 CIDA needs to develop the capacity to deal with the new kind of risk and uncertainty without abandoning traditional project management disciplines. More flexible project management and funding arrangements may be needed. However, they must be accompanied by a wider set of skills and knowledge of development priorities, more rigorous planning and better answerability of individual staff for achieving the results CIDA wants, on time and within budget. The nature of control may change but not the need for it. For example, if CIDA is going to continue to fund projects, say, to train nurses or to help a country develop and implement a national environmental strategy, it will need very careful and continual thinking through of all the tasks, steps, and performance indicators needed over an extended period to monitor progress and eventually to obtain meaningful results. If the new direction in CIDA is to enhance the long-term capacity for policy development in the recipient country, a stronger commitment on the part of CIDA and its counterparts in developing countries is also needed. Continuity of staff needs to be ensured in order to build longer-term relations with local officials. These are prerequisites for effective project management. It is not simply a matter of completing a technical task according to a blueprint, funding the activity and assuming that the desired results will follow.
12.55 In summary , CIDA's growing involvement in a wide range of "softer" projects requires that its staff maintain traditional project management skills while systematically developing the mindset, knowledge and skills necessary for the new types of projects. At stake is the Agency's ability to demonstrate achievement of results.
CIDA's Lesson # 5 CIDA has not appropriately acknowledged its accountability to Parliament for the management of development results.
Our view: Accountability needs to be clarified12.56 We feel strongly that CIDA's accountability to Parliament with respect to managing for results needs to be clarified. There are varying types of delivery mechanisms and financial arrangements that CIDA enters into with its partners. There is a need to clarify the respective accountability of CIDA's staff and its partners for obtaining results, as illustrated in Exhibit 12.3 . This would have a beneficial effect on the Agency's performance and the motivation of its personnel.
12.57 Until recently CIDA did not fully acknowledge its accountability to Parliament for managing in a way that would produce desired results - that is, for obtaining value for money. Its reluctance arose from a keen awareness of the uncertainties and risks that are common in the general environment of development projects and the multiple jurisdictions involved.
12.58 Agreements with partners in development are not enforceable in practice. CIDA has not focussed on the results that are to be achieved or that have been achieved. The "audit trail" often ends with intermediate organizations funded by CIDA.
12.59 The general memorandum of understanding usually signed with recipient governments does not spell out specific development objectives or results that are to be pursued, or how accountability will be served.
12.60 At the project level, memoranda of understanding with the appointed bodies of recipient governments - usually government institutions - are vague about development results.
12.61 One consequence is that costs become disassociated from development benefits. Projects are budgeted, and budgets extended, often without explicit comparison of costs and benefits. A project that goes smoothly and has some benefits can, when no explicit comparison of costs and benefits is visible, seem a success. We do not expect CIDA to adopt a simplistic "rate of return" criterion for project funding, but it needs some systematic method of assessing a project's potential for meeting the Agency's development objectives. We refer to one such method in paragraphs 12.90 to 12.95.
12.62 CIDA's external accountability for the stewardship of funds entrusted to it would be better served if internal accountabilities also were made more explicit. Within the Agency, we visualize CIDA's intervention in a country as requiring at least five levels of management accountability. In such a case, the Vice-President would be accountable to the President for managing the overall geographic Branch performance in terms of both the quality of country-specific strategies and objectives and the extent to which they are met in the Branch. Second, the Country Program Director logically would be responsible for managing the operational plan for his/her country and would be held to account for managing in a way that would produce results in that country. Third, the Project Team Leader would be responsible for specific projects and be held to account for achieving results at this level, particularly self-sustainability. Fourth, a specific role would have to be clearly established in the field for providing on-site monitoring of progress toward results, in some cost-effective way. Fifth, the President would be accountable for maintaining an organizational environment that promotes an action-oriented, learning culture throughout the Agency.
12.63 In summary , increasing budget constraints and a greater emphasis on results make it crucial that CIDA clarify the parameters of its accountability for achieving results. It also needs to come to clearer understandings with recipient governments about what specific objectives can be realistically pursued, what the Agency can reasonably be held answerable for, and those aspects for which the recipient government or others will be held to account. At stake is CIDA's ability to obtain worthwhile development results.
CIDA's Lesson #6 CIDA needs to be more open and transparent about its policy objectives, programming decisions, and the related outcomes.
Our view: CIDA needs to be more transparent about what it is trying to achieve and how well it is doing12.64 We stress the need for open discussion of CIDA's policies, programs and accomplishments. When multiple objectives are pursued, parliamentarians need to be informed about the relative significance of the objectives and how they impact on the overall outcomes or results to be achieved. CIDA's policies, plans and the extent of its accomplishments should be open to anyone with a legitimate interest; as well, CIDA needs to make a greater effort to communicate with Parliament and with people and organizations in recipient countries. Reporting to Parliament should focus less on inputs and more on accomplishments and lessons learned.
12.65 The purpose of CIDA's aid and the cost of delivery needs to be made clear in the vote structure of the Estimates. If the primary purpose is humanitarian or motivated by trade or political considerations, as opposed to economic and social development assistance, this should be reflected clearly in the expenditure structure; the cost of delivering programs should also be truly reflected. We note, for example, that the United Kingdom's Overseas Development Administration in its budget papers shows its provision for Aid and Trade as a separate allocation within the bilateral aid program.
12.66 CIDA also needs to use plain language to communicate within the Agency and to stakeholders outside. For example, it needs to explain in simple words exactly what it means when it uses all-encompassing terms such as development, sustainable development, and partnerships to describe its objectives and operating environment.
12.67 To assist the Minister in fulfilling his or her accountability to Parliament, Agency management needs to make available information on the achievement of results, on a periodic basis.
12.68 In summary , CIDA needs to be more forthcoming with full and frank information to the public in Canada and the recipient countries on what it is trying to achieve and how well it is doing. At stake is the credibility of the Agency, both with the people of Canada and in the developing countries with which it works.
Meeting the Challenges Ahead
CIDA recognizes the need for reform and has outlined a sequential approach to meeting the challenges ahead12.69 The Strategic Management Review of the Agency that was carried out during the period 1991 to 1993 has come to a close. A three-part action plan to implement its recommendations has been drawn up by the President of the Agency, and individuals responsible for undertaking the specific tasks involved have been identified. A timetable for implementation has also been established. Actions that are within CIDA's control will be implemented forthwith. In due course, CIDA also proposes to consult actively with its partners about changes that more directly affect their operations and relations with the Agency. A logical step will be a review of issues facing the Government of Canada on Official Development Assistance policy and CIDA's mandate.
12.70 CIDA anticipates that most, if not all, aspects of the first part of the action plan will be implemented before the end of the 1993-94 fiscal year. It expects that implementing them will:
- improve the Agency's corporate planning function, its accountability framework and its capacity for performance assessment. The objective is to introduce centralized planning and put in place a results-oriented management approach in lieu of the present emphasis on process and inputs.
- strengthen its policy-making capacity. The intention is to establish a broadly mandated Corporate Management Branch and rationalize the corporate committee structure, with the objective of achieving greater direction, coherence and efficiencies in overhead management activities.
- provide Program Branch managers with the delegated authorities, organization and resources required for more effective management. This will include planning, programming and delivery of the various programs consistent with Agency policies and intended results.
Need to Manage for Results12.71 In addition to the lessons CIDA has identified in its Strategic Management Review, there are areas in the management of the Official Development Assistance program that we believe need to be improved in the immediate future. The intent is not to add more processes and controls, but to redefine the existing administrative process while strengthening certain controls that are critical to achieving value for money, thereby enhancing CIDA's ability to manage for results. At issue is the lasting value of CIDA's interventions.
A better focus on what results realistically can be expected in each country12.72 The recipient country is an appropriate level at which the question of results can be addressed because it is the level where bilateral funding allocations are made by the Minister, where the strategy for results can be determined, where CIDA can focus attention in its planning and programming, where goals are set and where bilateral agreements are signed for programs and projects and the attendant commitments are made.
12.73 The first critical area for improvement is to bridge the present gap between the very general policy framework that CIDA has developed to guide its country programs and the specific individual projects within the country project "portfolio". Problems in different countries and regions call for different strategies: what works in Pakistan may not work in Sri Lanka, a country in the same region. It is important to be specific about strategies and objectives that can be pursued in a particular sector and to be clear about the results that reasonably can be expected.
12.74 As we have noted, CIDA programming is often influenced by a number of interest groups with diverse agendas. The choice and mix of projects delivered must be linked to the strategic development objectives for a country or sector to achieve meaningful results. Weak linkages can lead to a diffusion of activities, confusion among CIDA staff and a loss of impact, as we saw in CIDA's Bangladesh and Pakistan country programs.
12.75 CIDA's approach to planning so far has been to develop a broad policy framework for each of its major recipient countries but no operating plan to accompany it. The framework for Bangladesh, for instance, outlines economic, social, cultural and other policy issues as a basis for identifying the country's stage of development and the sectors where Canada has the capacity to respond to the need. The constraints in the policy environment that CIDA will face are listed. But because the framework does not include specific operating objectives and results to be achieved, its use as an accountability document is limited.
12.76 We expected to find in each country policy framework a statement of strategy that outlined specifically what CIDA expected to achieve over time and was willing to be held accountable for. We also expected to find an operating plan outlining the specific objectives and results - whether at the sector or other level - that CIDA wanted to achieve in each country, and how the mix of projects would accomplish this.
12.77 We observed that the country policy framework does not outline specifically and realistically what CIDA expects to achieve over time. The framework as currently structured can be interpreted too broadly. It is not at all clear how the collection of diverse projects undertaken in a country or sector will lead to a definitive result in relation to CIDA's mandate and its purpose for being in the country.
12.78 In our view, there is a gap between CIDA's broad policy framework for a country and its projects in that country. To bridge the gap it needs a policy framework that clearly articulates its specific strategy for the country, and an operating plan that links that strategy to specific objectives and intended results. This link could provide the discipline CIDA needs to target its efforts and reorient resources and staff in the most effective and productive directions. It would also allow it to direct its future activities in any given country to what Canada wants to achieve there over time, on its own or with other donors and with its partners.
12.79 In summary, a country-specific strategy linked to an operating plan would enable CIDA to focus on specific and measurable results, country by country. It would also allow CIDA to discuss its operations in a more practical way with its partners and host country officials.
Promoting enduring benefits through more self-reliant development12.80 In order to understand whether the programs and projects funded by the Agency were meeting Canada's expectations for development results, we asked three questions:
- How does CIDA define overall results at the project, country and Agency levels?
- What indicators does CIDA use to measure the potential for development results?
- What reporting system does CIDA use to communicate the achievement of development results up the line?
12.82 At the country level , lacking the kind of operating plan we have suggested, the Agency has not defined indicators to measure its performance in each country. It is thus difficult to report whether Canada is getting benefits commensurate with its investment in that country.
12.83 At the project level , CIDA generally monitors its performance by assessing whether the inputs agreed to have been delivered. It does not measure development results. Consequently, knowledge about the quality of individual projects is not gathered and assessed.
12.84 Development requires that, after the termination of aid funding, something of value be left behind that can be sustained and that enables countries to better manage their own development. This is fundamental to achieving the goal of "helping people help themselves", i.e. to become more self-reliant. Our review of best practices suggests that one indicator of results that can be applied is whether the project has the potential to promote enduring self-sustainable benefits, namely "the ability to deliver an appropriate level of benefits for an extended period of time after major financial, managerial and technical assistance from the donor is terminated." As acknowledged in this statement by the Development Assistance Committee of the Organization of Economic Co-operation and Development, sustainability is an appropriate indicator for judging a project.
12.85 In this regard, the experience of some donor agencies suggests that monitoring certain indicators of project self-sustainability ( see Exhibit 12.4 ) at appropriate milestones can provide an early warning system, to help managers identify and correct problems related to sustainability before they become unmanageable. We used these indicators to assess the potential of 18 CIDA projects to become self-sustainable and thereby promote more self-reliant development. Such indicators are particularly important in many of the new kinds of projects CIDA is undertaking in social sectors, where results are softer in nature, and in the policy advice area, where development results can be proven only in the long run.
12.86 We have two main observations at the project level.
(a) In our analysis, we applied indicators of self-sustainability that are relevant to CIDA and practical to use. We believe these indicators to be reasonable measures of a project's potential to promote enduring self-sustainable benefits, although others may exist. Such an approach would provide knowledge on the extent to which development results are likely to be achieved. It would also show the degree to which CIDA has been able to strengthen the capacity of its partners to deliver its projects. How best to use such indicators in a variety of circumstances may require ongoing refinement.
(b) Promoting enduring benefits through greater self-sustainability warrants more attention from management. The availability of financial and other resources to sustain CIDA's projects after the Agency's withdrawal was not assured in a large number of the cases we examined. Current performance has not adequately maximized the use of resources and has not led sufficiently to self-reliant development.
12.87 Overall, our analysis shows that the potential for enduring benefits is less than was anticipated when the projects were first approved.
12.88 Specifically, among the 18 projects we examined:
- The technical and managerial capability in the recipient country after CIDA's withdrawal ranged from satisfactory in some cases to weak and unsatisfactory in most others, depending on the country, the sector and the type of project.
- The commitment of the "inheritor" was generally satisfactory in the initial stages. It grew weaker over time in many cases, and was absent in some after CIDA's withdrawal from the project.
- The involvement of the target group of people in project design and evaluation varied considerably - from high in a few cases to low in most.
- Although the first three indicators are important in themselves, the fourth - the availability of sustaining resources to operate and maintain the project over its life cycle after CIDA withdraws - is critical in determining whether the project will be self-sustainable. It merits serious and ongoing attention. At the time of our audit, a few projects were clearly sustainable; the majority will require continued host government or donor support; and some are unlikely to be financially self-sustainable.
Improvements needed in the way CIDA manages its operating risks12.90 Quality control at entry . Factors, whether qualitative or quantitative, that could eventually impact on the Agency's ability to achieve the results it expects need to be rigorously assessed, before funding is approved and at appropriate subsequent milestones. Management of risk is required at both the country and the project levels. CIDA would benefit from a more objective review of a project's potential to promote self-sustainability before approving the funding. In making this point, we are not implying that CIDA does not scrutinize projects before approval, nor are we proposing that it set up a new system of review or add more paper and process. We are pointing out that CIDA needs to obtain some assurance up front that the risks associated with an activity it is prepared to fund are commensurate with the benefits expected. There are at least three specific areas where further improvement is needed.
12.91 First, CIDA needs to examine thoroughly the capacity of the recipient country and its main public institutions to absorb development assistance effectively. We found this factor to be paramount particularly in Bangladesh. That country could not possibly absorb and use in a cost-effective way all the development assistance it was already getting from multiple donors. This point has been acknowledged as a matter of great concern by many prominent Bangladeshi economists. The World Bank has identified effective use of aid as a key prerequisite for increasing economic growth and making lasting progress in alleviating poverty. Most projects we looked at suffered because the host government or its parastatal body was unable to deliver on its commitments. Often projects were funded simply because funds were available from donors, irrespective of past difficulties in obtaining the necessary support from the host country.
12.92 Second, there must be testing by CIDA of the assumptions it makes to support the recipient country's commitment, and increased participation of the targeted beneficiaries in the identification and preparation stages of the project. Commitment of the recipient country stems from a better understanding of the project's potential for enduring benefits. As well, ownership of the project is tied to meaningful early and ongoing participation by the beneficiaries. ( see photograph )
12.93 Third, the quality of risk assessment at the project level needs to be improved. All projects do not have the same level of risk associated with them; therefore CIDA's project management approach needs to be tailored to the type of project being undertaken and to the host country's institutional capabilities. High-risk projects warrant more monitoring than others, with appropriately placed milestones for additional funding.
12.94 The experience of the World Bank and other donors suggests that both the number of project components and the number of co-funders increases the risk of unsatisfactory performance. However, there may be a bias among donors toward complexity, partly because of the urge to include as many novel features as possible to increase a project's chance of acceptance, or because often it is the only way to "buy in" to a multi-donor project. This kind of bias needs to be addressed in the early stages of project approval.
12.95 The right questions have to be asked, such as how will the indicators of self-sustainability ( Exhibit 12.4 ) be satisfied, and what is the risk that they will not? CIDA project documentation identifies critical assumptions on which the long-term success of the project will depend. However, by and large these qualitative assumptions are stated but not sufficiently analyzed. Risks often are not seriously taken into account. Even when assumptions subsequently are proven unrealistic, we found that corresponding adjustments are not made on a timely basis in subsequent phases of the same project. The Small Scale Water Control project in Bangladesh is an illustration: in the third phase of this project CIDA continued to make the same assumptions about the Bangladesh Water Control Board's commitment and ability to provide technical and managerial capability and sustaining financial resources, despite indications that those assumptions were proving to be too optimistic. ( see photograph )
12.96 An early warning system to track potential for results . At present CIDA does not have a reporting system, even for its riskiest projects, that provides results-oriented and timely information on whether its projects are performing as well as expected. For instance, there is no monitoring of indicators of self-sustainability. At CIDA we note a serious lack of incentives to ensure that benefits will accrue after assistance ends. Information on projects is provided to CIDA managers periodically by executing agents, monitors and/or evaluators. The information is generally routine and deals with process. This falls considerably short of what is needed, namely, an early-warning system that will provide explicit data periodically up the line on what projects are in difficulty, and why. Solutions for alleviating difficulties could then be explored more selectively. If CIDA were to concentrate its efforts, as we have suggested, on fewer projects in fewer countries and with fewer "agents", the cost of introducing such a reporting system that highlights management's effectiveness would not be excessive.
Institutionalizing lessons learned12.97 Generally, learning organizations display four important characteristics: an organizational culture that stresses learning, places value on it, encourages it and rewards it; an institutionalized but simple and clearly understood mechanism that allows lessons learned to be captured and accessed; an environment that actively encourages learning by doing; and finally, a requirement and a responsibility to use lessons learned as part of the normal management process.
12.98 We observed that the Agency does not have an organizational "learning culture". Staff often are less than open in responding to questions about what problems were encountered in the implementation of their projects. Heads of Aid located in the field often are not fully aware of how their projects are progressing in terms of promoting self-reliant development. Because the Agency does not formally stress learning, it is difficult to determine how much and what kind of learning is taking place. As we have already noted, staff rotation policies and the contracting regime also inhibit learning.
12.99 We found that lessons learned are not incorporated in a systematic and timely way into operations. Internal audits, project termination reports and end-of-project evaluations are used very unevenly to guide future activities or to highlight what works to achieve results and what does not.
12.100 CIDA has acknowledged that there is no formal mechanism or requirement to capture lessons learned or to use them. CIDA's learning culture, we were told, is oral and staff will consult with each other informally as and when necessary.
12.101 For CIDA to transform itself into an action-oriented learning organization, it needs to institutionalize lessons learned at the project and country levels. It is important to understand why programs and projects are meeting or not meeting the Agency's expectations.
Upgrading CIDA's internal audit and project evaluations12.102 Internal audit and evaluation of development projects can play a critical role in ensuring the credibility of aid agencies, legitimizing development assistance to various constituencies, and improving the performance of the Agency. We observed, based on our review of the 18 projects in Bangladesh, Pakistan and Sri Lanka, that at present this is not the case. The benefit of audit and evaluation as a tool for wider learning in the Agency is not being maximized. Greater interaction, outreach and sharing of examination results are needed.
12.103 Internal audit: an emphasis on significance. There is a need to be more selective in identifying issues for audit. If internal audit is to play its role in the new accountability regime at CIDA, it needs to be focussed systematically on the issues most significant to achieving results.
12.104 The Audit and Evaluation Division has seldom undertaken significant internal audits of CIDA's country programs. Most internal audits tend to focus on the Agency's administrative and project management procedures rather than managing to achieve development results; such examinations can be useful if the objective is to assess whether major systems are working. There has been no specific requirement for internal audit to present its overall findings on the significant economy and efficiency issues that have been facing the Agency. Many of the significant lessons identified by the Strategic Management Review, for instance, had not been highlighted by internal audit.
12.105 The Agency could improve the value of its audit work and the significance of its audit findings by adopting a more selective approach: for example, determining whether appropriate mechanisms are in place (and are working) for monitoring results in high-risk situations could replace the routine examination of project management procedures. Also, delegation of audit to the host country could be explored, where appropriate.
12.106 Project evaluations . Bilateral Branches spend about $4 million each year on project evaluations. We reviewed those that related to the 18 projects we examined in depth. Most project evaluations were commissioned by the country program staff, not by the central Audit and Evaluation Division. We noted differing standards and terms of reference in the project evaluations we encountered. Also, most project evaluations were not focussed on the potential of the project to become self-sustainable. Nor are the results of evaluations systematically fed into future project appraisals - standard practice in the World Bank.
12.107 An increased emphasis on quality. There is a need to improve the quality of project evaluations undertaken by the operating branches in the Agency. Project evaluations normally are conducted on an ex post basis. The ability to monitor and evaluate the potential for results more selectively, so that changes in the environment are discerned and activities suitably adapted, needs to be strengthened if evaluations are to contribute to CIDA's strategic management capacity. An internal review of project evaluations conducted in 1991 showed that, while one third were fully satisfactory in methodology and implementation, the others were not. The principal weaknesses pointed out were: unstructured data collection; insufficient time spent in host countries, including in particular a lack of intensive on-site consultations with project beneficiaries; and a lack of explicit comparisons of the costs and benefits associated with CIDA's investments. Yet we noted that, at times, even when the quality of the project was not assured, conclusions were drawn that were used to justify more funding.
12.108 There is no ongoing quality assurance function in CIDA that examines whether minimum evaluation standards have been met. CIDA contracts out most project evaluations to the private sector. We believe that, in at least some of these examinations, CIDA could consider using "mixed teams" of CIDA staff and Canadian or local contractors, where appropriate, as a way to ensure more consistent quality and reduce costs.
12.109 Emphasis on sharing project evaluation results. Results of examinations are not widely shared with others, for example, recipient governments and other partners involved in development projects. Thus, the benefits of project evaluations as a tool for learning are not being maximized. It is difficult to tell how useful project evaluations are for judging their contribution to the learning process in the Agency.
12.110 There is also a need to guard against a perceived lack of objectivity. Best practices would suggest re-examining how this function should be done and located in the organization. Some organizations, like the World Bank and US A.I.D., assure quality by separating the evaluation function from line management. Others use independent reviewers.
Better internal control over contributions involving counterpart funds12.111 CIDA provides bilateral funds to other organizations to spend for development purposes in accordance with conditions specified in contribution agreements. Certain contributions are donations of food and commodities, which are converted into "counterpart funds" at an agreed value and set aside in local currency by recipient governments for use in development activities. Other contributions are advance cash payments to Canadian, international or recipient country non-governmental organizations.
12.112 We examined three counterpart funds in Bangladesh and Sri Lanka totalling $200 million over a four- to five-year period. We found that CIDA did not have adequate assurance that all moneys had been spent for the intended purposes and had been accounted for fully. CIDA relied on unaudited statements of account submitted by the recipient governments, some of which were not complete or submitted on a timely basis. This is contrary to CIDA's policy on the management of counterpart funds. CIDA's policy notes that independent audits are one way to gain assurance on whether conditions associated with the counterpart funds have been complied with. ( see photograph )
12.113 We brought this issue to CIDA's attention with specific examples to illustrate our main concerns. The Agency has recently undertaken to re-examine its practices and to implement new procedures by January 1994. It has also decided to consult with our Office to explore the possibility of engaging the services of the national audit offices in developing countries, where appropriate, as a way to strengthen the accountability of its partners.
12.114 Auditing is not absolutely necessary to achieve accountability for counterpart funds. There may be other ways. But auditing is one of the more effective ways to strengthen accountability in situations where the risk of non-compliance with conditions can be high.
12.116 Our audit coincided with a Strategic Management Review conducted by the Agency at the request of the Minister. Both examinations identified some important lessons from CIDA's twenty-five years of experience. Generally, these lessons point to a need for attention to self-reliant development and a more results-oriented, focussed, businesslike and accountable style of operation. Most important, they also call for dedication to the basic objectives - fighting poverty and promoting enduring benefits through an increased emphasis on helping people to help themselves - that Canadians have affirmed repeatedly over many years.
12.117 There needs to be more concentration, in the bilateral country programs, on those countries and activities where Canada can have the greatest impact. At present there is too much dispersion. As well, there is a need to pay more attention to the quality of aid provided, and a need for an environment within the Agency that will encourage the development of an action-oriented learning organization. None of these changes will be easy to make, since they require a new mindset, skills and culture, and strong ministerial support. Their successful implementation will depend on the will to reform, not only among the management and staff of the Agency but also among those who act as its development partners, in Canada and in the developing countries where CIDA operates.
CIDA's response and commitment
1. The Agency agrees:
- with the Auditor General's findings and with the thrust of the recommendations aimed at improving the Agency's accountability and strengthening its management effectiveness.
- that CIDA's ability to do this will require not only the support of the government but also the support of its development partners both in Canada and in recipient countries.
3. The Auditor General is invited to monitor the implementation of the Agency's reforms with a follow-up Report to Parliament in 1995.