This Web page has been archived on the Web.

2001 December Report of the Auditor General of Canada

Main Points

9.1 We looked at how well population health projects were managed in three programs of Health Canada's Population and Public Health Branch. We found inconsistent management practices.

  • The projects in the Canada Prenatal Nutrition Program were generally well managed.
  • Many projects in the HIV/AIDS strategy were poorly managed, regardless of the dollar amount funded. For example, projects were approved despite concerns identified in the selection process, and the Branch failed to take timely action to remedy identified problems.
  • Six large national projects in the Population Health Fund suffered from specific and significant problems. In particular, management did not have the proper authority to fund projects for prostate cancer research and enhanced fitness activities. Further, these projects were not subjected to the required selection process.

9.2 Choosing the "right" priorities at the outset is a critical step toward committing resources to areas that will yield the most benefit in improved population health. The Population and Public Health Branch has developed a framework or process for setting its priorities. However, the framework has never been approved and there is no plan or timetable for implementing it.

9.3 The evaluation of population health programs is generally weak. We found that the Department had made little progress in developing performance indicators for most programs. Such indicators are central to any meaningful evaluation exercise. Objective information on the effectiveness of the three programs that we looked at was limited. The information that Parliament receives on these programs does not provide a clear picture of the extent to which the $225 million spent on the programs has improved the health of Canadians.

Background and other observations

9.4 The federal government provides leadership and support for preventive health activities. These include health promotion, disease and injury prevention, health protection, and population health surveillance and assessment. Health Canada's Population and Public Health Branch has primary responsibility for supporting the federal initiatives.

9.5 The Population and Public Health Branch funds projects across the country that are delivered by other parties at the community level. Investment in these projects plays an important part in promoting quality of life, reducing costs to the health care system, and avoiding the loss of productivity due to ill health and injuries.

9.6 Our audit examined how the Branch sets priorities in choosing which population health programs to fund. We looked at the Branch's process for selecting, approving, monitoring, and evaluating a sample of projects in three specific programs: Canada Prenatal Nutrition Program; HIV/AIDS strategy; and the Population Health Fund. We also examined how the achievements of these three programs are measured and reported to Parliament.

9.7 The Branch does not have all the required evidence-based information that is critical for setting priorities. Sources of information include surveillance data and program evaluation. We found that the Branch lacks some of this information because it is either not co-ordinated or not available.

9.8 The Branch has a good process in place to manage its grant and contribution programs and ensure that public funds are managed properly. However, we found that management did not consistently follow its own management process.

9.9 In the fall of 2000, Health Canada's Internal Audit Directorate reported on an audit of the management of grants and contributions in the Population and Public Health Branch. One year later, management formally agreed to the recommendations and established an action plan. Timely implementation of the internal audit recommendations is crucial to correcting the identified weaknesses and improving the overall management of grant and contribution programs.

The Department has responded. Health Canada's responses to our recommendations are included in this chapter. The Department has responded positively to our recommendations and has agreed to take corrective action. In some instances, this action is already under way.

Top

Introduction

The federal government plays an important role in preventive health activities

9.10 The federal government plays an important role in providing leadership and support for preventive health activities such as health promotion, disease and injury prevention, health protection, and population health surveillance and assessment. Its investment in these activities helps to promote quality of life, reduce costs to the health care system, and avoid loss of productivity due to ill health and injuries. Preventive health activities are estimated to be 6 to 45 times more effective than dealing with health problems after the fact. The federal government has established programs across the country to support these activities.

9.11 Health Canada provides leadership and support for programs that promote good health and protect Canadians against various health risks and the spread of disease. The provinces and territories are also responsible for providing many public health services. While each province and territory has a legislated public health function, at the national level there is no public health legislation that co-ordinates activities in the provinces and territories. Instead, co-ordination is exercised through joint activities involving various players, including the federal, provincial, and territorial ministers of health.

9.12 Health Canada's formal mandate is described in the Department of Health Act. The Department's powers lie in more than 20 acts that the Minister of Health administers, entirely or in part. The mandate, summarized in Health Canada's mission statement, is "to help the people of Canada maintain and improve their health." To achieve this mandate, Health Canada has an annual budget of $2.3 billion and employs 7,100 staff (see the Department's organization chart, Exhibit 9.1).

Responsibility for federal population health initiatives

9.13 The goal of Health Canada's Population and Public Health Branch is to maintain and improve the health of Canadians. One way to do this is by funding programs that help keep people healthy.

9.14 The Branch spent roughly $350 million in 2000-01, or about $11 for each Canadian, and employed about 1,200 people. This funding supports all of the activities of the Branch, including population health programming. The Branch operates from offices in Ottawa and in six regions—Atlantic, Quebec, Ontario and Nunavut, Manitoba and Saskatchewan, Alberta and Northwest Territories, and British Columbia and Yukon—and two national laboratories.

9.15 The Branch spent $225 million of its funding for 2000-01 on population health programming through transfer payments to support 12 grant and contribution programs. At 31 March 2000 it was managing around 1,800 grant and contribution agreements.

Canada was a pioneer in health promotion

9.16 A 1974 report by Health Canada, A New Perspective on the Health of Canadians, argued that better health could be achieved through a healthier lifestyle, better nutrition, and a healthier environment. It said that human biology, environment, lifestyle, and health care organizations were the four main elements that affect health. That report led the federal government to direct its policies toward improving the health of the population (for example, making seatbelt use mandatory). It focussed on issues related to individual lifestyle choices, such as exercise, diet, and smoking. In effect, the report acknowledged that contributing to the health of the population meant much more than merely supporting a health care system; it also meant promoting health in a number of non-medical ways.

9.17 In 1977, the federal, provincial, and territorial health ministers identified health promotion as a priority for joint action. They agreed that health promotion was legitimately a federal activity; it avoided interference with provincial and territorial responsibilities and complemented the health care system. Indeed, health promotion was seen as an opportunity to help the provinces and the federal government control and reduce the costs of the health care system.

9.18 The mid-1980s saw a growing concern about the limitations of health promotion efforts, which were largely directed at influencing the behaviour and lifestyles of individuals. It became apparent that other factors (or determinants) such as income, employment, social status, and housing also affect health. The growing emphasis on these and other non-medical determinants of health coincided with the 1986 report by Health Canada, Achieving Health for All: A Framework for Health Promotion.

9.19 In 1986, the first International Conference on Health Promotion was held in Ottawa. At this conference, the World Health Organization adopted the Ottawa Charter for Health Promotion. The Charter broadened the interpretation of health promotion to include the various determinants of health. It has since been translated into 50 languages and has become a guide for health promotion worldwide. The budget of the Health Promotion Directorate at Health Canada nearly tripled in 1987, as new national strategies were developed in the areas of drugs, tobacco, impaired driving, and AIDS. In the five years following the Ottawa Charter, health promotion continued to advance.

Health Canada adopts the population health approach

9.20 By the early 1990s, a population health approach began to replace the health promotion approach in many government and health policy circles. The approach expanded on the determinants or factors influencing health and included the effects of conditions in the social and economic environments beyond the health care system. It was recognized that population health programs would have to encompass all of the determinants of health (Exhibit 9.2), many of which are outside Health Canada's mandate.

9.21 Using the population health approach, Health Canada tries to establish programs that help maintain and improve the health of the entire population and reduce disparities in health among different population groups. It directs its programs toward "at risk" Canadians (for example, inactive adults, pregnant teens, Aboriginals, smokers, intravenous drug users, and street youth). It often establishes these programs in collaboration with key partners in other government departments, provinces and territories, and the non-government sector so that all the determinants of health can be addressed, even those outside Health Canada's mandate. Exhibit 9.3 lists some of the key partners in the population health approach.

9.22 In July 2000, a reorganization of the Department created the Population and Public Health Branch to advance this approach. The Branch is now responsible for policies, programs, and systems related to prevention, health promotion, community action, disease surveillance, disease control, and the Department's emergency preparedness and response capacity. These responsibilities include most elements of the previous Health Promotion and Programs Branch, the former Laboratory Centre for Disease Control (responsible for national surveillance of diseases and injuries), and some responsibilities of the former Health Protection Branch. The inclusion of the Laboratory Centre for Disease Control brings together activities related to identifying and monitoring emerging and existing illnesses with health promotion and protection activities in the same unit of Health Canada.

9.23 The Population and Public Health Branch currently funds 12 programs through grant and contribution agreements. Exhibit 9.4 lists these programs.

9.24 Surveillance and the information it generates can provide a means of identifying possible links between a disease and, for example, what we eat, where we live, or what our social status is. Surveillance activities link the scientific and technical knowledge of many institutions and disciplines to promote better health and to prevent and control communicable and chronic diseases and injuries. The surveillance work of the provinces, territories, and a variety of federal departments and non-government organizations generates data for input to the Branch's surveillance and control systems.

Focus of the audit

9.25 The purpose of our audit was to find out how the Branch decides which programs it should fund. We looked at a sample of projects under three of the Branch's 12 grant and contribution programs to examine how it selects, approves, monitors, and evaluates the projects it funds. Finally, we examined how the Branch has evaluated and reported the achievements of these three programs. More information on our audit objectives, scope, approach, and criteria can be found at the end of the chapter in About the Audit.

Top

Observations and Recommendations

Priority setting

Setting priorities is critical to making choices that optimize the health of Canadians

9.26 One of Health Canada's goals is to set its policy and branch priorities in a way that optimizes health outcomes for Canadians. Setting priorities means choosing which key health issues, such as cancer or diabetes, or which key health areas, such as obesity in adults or prenatal and child nutrition, should receive attention given that they all present identified risks to the health of Canadians. Setting priorities requires a process or structure for making rational, evidence-based decisions.

9.27 Such a process needs to consider all of the many health issues and the available interventions associated with them, such as educating school administrators on playground safety, and providing nutritional supplements to pregnant women. It also needs to take into account the health, social, political, and economic risks of not addressing an issue. In addition, it must recognize the many potential partners and sources of health information that provide input to decision making. Such a priority-setting process, if used on a cyclical basis, could serve as a framework for making decisions to either terminate or refocus existing programs or create new ones.

9.28 We expected that the Branch would have such a process that would enable it to set priorities on the basis of good, evidence-based information such as surveillance data and evaluations of population health programs. We expected that a decision to fund a program or not would be based on this priority-setting process. We found that although the Branch has recognized the need for such a process and recently developed one, the process has not received Health Canada's final approval and has not been implemented.

The Program Impact Assessment Project recognized the need for a priority-setting process

9.29 In 1999, Health Canada's Program Impact Assessment Project examined a number of the Department's programs and functions. It identified certain key requirements for success: a departmental culture that emphasizes performance and continuous improvement, tools for gathering evidence-based data, and a process for establishing priorities. The project included selected health issues from the Health Promotion and Programs Branch (now part of the Population and Public Health Branch). The issues were selected because they needed more attention, given their importance to the well-being of Canadians. In December 1999, the Departmental Executive Committee accepted a number of recommendations in the project's report.

9.30 Several recommendations were directed at the Health Promotion and Programs Branch. In particular, the report recommended that the Branch develop a priority-setting process by which it could compare activities and identify those on which it should focus its efforts. The report also recommended that the Branch develop the following:

  • a means of assessing the relative importance of health issues and identifying the best interventions to deal with them;
  • a body of evidence to decide which new programs to fund and whether to continue funding existing programs; and
  • a future direction and possible options for its involvement in both new and existing programs.

9.31 The Program Impact Assessment Project examined seven selected health issues. Among these were mental health, fitness and active living, sexual and reproductive health, and injury prevention.

9.32 The project analyzed what these issues were costing the economy and determined that mental health was the costliest. Health Canada's subsequent analysis of the costs associated with depression support this conclusion (Exhibit 9.5). The project also noted that limited work was being done in the area of mental health. It recommended that the Branch begin the necessary planning and analysis to determine how best to address mental health concerns. Further, it recommended that the Health Promotion and Programs Branch take the lead in developing a proposal to present to the Departmental Executive Committee no later than April 2000. This proposal had not been presented at the time of our audit. However, work on mental health has begun. For example, the Branch is developing initiatives directed at children and adolescents, stress in the workplace, gender differences, and First Nations and Inuit people.

Recently completed framework for setting priorities has not been approved

9.33 The Population and Public Health Branch's Wellness Framework provides a structure for decision-making. It provides a framework for the following:

  • measuring the health of Canadians and identifying priority health issues;
  • identifying the social, political, economic, and health risks, and the factors, conditions, and determinants of health that increase the likelihood of suffering from an illness related to the priority health issues;
  • assessing available interventions for dealing with them; and
  • developing a better process to set priorities.

9.34 Although a few decisions in the last few months have used the Wellness Framework (Exhibit 9.6), senior management has not yet approved it and neither a strategy nor a timeframe for implementing it has been established.

The Branch lacks some evidence-based information needed for setting priorities

9.35 A formal process for setting priorities requires good, evidence-based information. One key source of information on health issues is surveillance data, which are provided by the surveillance units now being integrated into the Branch. Surveillance data, along with data from other important sources, are key to measuring the health of Canadians. In a recent analysis of the health of Canadians, the Branch used the three measures shown in Exhibit 9.7. However, not all of the data needed to create two of the measures were available. They either had not been collected from the many partners that play a role in population health (federal, provincial, and other organizations) or were not organized appropriately. Therefore, the Branch had to use data from another country to measure the health of Canadians. In previous audits, we reported weaknesses in the efforts of the former Laboratory Centre for Disease Control to collect and co-ordinate surveillance data.

9.36 Information on what has worked and what has not, gathered through program evaluation, is also needed to set priorities properly. Evaluation can help determine what interventions are available, how effective they have been, and at what stage of life they produce the best health outcomes. Information is available from a number of sources, including university-sponsored population health research centres, provincial and international evaluations, and evaluations of Health Canada's own programs. However, we have identified several problems in Health Canada's evaluation activities that are similar to problems we have reported before.

9.37 Recommendation. The Population and Public Health Branch should obtain approval from the Departmental Executive Committee for a formal priority-setting process. It should also develop a strategy and timetable for implementing that process.

Department's response. Agreed. The Departmental Executive Committee has approved the Branch priority-setting process. The Branch is now implementing this new priority-setting process.

9.38 Recommendation. The Population and Public Health Branch should ensure that it has the necessary processes to collect and co-ordinate the evidence-based information it needs to set priorities.

Department's response. Agreed. Evidence-based information is a major component of the priority-setting process. A joint developmental project is in place between Health Canada and Statistics Canada that provides the Department with the capacity to collect and co-ordinate evidence-based information. This project is focussed on obtaining a more objective assessment of the relative magnitude of different diseases, injuries, and risk factors in order to balance investments. This project has been presented to the Canadian Population Health Initiative fund, and has been unanimously supported by the Federal/Provincial/Territorial Advisory Committee on Population Health.

Top

Managing funded projects

Projects funded through grant and contribution agreements

9.39 The government can pursue public policy objectives in a number of ways. It can use legislation and regulation; provide information and advice; deliver programs itself; or make transfer payments to individuals, organizations, and other levels of government.

9.40 Once the Population and Public Health Branch has identified its priorities for funding, it must decide what funding mechanism to use. The Branch delivers much of its population health programming through transfer payments to others. It has used $225 million of its budget to transfer funds to community groups, organizations, and individuals who have a direct link to the community that the government wishes to serve. As already noted (and shown in Exhibit 9.4), the Branch currently funds 12 population health programs through grant and contribution agreements.

9.41 Grant agreements transfer funds unconditionally to recipients who meet the terms of eligibility. Contribution agreements, however, have conditions attached to the funds: the recipient must not only meet the terms of eligibility but also the conditions included in the agreement. These conditions help the federal government ensure that its objectives will be achieved and that those who receive funds will manage them properly as they carry out their projects.

9.42 Our audit found that in all three programs, there is a well-established project management process and good program guidelines that clearly describe program objectives, priorities, and eligibility criteria (Exhibit 9.8). The management process helps ensure that funded projects respect the terms and conditions approved by the Treasury Board, the program guidelines, and financial authorities.

How we assessed the adequacy of project management

9.43 To assess how well the Branch manages its grant and contribution programs, we examined a sample of 38 projects from across Canada, under three of the 12 programs funded through grant and contribution agreements: the Canada Prenatal Nutrition Program; the HIV/AIDS strategy; and the Population Health Fund. Our sample comprised all projects over $2 million and 30 projects selected at random from the rest of the projects funded by the three programs.

9.44 Our observations in this section are drawn from those 38 projects. We are concerned about the significant number of our project reviews that identified problems in the project management process. In particular, we noted that the Branch did not subject high-value projects to a rigorous selection process, nor did it monitor those projects adequately. Our observations raise the question of whether the weaknesses we found in project management are broadly applicable across the Branch.

9.45 We selected these three programs because together they cover all stages of life (children and youth, adult, and later life), are administered nationally and regionally, use both grant and contribution agreements, and provide significant amounts of funding compared with the Branch's other population health programs. The combined annual budget of the three programs is roughly $70 million, or 35 percent of total spending on grants and contributions in the area of population health.

9.46 Our objective was to determine whether, in selecting and approving proposals for projects and subsequently monitoring and evaluating the projects, the Branch had met the terms and conditions approved by the Treasury Board and had adhered to its own program guidelines. We also wanted to determine whether payments had been made in accordance with the Treasury Board's Policy on Transfer Payments and the Financial Administration Act. These four sources of authority represent the standards for managing grant and contribution programs. The project management process described in Exhibit 9.9 reflects these standards.

9.47 We expected that the Branch would have a process for ensuring that each project respects the terms and conditions approved by the Treasury Board, the program guidelines, and the financial authorities. We also expected that the process would be clear, openly communicated, and consistently applied. Finally, we expected that by following the process, the Branch could ensure that project proposals would be appropriately selected and properly approved; that projects would be adequately monitored and evaluated; and that financial authorities would be respected. As outlined in Exhibit 9.9, managing projects entails four steps: project selection, approval, monitoring, and evaluation.

9.48 Selecting projects for funding requires that the Branch communicate with potential applicants and other parties to inform them that funds are available for projects and invite them to submit proposals. It reviews project proposals to determine whether they are eligible for funding. External experts then assess the proposals and provide advice on the merits of the projects and their potential contribution to the program's objectives. Then an advisory committee uses the assessments to rank the proposals.

9.49 The project approval process requires Branch officials to recommend whether a project proposal should be funded. These recommendations are forwarded to the Minister of Health, who makes the final decision to fund them or not.

9.50 The project selection process is important because it ensures that the Branch selects, from a number of proposals, those that contribute most to meeting program objectives. The process provides an opportunity for a range of organizations and individuals to submit proposals for funding. The project approval process requires that only projects subjected to the selection process be approved.

9.51 Project monitoring is the third key step in the management of grants and contributions. Monitoring means making regular contact to determine the status of the project, ensure that all required documents are submitted, and identify any problems as early as possible. Monitoring is also important for the Branch to ensure that recipients of contributions respect the terms and conditions of their agreements. Should the Branch identify any problems, it needs to take timely action to correct them by, for example, developing an action plan with the funding recipient, requesting an audit, or withholding funding.

9.52 Failure to monitor projects properly could mean that problems go unnoticed by the Branch. Problems include funds spent on activities other than those intended, or expenditures made that are ineligible for funding. Problems could mean that the project is no longer contributing to the program's objectives. As a steward of public funds, the Branch needs to take timely action when it finds problems.

9.53 Ongoing monitoring could show that an agreement covering a project may have to be amended because of changing circumstances. Grant and contribution agreements can be amended to provide additional funding or to extend a project's timeframe. The use of amendments needs to be limited to appropriate circumstances because amendments involve fewer controls. Funding provided through amendments is not subject to the same project selection process as funding through a new contribution agreement. Furthermore, staff who approve amendments are generally at a lower level than staff who approve new agreements. They also follow different administrative procedures.

9.54 Without proper guidance, amendments can be used as an informal way to extend a project's funding or timeline. They can also be used inconsistently and processed incorrectly.

9.55 Project evaluation, the fourth step, is a source of information that can be used to assess population health interventions. Contribution agreements require that each funding recipient complete an evaluation of the project. The evaluation is to identify how successfully the project has contributed to the objectives of the program. This information is used in deciding whether funding should be continued. Further, it can be used in an overall evaluation of the program.

9.56 The following paragraphs (9.57 to 9.88) briefly describe the three programs that we audited and present our observations on project management activities.

The Canada Prenatal Nutrition Program

9.57 At the 1990 United Nations World Summit for Children, the leaders of 71 countries made a commitment to invest in the well-being of children. In 1992, Canada responded to this challenge by developing the Community Action Program for Children (birth to six years). In 1994 the federal government announced the Canada Prenatal Nutrition Program (conception to six months), extending its support to the period before birth.

9.58 The Canada Prenatal Nutrition Program is a comprehensive, community-based program that supports pregnant women who live in conditions that threaten the healthy development of their babies. They include adolescents; women who abuse alcohol and other substances; those who live in violent or isolated situations or have no access to other services; refugees and recent immigrants; and off-reserve Metis, Inuit, and First Nations women. The program's aims are the following:

  • improve the health of pregnant women (for example, by providing nutritional supplements and nutritional counselling);
  • reduce the number of babies weighing under 2,500 grams at birth;
  • promote breastfeeding; and
  • increase access to services for pregnant teens and women (services such as support, education, referral, and counselling on lifestyle issues such as alcohol abuse, smoking, and family violence).

Exhibit 9.10 describes a project under this program.

9.59 Although the program had a number of projects running in 1994-95, the $12.3 million in funding provided in the 1997 Budget allowed for more community involvement. The program was renewed in 1998 and again in 2000, and its funding has now grown to $27.2 million for 2000-01. The program currently funds 277 projects in more than 680 communities across Canada. The Canada Prenatal Nutrition Program is delivered through Health Canada's regional offices.

9.60 Canada Prenatal Nutrition Program projects are selected, approved, and monitored properly. We audited 14 prenatal nutrition projects, all funded through contribution agreements. Many of them have operated for several years. We examined the project management process from the beginning of the program, but our observations focus on the process followed during the renewal in 2000. We observed improvements over previous years. In general, the project proposals were subjected to an adequate selection and approval process and the projects were monitored adequately.

9.61 Canada Prenatal Nutrition projects are not subject to evaluation but are required to participate in the national data collection exercise organized by the Branch. In the files we reviewed, all projects had provided data on the projects and participants that included, for example, client history, participation rates, and project outputs such as birth weight and breastfeeding initiation. This data collection exercise provides performance information that can be used in evaluating the overall program (see Exhibit 9.11 for information on who is being served by the program).

The HIV/AIDS strategy

9.62 The Population and Public Health Branch's HIV/AIDS strategy is part of the Canadian Strategy on HIV/AIDS, which was initiated in 1990 for a number of reasons:

  • HIV/AIDS imposes a significant cost on society. Health Canada estimates that the roughly 50,000 people currently infected will create an economic burden of $27 billion during their lifetime.
  • HIV/AIDS is a growing global epidemic.
  • The demographics of the disease continue to change. Health Canada recognizes that HIV/AIDS is no longer an epidemic affecting predominantly gay men, given the sharp increase in infection rates among women, Aboriginal peoples, and intravenous drug users.
  • Transmission of HIV/AIDS is entirely preventable.

9.63 Health Canada is the lead federal agency responsible for the Canadian Strategy on HIV/AIDS; as such, it receives $42.2 million or 98.6 percent of the Strategy's annual funding. The Population and Public Health Branch, through its HIV/AIDS strategy, manages $21 million of the funds allocated to Health Canada.

9.64 The HIV/AIDS strategy is designed to fund community organizations that deliver projects using one of the following approaches (see Exhibit 9.12 for an example of an HIV/AIDS project):

  • creating supportive environments;
  • health promotion for people living with HIV/AIDS;
  • prevention initiatives;
  • strengthening community-based organizations;
  • prevention, care, and treatment; and
  • supporting national non-government organizations.

9.65 Little action was taken to address concerns about projects. We reviewed files on 11 projects, eight of them administered nationally and three regionally. Included were three high-value projects (more than $2 million), discussed in Exhibit 9.13. We found that the Branch could not provide any documents detailing the selection process for any of the three high-value projects. We would expect that projects receiving this much funding would be managed more rigorously, and we consider it unacceptable that important documentation on these large contribution agreements was not available. In two of the projects, we found that the Branch had identified problems but had failed to take timely action to resolve them. The approval documents for the third project noted that the project received more funding than the reviewers had recommended.

9.66 In the eight national and regional files on projects under $2 million, we found five cases that were not subjected to an adequate selection process and yet the Branch recommended them for approval:

  • We found no evidence that one of the projects had been subjected to any selection process.
  • In two cases, the Branch specifically directed funding to the organizations. Funding was thus recommended and approved without any solicitation of proposals, internal or external review, or advisory committee review. One agreement was later amended to provide more funds.
  • In another case, the internal reviewer suggested that the Branch reject the request for $84,000 in funding. Two of the five external assessments suggested approving $84,000 only if major revisions to the project were made. Despite these negative assessments, the project was ultimately approved for an even larger amount, $130,000.
  • We also noted a similar case in which one assessment suggested rejection, and two suggested approval only if there were major revisions to the project. It appears that the revisions were never made, but funding was recommended and approved.

9.67 We found some evidence that the Branch monitored the eight projects under $2 million. However, we identified a number of weaknesses in its financial monitoring and control of payments:

  • One project made an ongoing expenditure that was ineligible. The Branch warned the funding recipient that it would not fund the expenditure, but it continued to do so for 10 months.
  • Three projects used program funding for expenditures that were ineligible.
  • Four projects did not submit documents required by the conditions of the agreement, such as progress reports, audited financial statements, and final reports.
  • One project's funding advance exceeded the maximum allowed under the Treasury Board Policy on Transfer Payments.

9.68 Many of the HIV/AIDS projects that we examined had no evaluations on file. When they did, the Branch often failed to use evaluation results to decide whether to renew or to terminate the project.

9.69 Recommendation. The Population and Public Health Branch should take corrective action to ensure that selection, approval, monitoring, and evaluation of all projects funded under its HIV/AIDS strategy follow the established processes.

Department's response. Agreed. A review of all current files has been launched. This review will identify remedial action, where required, and thus ensure that all projects funded under the HIV/AIDS strategy are in compliance with program guidelines and project management processes.

The Branch has established a grants and contributions monitoring capacity that will integrate with the work of the departmental quality assurance unit to ensure effective management of the Branch's grants and contributions.

As well, Health Canada's Grants and Contributions Steering Committee has developed a mandatory training program to be delivered to all departmental staff involved in the management and administration of departmental grants and contributions.

This training program has been piloted with some departmental staff in anticipation of rollout starting in early 2002.

Additionally, the Branch is reviewing the program guidelines for HIV/AIDS in order to ensure that they meet the requirements of the Treasury Board-approved terms and conditions. The Branch will also establish criteria for the funding, in exceptional circumstances, of non-solicited proposals.

The Population Health Fund

9.70 When the health promotion approach was replaced by the population health approach, the Population Health Fund was established to support projects that would increase community capacity to address the determinants that affect health. Projects can be directed at any of the three stages of life (children and youth, adults, and later life) and at any of the determinants of health (see Exhibit 9.14 for an example). Another purpose of the Fund is to give the Branch the flexibility to fund population health projects for which there is no specific program, perhaps because the issue is emerging or does not warrant specific programming.

9.71 The Population Health Fund was established in 1997 and has an annual allocation of $14.1 million. It currently administers about 280 contribution agreements and 200 grant agreements.

9.72 The objectives of the Population Health Fund are the following:

  • develop, implement, evaluate, and disseminate community-based models for applying the population health approach;
  • increase the knowledge base for program and policy development on population health; and
  • increase partnerships and develop intersectoral collaboration to address specific determinants of health or combinations of determinants.

9.73 The $14.1 billion annual budget of the Population Health Fund is split equally between national and regional projects. We were informed that most of the budget for national projects was committed to six national projects in the areas of prostate cancer research and fitness. Consequently, only about $1.7 million was available for other national projects. Only $5.5 million was available for regional projects because $1.5 million of the regional budget was used to fund the national fitness projects. This situation meant that far less money than envisioned was available to achieve the Fund's objectives.

9.74 Ineligible projects were approved for funding. The objectives of the Population Health Fund are clearly defined in the program's guidelines. When a project is funded whose objectives do not correspond with those of the program, funds are, in effect, being spent for purposes other than those approved for the program.

9.75 We reviewed 13 projects under the Population Health Fund, eight of them administered nationally and five regionally. Under the Population Health Fund guidelines (Exhibit 9.15), six of the national projects were not eligible for funding. Further, there was no evidence of communication with interested parties to invite project proposals; nor was there evidence of internal or external review or consultation with advisors. Yet all six of these projects were recommended and approved for funding, as discussed in Exhibit 9.16 and in the following paragraphs.

9.76 Our review of the three projects funded under the Prostate Cancer Research Initiative found that the Branch spent $15 million on projects, much of which was not eligible for funding under the program's guidelines. These projects are related to pure research and overhead fees that do not support the objectives of the program. This represents a violation of the program's terms and conditions. Further, the Branch paid one recipient $2 million even though it knew that the terms and conditions of the agreement had not been respected. We also found a disregard for the project management process, in particular, a failure to follow the project selection process. This too, violated the terms and conditions approved for the program. The approval of funding violated the program guidelines as well. These violations meant that not all potential applicants had an opportunity to apply for funding, and the funded projects were not compared with other potential projects to assess their relative merits.

9.77 Our review of the three projects funded under Enhanced Fitness Activities ($3.5 million over three years) found that they received funding for overhead fees that were not eligible under the Population Health Fund guidelines. This is another example of a violation of the program's terms and conditions. We also found a disregard for the project management process. The projects were not subjected to the selection process and, through the continued use of amendments, bypassed the project selection process again and continued to receive funds after the initial funding ended. We found no evidence that the $3.5 million in project funding went to the organizations that would contribute the most to the program's objectives.

9.78 We did not have similar concerns about the remaining seven projects, two of them national and five regional.

9.79 As was the case with the HIV/AIDS projects, many of the Population Health Fund projects that we examined had no evaluations on file. When they did, the Branch often failed to use evaluation results as part of the decisions to continue project funding.

9.80 Recommendation. The Population and Public Health Branch should ensure that the Population Health Fund is used to fund only projects that contribute to the program's objectives and meet its eligibility requirements.

Department's response. Agreed. The Branch has a management control framework in place that ensures that the Population Health Fund is used only to fund projects that meet program objectives and eligibility requirements. The Branch will reinforce the requirements of the framework to program staff as part of the department's training initiative.

9.81 Recommendation. The Population and Public Health Branch should ensure that it has proper authority to fund the six national projects. It should follow the established project management process.

Department's response. Agreed. Although Health Canada has approval to provide grants for research through its appropriations, it will clarify the grant authority for population health research. The Branch guidelines are being amended to clarify an inconsistency in wording between the guidelines and the Treasury Board-approved terms and conditions. A program directive, designed to clarify program authorities for fitness consistent with the terms and conditions, will be issued in December 2001.

9.82 Recommendation. The Population and Public Health Branch should strengthen its capacity to ensure that projects funded through contribution agreements are properly evaluated.

Department's response. Agreed. The Population and Public Health Branch has directed program staff to ensure that clear indicators of success are set out for all projects recommended for approval, and has reaffirmed the requirement for project evaluation, as set out in the revised standard contribution agreement template. Responsibility for strengthening program and project-level evaluation has been assigned to the Management Planning and Operations Directorate of the Branch. Additional steps include mandatory staff training and the revision of the appropriate processes and procedures in order to ensure compliance.

Program staff need additional guidance

9.83 Our audit identified several cases of non-compliance that occurred because staff lacked guidance on how to manage agreements. During our file review, for example, we asked who had the signing authority for approvals and amendments. We were told that there is no documented delegation of signing authority. Consequently, each program has developed its own generally accepted practices. However, the practices differ from program to program and are not always followed.

9.84 We observed that no departmental guidance exists on the use of amendments to contribution agreements. During our interviews, program staff expressed the need for such guidance. In the 14 prenatal nutrition projects we reviewed, we found that 156 amendments providing $8.8 million have been issued since the projects were initiated. Amendments were used in a number of different circumstances, for example, as part of the program renewal process in 1996, 1998, and 2000 (amendments totalling $6.3 million), as a means of bridge financing between an expired agreement and a new agreement, and as a means of providing additional funds to projects under existing agreements ($2.5 million). We identified six projects funded under agreements that were amended improperly, which means that funds were advanced in the absence of a valid contribution agreement.

9.85 Recommendation. Health Canada should ensure that staff are given adequate guidance to manage grant and contribution agreements, including the proper use of amendments.

Department's response. Agreed. Health Canada has launched a comprehensive initiative designed to improve the management of grants and contributions. This includes mandatory staff training and the establishment of a grants and contributions policy monitoring and quality assurance unit.

Consistent with the above, the Branch has established a grants and contributions accountability project that will ensure a timely and effective response to the recommendations of this audit, and other related initiatives.

The Population and Public Health Branch has issued direction to program staff on amendments to ensure they are properly used. A new amendment policy is now under consideration by the departmental Grants and Contributions Steering Committee.

Prompt action needed on internal audit recommendations

9.86 Within the last year, the Department carried out an internal audit of the Branch's grant and contribution programs as well as a study of the Department's framework for all grant and contribution programs. Although the scope of these two reviews was different from ours, we observed that some of their recommendations were related to issues that we included in this audit.

9.87 In the fall of 2000, the Internal Audit Directorate reported on its audit of grants and contributions managed by the Population and Public Health Branch. While that audit focussed on the adequacy of some key controls that we also identified and assessed, it did not look at the selection process or at whether evaluations of projects had been undertaken as the agreements required. A year after receiving the internal audit report, management formally accepted the recommendations and approved an action plan to implement them. It is important that there be no further delay in putting this action plan into effect.

9.88 In March 2001, Health Canada commissioned an outside consultant to review the Department's management framework for all grant and contribution programs. The review identified the need for a number of improvements and proposed an action plan. At the time of our audit, management at Health Canada had not yet approved that plan. Its timely implementation is crucial to correcting identified weaknesses.

Top

Measuring and reporting results

Measuring program results is part of good program management

9.89 In this section of the chapter we report our observations on the measurement and reporting of results at the program level. Periodic program evaluation and ongoing measurement of performance tell management what has worked to produce desired results and what has not. The Branch needs this information for good program management; it provides a basis for planning and for setting priorities. In addition, it allows the Department to report publicly on the effectiveness of its programs.

9.90 We expected that the Department would have a plan for evaluating the programs of the Population and Public Health Branch and that it would carry out the evaluations in that plan. Further, we expected that the Branch, too, would evaluate its programs.

Few population health programs have been evaluated

9.91 We found that the Population and Public Health Branch does not have a formal evaluation plan for its 12 contribution programs. It has allocated few resources to program evaluation and has completed no evaluations.

9.92 However, Health Canada's Departmental Program Evaluation Division has a departmental evaluation plan that calls for evaluations of three population health programs: Canadian Strategy on HIV/AIDS (July 2001), Canada Prenatal Nutrition Program (October 2001), and Community Action Program for Children (2003). These are under way. In addition, the Department has committed to providing the Treasury Board with an evaluation of population health programming by 2004.

9.93 Funding for the 1998 renewal of the Canadian Strategy on HIV/AIDS was approved with a condition that a threeyear report on the Strategy's success be submitted to Cabinet no later than July 2001. This evaluation was under way at the time of our audit, but the Department had not yet set a completion date. When the Canada Prenatal Nutrition Program was established in 1994-95, the Department committed to conducting an evaluation after four years. In part because of lack of data, the evaluation was delayed. Since then, however, the Branch has collected some national data on the program. The evaluation that was due in October 2001 is now scheduled for completion in March 2002, eight years after the program's inception. No evaluation of the Population Health Fund is scheduled.

9.94 We noted that the prenatal nutrition program has collected data on individual projects through questionnaires completed by the funding recipients and by participants in the projects. These data have been rolled up nationally to provide information about infant and maternal health outcomes across Canada that will be valuable to the current evaluation. However, as noted earlier, we are concerned that many HIV/AIDS strategy and Population Health Fund projects did not have evaluations on file. Without evaluations at the project level, potentially important data are not available for evaluations at the program level.

9.95 Some regional offices have initiated studies of aspects of population health programs in their areas. However, these studies are not part of any consolidated Branch-wide effort, making it difficult to understand the value of interventions on a national basis.

The Branch needs to develop indicators for measuring performance

9.96 While evaluation is carried out periodically, performance measurement is an ongoing activity that measures results. Performance measurement helps managers know whether a program is moving in the right direction. Central to measuring performance are performance indicators, which specify the terms in which performance or success will be measured. We expected that the Department would be using or developing performance indicators.

9.97 We found that the Population and Public Health Branch twice began to develop a framework for better management of its performance, but it has not yet completed one. Nor has it made much progress in developing performance indicators for most population health programs.

9.98 In late 1998, Health Canada developed a population health performance and accountability framework that outlined objectives, key expected results, performance indicators, a performance measurement strategy, and responsibility for ensuring accountability. This performance and accountability framework was general in nature and applied to specific population groups. It was never implemented.

9.99 The 1999 Budget directed $43 million over three years to Health Canada for the Federal Accountability Initiative to improve the Department's reporting to Canadians on its performance. The Information, Analysis and Connectivity Branch of the Department is leading this initiative, which provides funding to Health Canada's branches. The Population and Public Health Branch received almost $1.2 million, which it matched from its own budget, to develop an integrated performance measurement and reporting framework. It has not yet finalized the framework, although Branch management has committed to do so by the end of 2002.

9.100 The Branch has made little progress in finalizing performance indicators for population health programs. It has outlined indicators for 12 of 27 program areas but in most cases has not identified data sources or specified targets. Of the programs we audited, only the Canada Prenatal Nutrition Program shows some progress. It has developed project and client questionnaires to capture data on infant and maternal health outcomes and breastfeeding initiation. It has reported these data nationally and regionally since 1998. Neither the HIV/AIDS strategy nor the Population Health Fund has finished developing performance indicators.

9.101 With neither program evaluations nor performance indicators, the Branch cannot say to what extent its programs have helped improve the health of Canadians.

The reporting of results to Parliament needs to improve

9.102 The Department reports to Parliament only limited information on the effectiveness of its population health programs. We reviewed its reports on plans and priorities and its performance reports for the last three years and found few references to the effectiveness of population health programs.

9.103 Recommendation. Health Canada and its Population and Public Health Branch should strengthen the evaluation of the population health programs.

Department's response. Agreed. The Population and Public Health Branch will work with the Departmental Program Evaluation Division (DPED) to implement the Treasury Board's new Evaluation Policy. As part of this initiative, the Population and Public Health Branch and DPED will immediately undertake to invest in developing the required capacity, including specialized human resources, and the development and implementation of a multi-year evaluation plan for population health programs. Some evaluations of population health programs are currently under way.

9.104 Recommendation. The Population and Public Health Branch should finalize performance indicators for its population health programs.

Department's response. Agreed. A performance measurement project is under way in the Branch, as part of a department-wide initiative to improve reporting to Parliament and citizens on the results of Health Canada's programs. The selection of program performance indicators, as well as the identification of data sources for these indicators and a measurement and reporting strategy, are included as part of the performance measurement framework.

9.105 Recommendation. Health Canada should ensure that the results of its population health programs are reported to Parliament in its performance report.

Department's response. Agreed. Health Canada will report on results of population health programs in its performance report for the period ending March 2002.

Top

Conclusion

9.106 Health Canada has a responsibility to help the people of Canada maintain and improve their health. As one means to achieve this, it provides leadership and support for preventive health activities like health promotion, disease and injury prevention, health protection, and population health surveillance and assessment. The Population and Public Health Branch has primary responsibility for federal population health initiatives, which it delivers largely through grant and contribution programs.

9.107 The Population and Public Health Branch has no formal basis for determining what, where, and how population health activities should be undertaken. The Branch recently developed a framework for setting priorities. However, the framework has not been approved, nor is there a strategy or timetable for implementing it. Such a framework would provide a rationale for terminating or refocussing existing programs and creating new programs.

9.108 In all three programs we examined, we found a well-established project management process and clear program guidelines. If followed, both help ensure that projects respect the terms and conditions approved for the program by the Treasury Board, the program guidelines, and the applicable financial authorities.

9.109 In the Canada Prenatal Nutrition Program, we found that the projects we reviewed had generally been managed well. We did point out, however, that program staff were frequently using amendments to agreements without proper guidance in order to continue project funding.

9.110 In the two other programs, we observed many situations where management failed to follow established processes. In particular, we found that several projects were funded under the HIV/AIDS strategy and six national projects under the Population Health Fund without proper adherence to important steps in the selection process. In these cases, there was no solicitation of proposals from other potential applicants. Also, proposal assessments either had not been completed or were weak.

9.111 In our review of the HIV/AIDS strategy, we noted that files on two highvalue projects lacked important documents. Further, the Branch had failed to resolve a number of concerns that were raised about projects before it funded them. Several projects under $2 million were not subjected to an adequate selection process and yet the Branch recommended that they be approved for funding. Consequently, the Branch has no assurance that the best projects were funded or that those projects are being properly managed. As well, we concluded that the Branch did not exercise proper stewardship over public funds to these projects.

9.112 In the Population Heath Fund, we noted that funds had been used to support six large national projects that were not eligible under the program's guidelines. Two of these six projects involve pure research, which is clearly not one of the program's objectives. Further, they do not support the Branch's responsibility to promote health across the population. Management had not ensured that it had the proper authority to fund the six projects, in the areas of prostate cancer research and fitness. The failure to respect the established project management process resulted in funding decisions that were arbitrary or inappropriate.

9.113 Clearly there are a variety of factors, including political, national, and international pressures, that impact on how grants and contributions are used to fund projects. However, we identified a number of projects that management had recommended for approval and approved without having followed its own project management process. We believe that this is unacceptable.

9.114 The Branch needs to improve its measurement and reporting of results of grant and contribution programs. None of the programs we audited had been evaluated to determine whether they had achieved their objectives, whether they were still relevant, or whether alternatives should be considered. Such evaluations would examine the difficult questions of the programs' rationales and their effectiveness. Answers to these questions are important if Parliament is to know whether these programs are valuable or not, and to what extent they have an impact on the health of Canadians.

Top

About the Audit

Objectives

The objectives of the audit were to determine the following:

  • whether there is a rational basis for determining what, where, and how population health activities should be undertaken;
  • based on the Branch's project management process, whether its management of grant and contribution agreements is adequate; and
  • whether the Branch adequately measures and reports the results achieved by population health programs.

Scope and approach

Our audit focussed on Health Canada's population health activities managed by the Population and Public Health Branch. We examined how the Branch establishes program priorities. We examined the project management of three grant and contribution programs used to fund population health activities. Further, we looked at how program achievements are evaluated and reported.

We interviewed departmental staff and several provincial officials involved in public health. We reviewed documentation, including legislation, corporate and branch documents, grant and contribution agreement files, planning documents, studies, internal audit reports, and information from a variety of Internet Web sites.

Our audit included a detailed examination of files on 38 projects from the Canada Prenatal Nutrition Program, the HIV/AIDS strategy, and the Population Health Fund. We examined whether the 38 projects were selected, approved, and subsequently monitored and evaluated according to the terms and conditions approved by the Treasury Board and the program guidelines developed by the Population and Public Health Branch. We also examined whether payments were made in accordance with the Treasury Board's Policy on Transfer Payments and the Financial Administration Act.

For each project, we looked for evidence that Health Canada had done the following:

  • followed the established selection process, including
    • communicating with stakeholders to inform them that funds were available for projects
    • completing internal assessments of proposals as part of the selection process (to determine the eligibility of the organization and project; the capacity of the project to achieve the program's objectives and respect program priorities; and the eligibility of proposed expenditures)
    • completing external assessments of proposals and consulting with advisors (to determine the acceptability of projects and their funding levels and the relative priority of projects)
  • followed the approval process (to ensure that projects were recommended for approval appropriately and that funding was approved by the appropriate authority)
  • followed the project monitoring process (to ensure that the project remained in compliance with the terms and conditions of the contribution agreement; that the organization had the capacity to continue to achieve program objectives; that financial requirements were respected; and that actual expenditures were eligible)
  • followed the amendment process (to ensure that amendments were used appropriately)
  • followed the project evaluation process (to ensure that evaluations were completed, used in project renewals, and used in evaluation of programs).

Criteria

We expected that

  • the Population and Public Health Branch's population health activities would be based on a rational framework for setting priorities
  • the objectives of the grant and contribution programs would be clearly stated. Program criteria would be consistent with program objectives and the requirements of central agencies
  • appropriate procedures for selecting projects would be clearly documented and communicated and consistently applied
  • the Population and Public Health Branch would appropriately monitor approved grant and contribution agreements and report the results
  • financial and management control of project funding would comply with relevant government policies and legislation.

Audit team

Assistant Auditor General: Maria Barrados
Principal: Patricia MacDonald
Director: Linda Anglin

Theresa Bach
Karen Dornan
Chris Kelly
Jacques Maziade
Glenn Wheeler

For information, please contact Patricia MacDonald.

Top