This Web page has been archived on the Web.

2002 December Report of the Auditor General of Canada

Main Points

6.1 Good-quality statistics are fundamental to an informed public debate on health issues. Statistics Canada is one of the key providers of health statistics. It has put in place a wide range of systems and practices to build quality into its statistical programs and to ensure that data quality remains a priority. Our audit found that Statistics Canada's data quality assurance systems and practices are sound and that the Health Statistics Division applies these to its health statistics programs. This has resulted in statistics that meet the Agency's data quality standards. We found that the health statistics produced by Statistics Canada for the September 2002 federal, provincial, and territorial health indicators reports are sufficiently accurate for this use.

6.2 Because Health Statistics Division's surveys are under its direct control, the Division makes necessary improvements to maintain data quality when the need arises. In co-operation with the organizations that provide administrative data to Statistics Canada, the Division is leading work to maintain and improve the quality of the data. We concluded that, with the exception of one database, Statistics Canada's quality assurance systems and practices adequately ensure the quality of its health statistics.

Background and other observations

6.3 Current attention to health issues includes a focus on developing health indicators that measure the health status and health outcomes of Canadians. The Communiqué on Health from the September 2000 First Ministers' Meeting included a commitment for governments to report regularly on health indicators, beginning in September 2002.

6.4 Our audit focussed on databases and surveys that provided data for these reports. At the federal level, the two main organizations involved in collecting health statistics are Statistics Canada and Health Canada. Most of the data came from Statistics Canada, with some from Health Canada and the rest provided by the Canadian Institute for Health Information (CIHI) and by the provinces and territories.

6.5 Statistics Canada's Health Statistics Division has a mandate to provide accurate, timely, and relevant information about the health of Canadians and the health care system. The Division works to ensure that its health statistics meet Statistics Canada's data quality standards. It is responsible for vital statistics (births, deaths), cancer data, data for indicators on health outcomes, and health surveys.

6.6 All statistics are, to some extent, estimates of the reality they seek to convey. Therefore, they have to be used with a sound understanding of their strengths and limitations. However, the Health Statistics Division's application of Statistics Canada's policy on informing users of data quality and methodology is inconsistent. While Statistics Canada has established a single database for informing users on data quality, the information in the database is not complete. The Agency needs to develop an approach to help all program managers maintain current, complete information in the database.

6.7 It is important that the Division determine the quality of data it receives from other organizations so it knows to what extent these data can be relied on for their intended use. The Division receives data from the CIHI Hospital Morbidity Database that it uses to construct indicators on patient outcomes; however, the Division did not make a formal determination of data quality. We also found that the Agency did not formally monitor the accountability requirements of the Health Information Contribution Program. The Program's contribution agreement provides funding for the statistical databases that Statistics Canada transferred to CIHI, including the Discharge Abstract Database. This has implications for maintaining the quality of the transferred databases. We concluded that, with the exception of this database, the Health Statistics Division has quality assurance systems and practices to adequately ensure the quality of its health statistics.

6.8 Health Canada produces health statistics through national surveillance systems. These systems collect data on communicable and chronic diseases and injuries. Three of these systems provided data for the September 2002 health indicators reports. However, the quality assurance systems and practices for these databases do not ensure the accuracy of the data. Therefore it was not possible to form an opinion on the accuracy of these data. Improvements in these processes are needed.

6.9 Health Canada is also responsible for monitoring funding provided to CIHI under the Health Information Roadmap Initiative. We found that the Department does not actively monitor this initiative in accordance with the grant's requirements. As a result, it has no formal basis for knowing whether the Initiative's intended objectives are being achieved.

6.10 CIHI's Discharge Abstract Database and Hospital Morbidity Database provided data for several of the indicators in the September 2002 health indicators reports. Audit work led by the Office of the Auditor General of British Columbia found that documentation on the quality assurance process for the systems that provide the data is inadequate. As well, a three-year study that will provide information on the quality of the input data will not be completed for another two years. Therefore, it was not possible to form an opinion on the accuracy of the specific data provided by CIHI for the health indicators reports.

Statistics Canada and Health Canada have responded. Statistics Canada and Health Canada agree with our recommendations. However, Health Canada considers that our chapter does not acknowledge the substantial flow of information to Health Canada from the Canadian Institute for Health Information. Both entities have indicated in their responses that a number of actions are under way to deal with our recommendations.

Introduction

6.11 Health and health care issues continue to be intensely debated in Canada. Our aging society, evolving technologies, and access to services often feature in the news. Canadians are demanding transparency and accountability for decisions that affect their health services. Health care practitioners stress the importance of accurate information on diseases and their treatment. Fiscal restraint has led to greater discussion about the information needed to make funding and policy decisions. Reliable and meaningful health statistics can contribute to an informed debate on health issues.

6.12 By health statistics we mean data describing the health of Canadians and our health care system. Health statistics answer questions such as the following: What are the factors that influence good health? How many Canadians have heart attacks each year? Is that number going up or down? What health interventions are these people receiving? Are some interventions more effective than others, and what do they cost taxpayers? What preventive measures are effective and how widespread is their use?

6.13 The concept of health statistics has expanded over the years in Canada. Until the 1960s, most health statistics dealt with the incidence of illness. In the 1960s and 1970s, they began to reflect the financial and other costs of treating health conditions. The 1990s saw data used increasingly to construct indicators and measures related to issues such as determinants of health. Currently, much attention focusses on developing health indicators that measure the health status of Canadians and patient outcomes.

6.14 Better health statistics can help

  • Canadians make more informed decisions about their health,
  • physicians provide better care to patients by giving them empirical evidence on the outcomes of available medical procedures,
  • health care managers improve the cost-effectiveness of care,
  • researchers better understand the determinants of health, and
  • governments formulate better health policy by giving them critical evidence and feedback to evaluate past decisions and the implications of future decisions.
Recent developments in Canada related to health statistics

6.15 Starting with the 1991 Report of the National Task Force on Health Information, there has been a greater focus in Canada on the importance of health statistics and on ways of improving them. Other key developments include the creation of the Canadian Institute for Health Information (CIHI) and the establishment of the Commission on the Future of Health Care in Canada. As well, the Communiqué on Health from the September 2000 First Ministers' Meeting included the commitment for governments to report regularly on health indicators, beginning in September 2002 (Exhibit 6.1).

Collecting health statistics

6.16 There are three main organizations that provide Canadians with health statistics at the national, provincial, and territorial levels (Exhibit 6.2). The two principal organizations in the federal government involved in collecting health statistics are Statistics Canada and Health Canada. The third organization, CIHI, collects mostly administrative health data from across the country.

6.17 Statistics Canada. Statistics Canada's mandate derives mainly from the Statistics Act. The Act requires that the Agency collect, analyze, and publish statistical information on the economic, social, and general conditions of the country and its citizens. It is the lead organization for co-ordinating the country's statistical system.

6.18 The Health Statistics Division at Statistics Canada is primarily responsible for health statistics programs. Its mandate is to provide accurate, timely, and relevant information about the health of Canadians and of the health care system. The Health Statistics Division's key activities include running its health statistics programs, producing related products and services (such as the quarterly journal, Health Reports), and participating in projects as part of the Health Information Roadmap Initiative. The Division receives data from two sources—administrative records that other organizations provide and its own surveys. In 2001-02 the Division's budget was approximately $8.0 million and it had about 100 full-time employees. In that same year, the Division also received $10.5 million from CIHI in funding for projects under the Health Information Roadmap Initiative.

6.19 Health Canada. Health Canada also provides many health statistics. Its national health surveillance systems monitor and collect data on communicable and chronic diseases and injuries. These systems allow it to monitor diseases and injuries on a national level. The provinces and territories and a variety of other federal and non-governmental organizations also contribute data to these systems. As well, Health Canada is responsible for delivering some health services to First Nations and producing statistics on First Nations health.

6.20 Canadian Institute for Health Information. Founded in 1994, CIHI is a federally chartered, but independent, not-for-profit corporation. Its mandate is to co-ordinate the development and maintenance of an integrated national approach to health information. CIHI is funded primarily by the provinces and the federal government. When it was created, Statistics Canada transferred several data holdings to CIHI—hospital morbidity statistics, mental health statistics, therapeutic abortion statistics, an annual hospital survey, and a survey of registered nurses. Health Canada also transferred several databases to CIHI. These include the national health expenditures database and the national physician and medical care database. Among other responsibilities, CIHI administers Health Information Roadmap Initiative funds.

Focus of the audit

6.21 This audit is a continuation of our past examination of efforts by Statistics Canada to ensure the quality of national statistics. Previous work included our 1999 audit, Managing the Quality of Statistics, and follow-up work in 2001 on progress made in implementing our recommendations. In our 1999 audit, National Health Surveillance: Diseases and Injuries, we observed weaknesses in Health Canada's systems for collecting surveillance information. In September 2002, we reported that Health Canada had made limited progress toward resolving some of these weaknesses.

6.22 The objectives of this audit were to determine whether

  • Statistics Canada has put in place appropriate systems and practices to adequately ensure the quality of its health statistics;
  • Statistics Canada, Health Canada, and CIHI have applied quality assurance systems and practices to ensure the accuracy of the health indicators data that they produced for inclusion in the federal, provincial, and territorial health indicators reports; and
  • the implementation of the Health Information Roadmap Initiative is consistent with the funding agreement.

6.23 The current audit is timely given the attention that health statistics are now receiving in Canada. With this increased attention, the quality of statistics that Statistics Canada, Health Canada, and others produce is coming under greater scrutiny. Specifically, this audit focussed on the adequacy of Statistics Canada's quality assurance systems and practices for its five health statistics programs that provided health indicators data for the most recent year included in the September 2002 health indicators reports. These programs include Vital Statistics, the Canadian Cancer Registry, the Hospital Person-oriented Database, the Canadian Community Health Survey, and the Health Services Access Survey. The five programs represent the major health statistics databases within the Health Statistics Division. Our work included an examination of how the Agency determines the quality of the data that it receives from other organizations and how it informs users about data quality.

6.24 We also examined the accuracy of data that Statistics Canada and Health Canada produced for inclusion in the federal, provincial, and territorial governments' health indicators reports in September 2002. Some of the indicators for these reports came from two of CIHI's many databases (the Discharge Abstract Database and the Hospital Morbidity Database). The Office of the Auditor General of British Columbia led a team of legislative auditors—of which we were a member—to examine these databases. Finally, we reviewed two programs, the Health Information Roadmap Initiative and the Health Information Contribution Program, to ensure that the requirements of the agreements were met. Further details on our audit are included at the end of the chapter in About the Audit.

Observations and Recommendations

Quality of Statistics Canada's health statistics

Data quality assurance is well established in Statistics Canada

6.25 Statistics Canada has put in place a wide range of systems and practices to build quality into its statistical programs and to ensure that data quality is a priority. Our 1999 audit noted that the Agency's Quality Assurance Framework, in common with other reputable statistical agencies, approached data quality from the user's perspective. The Framework identifies six characteristics or standards of quality and defines a quality management process for data quality assurance systems and practices (Exhibit 6.3). These characteristics are consistent with our own audit criteria for adequately assuring the quality of statistics. We found that Statistics Canada had used a number of formal mechanisms for assessing data quality but that individual programs had not consistently applied them.

6.26 Our 2001 follow-up to our 1999 audit observed that the Agency had developed a revised formal quality assessment mechanism for all of its statistical programs. Building on the Quality Assurance Framework, Statistics Canada designed the Integrated Program Reporting process as a thorough and rigorous self-assessment of data quality. Exhibit 6.4 summarizes the Integrated Program Reporting process and the Agency's guidelines.

The Agency's health statistics programs meet its data quality standards

6.27 This year's audit found that Statistics Canada's quality assurance systems and practices are sound. The Health Statistics Division followed the Agency's Quality Assurance Framework and produced measures demonstrating that it met the data quality requirements. Wherever the Division identifies quality issues, it undertakes work to address them. The Division reports on the results of its quality assurance practices in its Biennial Program Report and Quadrennial Program Report; however, measures of quality need to be better documented so that internal and external stakeholders will be better informed about whether the data are sound.

6.28 Because Health Statistics Division's surveys are under its direct control, the Division makes necessary improvements to maintain data quality when the need arises. In co-operation with the organizations that provide administrative data to Statistics Canada, the Division is leading work to maintain and improve the quality of the data. We concluded that, with the exception of a component of the Hospital Person-oriented Database, Health Statistics Division's quality assurance systems and practices result in data that meet Statistics Canada's standards to adequately ensure the quality of its health statistics.

The Health Statistics Division applies and systematically assesses its data quality procedures through Integrated Program Reporting

6.29 We expected that Statistics Canada would consistently apply its quality assurance systems and practices to all its health statistics programs. We also expected that it would assess the adequacy of these systems and practices for assuring quality in accordance with the Quality Assurance Framework and the guidelines for Integrated Program Reporting.

6.30 Integrated Program Reporting process assesses quality assurance. Our audit found that the logic and the design of Statistics Canada's Integrated Program Reporting process are consistent with the Agency's Quality Assurance Framework. This process requires managers to review quality assurance systems and practices in place and the results of those practices. The guidelines require that estimates of the accuracy of statistical programs be examined and that analytical programs disclose the accuracy of their main data sources.

6.31 The Health Statistics Division completed its Biennial Program Report in February 2000 and its Quadrennial Program Report in November 2001 (see paragraph 6.75 for our observations on the Annual Consolidation of Indicators). Both reports provide valuable information on the Division's programs and products, including information on challenges that its programs are facing. As well, they discuss some data quality issues for major statistical publications and releases. These two program reports were formally reviewed by senior management committees, who then provided comments and asked for clarification, where necessary.

6.32 Biennial and Quadrennial Program reports do not provide sufficient results about data quality. Although the Health Statistics Division applied quality assurance systems and practices to its statistical programs, the Biennial and Quadrennial Program reports did not provide sufficient specific results of data quality measures, as required by the guidelines. Therefore, we could not reach a conclusion on whether the Division's programs were meeting the Agency's data quality assurance systems and practices. The lack of specific measures of data quality in the reports limited the Division's ability to demonstrate the extent to which it met the data quality requirements. It also had an impact on the Division's capacity to direct senior management's attention to areas that needed improvement.

6.33 As a result, the two program reports did not provide the necessary information to allow us to reach a conclusion on the adequacy of data quality assurance systems and practices. We therefore did some additional work to determine whether this problem was due to poor documentation in the reports or to actual shortcomings in data quality assurance. We found that Statistics Canada did more quality assurance work than it documented in its two program reports.

6.34 Recommendation. Statistics Canada should provide better documentation and references to more complete documentation of the results of its data quality systems and practices in the Biennial Program Report and the Quadrennial Program Report.

Agency's response. Agreed. Specific data quality reports will be cited in the Biennial Program Report and the Quadrennial Program Report to support important program activities. Statistics Canada will continue to enhance the utility of the Integrated Program Reporting process through regular review.

Statistics Canada is leading work to strengthen integrity of vital statistics data

6.35 Statistics Canada uses the results of quality assurance systems and practices documented in program reports, in addition to the results of other work, to manage the quality of health statistics it produces. We found that the Agency applied quality assurance systems and practices to its vital statistics data and that the data met the Agency's data quality standards. In addition to what it included in the program reports, Statistics Canada measured data quality and identified some areas where the integrity of the data can be strengthened. In partnership with the provinces and territories, Statistics Canada is leading work to improve the integrity of vital statistics data across Canada.

6.36 Due to legal reporting requirements, the registration of vital statistics—births, deaths, and stillbirths—is virtually complete. The provinces and territories are responsible for collecting the statistics, editing them, and sending them to Statistics Canada. The Agency then collects and processes the vital statistics data. Statistics Canada has undertaken work to measure data quality, including verifying problematic records, carrying out numerous edits and imputations, and tracking their potential impact on data quality. It produces reports on the capture, editing, and quality control of data. The Agency also takes an annual sample of records and independently verifies them with the original registrations. It has documented the sources and causes of deficiencies and is actively working to correct them.

6.37 Statistics Canada works with individual jurisdictions on the quality of their data. It also works to improve data quality through its participation in the Vital Statistics Council for Canada, which is a partnership between provincial and territorial vital statistics registrars and Statistics Canada. The Council developed a business plan for 1999-2004 that commits to improving national standards in areas such as data definition, collection, and reporting. In addition, as part of the Health Information Roadmap Initiative, Statistics Canada is working on a project to improve the quality of the data and enhance the capacity to use these data in developing health outcome information.

Statistics Canada is working with provincial and territorial cancer registries to improve data quality

6.38 We found that Statistics Canada's application of its quality assurance systems and practices to Canadian Cancer Registry data met its standards for data quality. The Agency measured data quality, identified ways to make improvements, and is working with provincial and territorial cancer registries to implement these improvements.

6.39 The Canadian Cancer Registry contains data collected by provincial and territorial cancer registries. Data files are sent to Statistics Canada for edit checks, elimination of duplicate and redundant files, entering into a national registry, and analysis. Statistics Canada has incorporated a wide range of edit checks to ensure the completeness and quality of the data. It also regularly produces a set of quality control reports. These are shared with the provincial and territorial registries to improve overall data quality and to target areas that need improvement.

6.40 Statistics Canada undertakes other work to promote the quality of its data. It sponsors an annual workshop on coding issues concerning the International Classification of Diseases. It also commissions studies on data quality to address specific concerns. Under the Health Information Roadmap Initiative, Statistics Canada is leading joint work with provincial and territorial partners on a project that includes assessing the completeness and quality of existing information.

6.41 Recommendation. Statistics Canada should continue to work with the provinces and territories to improve the quality of health statistics databases.

Agency's response. Agreed.

Additional data quality work is needed for the Hospital Person-oriented Database

6.42 The objective of person-oriented information is to analyze and describe how Canadians fare as a result of their contact with the health care system (that is, patient outcomes). The Hospital Person-oriented Database is created by transforming hospital morbidity data from being event-oriented (that is, hospital visits) to person-oriented (that is, all hospital visits by the same person). We found that Statistics Canada had not formally determined the quality of the hospital morbidity data received from CIHI (paragraphs 6.60 to 6.64). Therefore, we were unable to reach a conclusion on whether the Hospital Person-oriented Database met the Agency's data quality standards.

6.43 The database combines records of hospital visits to create information that can be used to track a person's health outcomes. Examples of person-oriented information that address patient outcomes include the 365-day net survival rate for acute myocardial infarction and the 180-day net survival rate for stroke.

6.44 The two program reports described earlier state that reports are generated to assess the quality of the linkage on individuals. As well, under the Health Information Roadmap Initiative, Statistics Canada is conducting work as part of the Person-Oriented Information Project. This is intended to increase its capacity to combine health care data and to use this capacity to provide information on the health of Canadians as well as on the performance of the health care system.

6.45 Nonetheless, the program reports do not refer to specific data quality measures for the Hospital Person-oriented Database. The reports state that the Agency produces a set of quality control reports regularly to monitor the quality of the main person-oriented data items of the hospital morbidity files. We found that these quality control reports focussed only on very specific quality issues and on instances where transforming hospital morbidity data yielded errors. The Agency did not undertake other work to determine the overall quality of the data it received from this source.

The Canadian Community Health Survey meets data quality standards

6.46 Statistics Canada has applied a comprehensive set of data quality assurance systems and practices to the Canadian Community Health Survey. The results were used in managing data quality, and we found that the survey met the Agency's standards for data quality.

6.47 The Canadian Community Health Survey is a new biennial survey. It provides data on health determinants, health status, and health system use to address information needs at the national, provincial, territorial, and regional levels. Statistics Canada released its first survey data in May 2002.

6.48 At the outset of this audit, Statistics Canada agreed that it would carry out a self-assessment of the quality management systems and practices applied to the survey. This was because the survey was still in progress at the time when the Health Statistics Division prepared its Biennial and Quadrennial Program reports.

6.49 The self-assessment concluded that this survey is well supported by a comprehensive quality assurance program. This program maintains data quality through extensive consultations, testing and monitoring, and other data quality assurance activities. Our further examination found that Statistics Canada had

  • undertaken appropriate consultations to help ensure the relevance of the statistics to users,
  • developed and tested much of the survey tool through use in a previous survey,
  • yielded reliable estimates as a result of the sample size and method at the level of health regions,
  • put in place a solid infrastructure to collect the data,
  • carried out quality assurance reporting monthly, and
  • released the data on a timely basis.
The Health Services Access Survey meets data quality standards

6.50 In 2002 Statistics Canada conducted the Health Services Access Survey, a supplement to the Canadian Community Health Survey, to provide some additional data for the September 2002 health indicators reports. It used the same quality assurance systems and procedures as the Canadian Community Health Survey. We concluded that the Health Services Access Survey had adequate quality assurance systems and practices to ensure the quality of the data.

Getting to good-quality statistics

6.51 Although Statistics Canada's health programs meet data quality assurance standards, there is always room for further improvement. Quality in statistics is not an absolute standard but is relative to the elements of the Quality Assurance Framework and to the intended use of the statistics. Implementing a quality assurance framework requires continuous work, monitoring, review, and adjustment. Statistics Canada is very concerned about data quality. It appropriately identifies areas for improvement and takes steps to act on them. Further, one of the underlying principles of Statistics Canada's Quality Assurance Framework is that continuous work is necessary to ensure that data quality is maintained.

6.52 We found that the quality assurance systems and practices in place meet the standards of Statistics Canada's Quality Assurance Framework. These standards seek to ensure, among other things, that the data are accurate. We found that the data are accurate for the purpose of reporting at the jurisdictional level (national, provincial, and territorial figures for the September 2002 health indicators reports). However, users focussing on data at a lower or more detailed level (for example, an incidence rate for a disease within a regional health authority) need to be aware that any improvements in data quality made by Statistics Canada at the provincial and territorial levels may not lead to data accuracy at lower levels.

Statistics Canada's disclosure to users of quality of health statistics is uneven

6.53 All statistics are, to some extent, estimates of the reality they seek to convey. Therefore, they have to be used with full awareness of their strengths and limitations. Statistics Canada's policy on informing users of data quality and methodology (Exhibit 6.5) is to provide users with the information they need to determine whether the statistics fit their purposes. Our 1999 audit found that the Agency's implementation of the policy was inconsistent. Our 2001 follow-up report noted that Statistics Canada had taken steps to address our observations. It had begun to create a central source of information to support internal reporting and external disclosure of the quality of statistics. Statistics Canada refers to this as an Integrated Meta-database. The Agency introduced the Integrated Meta-database in 2000 and planned that it would have information on the data quality, concepts, and underlying methodology of each of its databases. This was to ensure a central source of information to support both consistent and effective internal reporting and external disclosure of the quality of the Agency's statistics.

6.54 We expected that, as specified by the policy, individual statistical programs would appropriately inform users about data quality and methodology when data are disseminated.

6.55 We found that the Health Statistics Division used the policy on informing users of data quality and methodology; however, the application of the policy was not consistent. Not all of the policy requirements were being adhered to in disclosing the quality of health statistics. The Integrated Meta-database provided some information on data quality but less than the policy requires.

6.56 Policy inconsistently applied. In 2001-02, Statistics Canada completed an internal audit of Agency-wide compliance with the policy. This internal audit reviewed samples of recent statistical products as well as the surveys included in the Integrated Meta-database. The internal audit found that Statistics Canada products did not have a standardized message to users and that program staff had difficulty determining some of what was required by the policy. As well, neither current health surveys nor current databases were included in the sample for the internal audit. In light of that audit, we examined whether those health programs now included in the Integrated Meta-database had appropriately notified users about data quality.

6.57 Integrated Meta-database has incomplete information on health statistics. We found that, in all cases, the information in the Integrated Meta-database was less than required by the policy. Key measures of data quality were missing. Overall, the database did not provide a complete picture of the quality of health program data.

6.58 The lack of complete information on quality measures is recognized by the Standards Division, the unit within Statistics Canada responsible for maintaining the database. Division management indicated that it plans to improve the database by developing templates tailored to the specific needs of some program areas. Management recognized that a new approach is needed to help program managers maintain current information in the database.

6.59 Recommendation. Statistics Canada should ensure that the information in the Integrated Meta-database on data quality of health programs is complete.

Agency's response. Agreed. Statistics Canada will continue its efforts to improve the availability of complete and up-to-date information on the data quality of health programs through the Integrated Meta-database.

Insufficient monitoring of the quality of statistics supplied by the Canadian Institute for Health Information

6.60 Survey data and administrative data are substantially different in the degree of control that Statistics Canada has over quality assurance (paragraph 6.28). The quality of surveys conducted by the Agency is within its direct control. By contrast, the Agency does not have direct control over the management of the quality of administrative data collected by other organizations. Data from CIHI fall under this latter category. We expected that Statistics Canada would determine the quality of data from CIHI.

6.61 Although Statistics Canada has taken action to address concerns about some CIHI data, it has not taken steps to formally determine the quality of hospital morbidity data received (Exhibit 6.6). This concerns us because these data are used to construct health indicators and we could not conclude whether the data are accurate.

6.62 Statistics Canada's program reports state that the Agency maintains a data quality role for databases transferred to CIHI. They also state that, prior to release, the Agency examines the completeness of data, performs trend analysis, and does selective editing of records. The reports point to an extensive 1998 study that Statistics Canada conducted on data quality of the Annual Hospital Survey. As well, although Statistics Canada transferred the Residential Care Survey to CIHI, the survey was returned to Statistics Canada because of serious shortcomings in data quality.

6.63 However, we noted that Statistics Canada took an informal approach to the quality of hospital morbidity data from CIHI. We found little information on data quality for the two relevant databases. CIHI completed the first part of a data re-abstraction study on the Discharge Abstract Database, from which the Hospital Morbidity Database is derived. It published a summary of the study and, although a more detailed version of this document was available, this was not requested by Statistics Canada.

6.64 Data from these two databases are used extensively to monitor the use of acute care health services and to conduct analyses of health conditions and injuries. The data are also increasingly being used to track patient outcomes and are a major contributor to various reports and publications. Given their extensive use, the issue of data quality requires close attention.

6.65 Recommendation. Statistics Canada should make a regular, formal determination of the quality of the hospital morbidity data it receives from the Canadian Institute for Health Information (CIHI).

Agency's response. Agreed. Statistics Canada will work with CIHI to ensure regular reporting of data quality of the Hospital Morbidity Database.

6.66 Accountability requirements for the Health Information Contribution Program are not met. Following the creation of CIHI in 1994, Statistics Canada transferred several health statistics programs to CIHI. Under the five-year Health Information Contribution Program Agreement between the two organizations, Statistics Canada transferred $4.75 million between 1994-95 and 1999-2000. These funds accompanied the transfer of responsibility to CIHI for several surveys—mental health, therapeutic abortions, morbidity data, and the annual hospital survey. It should be noted that Statistics Canada did not formally delegate authority to CIHI under the Statistics Act to do this work; the work was done under the contribution agreement.

6.67 The contribution agreement outlines accountability requirements for the Health Information Contribution Program. These include the requirement for CIHI to make information on databases available to Statistics Canada and to provide Statistics Canada with a program evaluation and annual audit reports.

6.68 CIHI provided the annual audit reports to Statistics Canada through the Agency's representation on the CIHI Board. The agreement specified that CIHI provide a program evaluation in 1997-98 to Statistics Canada on the first three full years of its operation (and every three years thereafter, as long as CIHI receives funding under the agreement). This evaluation was to review and assess the value of the key objectives and results achieved, including impacts achieved in areas targeted by CIHI.

6.69 Instead of a formal evaluation, a consulting firm completed an external review for CIHI in 1997. It addressed issues related to the CIHI mandate, governance, and operations; however, it did not cover the key objectives and results achieved. Statistics Canada renewed the contribution agreement in 2000 for another five years ($2.8 million) without asking for a second evaluation, as required by the agreement. In addition, the renewed contribution agreement required CIHI to complete an evaluation framework for its overall activities and forward it to Statistics Canada by the end of 2000-01. However, Statistics Canada did not ensure that it was forwarded. We found that the Agency did not take action to ensure that the accountability provisions of the contribution agreement were respected.

6.70 Health Canada also contributed funds to CIHI under a Health Information Contribution Program contribution agreement. The Department transferred $13.4 million and several databases to CIHI between 1994-95 and 1999-2000. As was the case with Statistics Canada, Health Canada renewed the agreement for an additional five years in 2000-01 ($10.4 million) without ensuring a program evaluation was done. As well, the renewed contribution agreement required CIHI to complete an evaluation framework for its overall activities and forward it to Health Canada by the end of 2000-01. However, Health Canada did not ensure that it was forwarded. We found that the Department did not take action to ensure that the accountability provisions of the contribution agreement were respected.

6.71 Recommendation. Statistics Canada and Health Canada should formally monitor the requirements of their respective Health Information Contribution Program contribution agreements and ensure that requirements are met.

Statistics Canada's response. Statistics Canada has received and reviewed an evaluation framework for the contribution agreement with the Canadian Institute for Health Information. Plans are currently under way to undertake a formal evaluation.

Health Canada's response. Health Canada has already considerably increased its capacity to monitor the contribution agreements and improve its interaction with the Canadian Institute for Health Information. In addition, an evaluation framework has been jointly developed, by CIHI, Statistics Canada, and Health Canada, to be used in evaluating the contribution agreement. This framework will be used to guide a joint evaluation of the Health Information Contribution Program and the Health Information Roadmap Initiative to be completed by September 2003.

Limited reporting to Parliament on the quality of health statistics by Statistics Canada

6.72 In our 1999 audit we noted that, while the quality of statistics figured prominently in Statistics Canada's commitments to Parliament for results, its annual Departmental Performance Report provided limited information on the quality of its statistics. We recommended that this information be improved.

6.73 We found that Statistics Canada has restructured its Performance Report since our 1999 audit to address our recommendation. It now reports performance based on its Quality Assurance Framework. Several mission-critical statistical products are systematically reported on for relevance, accuracy, timeliness, accessibility, interpretability, and coherence.

6.74 We noted that the Health Statistics Division has undergone significant changes over the last few years to address health information needs; however, there is little mention of this in the Agency's performance reports for the periods ending 31 March 2000 and 31 March 2001. The Division has expanded in the last several years, in part because of Roadmap Initiative funding. Statistics Canada received about $40 million over four years (1999-2000 to 2002-03) to enhance or improve the quality of health statistics, including the Canadian Community Health Survey, Vital Statistics, and Canadian Cancer Registry. The performance reports do not identify accomplishments related to this increased funding; nor do they identify the Agency's partners in the production of health statistics at the federal, provincial, and territorial levels.

6.75 Annual Consolidation of Indicators not done. The Agency's 2000-01 guidelines for Integrated Program Reporting stated that the Annual Consolidation of Indicators of performance would be completed in the same year. Although we found that most of the information was available in Statistics Canada, it had not completed the consolidation work for the Health Statistics Division. Corporate Planning Division officials told us that this was because the full implementation of Integrated Program Reporting needed substantial time and effort and that the Annual Consolidation of Indicators component would be implemented over time. This represents a setback because these indicators provide a source of performance information for the Agency's Performance Report.

6.76 Recommendation. Statistics Canada should provide information in its Departmental Performance Report on the key results of its work on health statistics, including work undertaken as part of the Health Information Roadmap Initiative.

Agency's response. Agreed. Statistics Canada will include the results of its work on health statistics in its annual Departmental Performance Report, when appropriate.

Quality of Health Canada's health statistics

Limited application of data quality practices to health statistics

6.77 Health Canada collects data on communicable and chronic diseases, as well as on injuries, through its national surveillance systems. Our examination of Health Canada surveillance systems was limited to the three databases that provided data for the September 2002 health indicators reports—the Notifiable Disease Reporting System, the National Diabetes Surveillance System, and the Canadian Tuberculosis Reporting System. These three databases rely on administrative data provided voluntarily by the provinces and territories.

6.78 We expected that Health Canada would have applied quality assurance systems and practices to these three databases. We also expected that the Department would use the results of quality assurance systems and practices in managing quality and reporting performance.

6.79 We found that Health Canada does not have an overall quality assurance framework for its surveillance systems. Although the Department does conduct some quality assurance work, it is primarily ad hoc and varies by surveillance system. Quality assurance mechanisms are weak and do not result in concrete quality measures. It is therefore not possible to conclude whether the surveillance data are accurate.

6.80 We reviewed the work undertaken by Health Canada on the three surveillance systems to ensure good-quality statistics. The Department does some work to improve data quality, including both automated and manual checks for the data it receives from provinces and territories. These include checks to ensure interprovincial consistency of data coding. Health Canada sends data with errors back to the source jurisdictions to confirm the accuracy of the data. As well, because not all jurisdictions use the same version of the International Classification of Diseases codes, the Department does some coding conversions to ensure that the integrity of its databases is maintained with one standardized coding system.

6.81 However, Health Canada has no systematic procedure for edit checks and those that are done are ad hoc rather than in response to a formal policy requirement. As well, the Department has not established standards for data quality. Nor has it measured the quality of the data, particularly in terms of completeness and accuracy. Therefore, Health Canada cannot determine the accuracy of its data for these three surveillance systems.

6.82 Our observations reflect those made in our September 2002 Report, Chapter 2, National Health Surveillance. The chapter noted that Health Canada had made limited progress toward resolving weaknesses identified in our 1999 audit. It also noted that many surveillance systems still lack timely, accurate, and complete information on diseases.

6.83 Recommendation. Health Canada should adopt, in collaboration with the provinces and territories, a common quality assurance framework and standards that would outline quality requirements for its health surveillance systems.

Health Canada's response. Health Canada will work with the provinces and territories to develop a quality assurance framework and standards for its health surveillance systems. These standards will make explicit what heretofore have been informal procedures and will make it easier to assess the completeness and accuracy of the surveillance data received from the provinces and territories.

Verifying the accuracy of health indicators data

6.84 In September 2000, First Ministers issued their Communiqué on Health that committed health ministers to report regularly on health status, health outcomes, and the performance of publicly funded health services (Exhibit 6.1). The Conference of Deputy Ministers of Health approved 67 indicators against which jurisdictions were to report. The First Ministers directed health ministers to determine appropriate, independent, third party verification of the accuracy of the health indicators data.

Work undertaken by the Canadian Council of Legislative Auditors

6.85 In the same way that it is important for jurisdictions to achieve comparable reporting on indicators, it is also important that health indicators reports be verified in a comparable manner. The Canadian Council of Legislative Auditors, made up of provincial auditors general and the Auditor General of Canada, established the Health Indicators Study Group. This group assists legislative audit offices in their respective responsibilities related to the First Ministers' agreement. Legislative audit offices across the country were nominated by their governments to do the verification.

6.86 Health Canada was tasked with compiling the federal health indicators report. We accepted the Department's request for us to undertake third party verification of the report. As the legislative auditors for Nunavut, the Northwest Territories, and Yukon, we also verified their health indicators reports upon their request.

6.87 Members of the Health Indicators Study Group worked with their respective health ministries to reach an agreement on the form the verification would take. Although the overall goal of the Group was to provide audit-level assurance, this was not possible in all cases. In some cases, external auditors provided audit-level assurance. In other cases, they were engaged to perform specified auditing procedures (see the Appendix).

6.88 Health indicators data came from Statistics Canada, Health Canada, CIHI, and provincial and territorial databases. Statistics Canada and, to a lesser extent, Health Canada and CIHI, provided data for the majority of health indicators. We provided provincial audit offices with the results of our audit of the quality of statistics produced by Statistics Canada and Health Canada that were relevant to their third party verification work. We also participated in work led by the Office of the Auditor General of British Columbia on statistics produced by CIHI for seven of the health indicators. The legislative audit offices used the results of this examination of CIHI for their verification work.

6.89 We expected that, in producing data for health indicators reports, a determination of the quality of data received from other organizations would be made, data elements would be accurately extracted from the data source, definitions and formulae would be correctly applied, calculations and presentations would be free of errors, and quality limitations would be adequately disclosed.

6.90 We found that Statistics Canada databases met the Agency's standards for quality assurance, with the exception of the Hospital Person-oriented Database, which relies on CIHI data (paragraphs 6.35 to 6.50). The data were accurately extracted from the data source, definitions and formulae were correctly applied, and calculations and presentations were free of errors.

6.91 We found that Health Canada's quality assurance systems and practices were inadequate and did not ensure accurate data (paragraphs 6.77 to 6.82). The Department was unable to demonstrate how complete or accurate its data were. We were therefore unable to form an opinion on the accuracy of these data.

6.92 In the work led by the Office of the Auditor General of British Columbia to determine the accuracy of the data CIHI provided for the September 2002 health indicators reports, the required data were drawn from both the Discharge Abstract Database and the Hospital Morbidity Database. As a result of this work, it was concluded that documentation on the CIHI quality assurance process for the systems that support the indicators was inadequate. As well, a three-year re-abstraction study, which will provide information on the quality of the input data, will not be completed for another two years. Therefore, it was not possible to form an opinion on the accuracy of the specific data provided by CIHI for the health indicators reports.

Meeting health information needs

6.93 The importance of developing health indicators has been recognized for several years. In its 1999 Budget, the federal government identified a number of specific priority projects and activities in the health information field and allocated $95 million over four years toward their completion. CIHI was expected to work with provincial and territorial governments to build consensus over selecting which health indicators to report, developing standards for data, filling key gaps in information, and building capacity to analyze data and disseminate information to those who need it. In its 2001 Budget, the federal government announced that it would provide a further $95 million to CIHI, to be used in co-operation with Statistics Canada. This included funding for developing common health indicators so that nationwide, comparable information would be available to Canadians.

6.94 The grants were made by Health Canada to CIHI. Under the terms of the 1999 grant, Health Canada was to receive an annual report from CIHI. It was to include a statement of CIHI's objectives for that year under the Health Information Roadmap Initiative and the extent to which CIHI met those objectives. As well, CIHI was to develop an evaluation plan in 1999-2000, to be provided to the Minister of Health, and it was to keep records to facilitate an evaluation of the grant.

6.95 We expected that the Roadmap Initiative grant would be monitored by signatories according to responsibilities outlined in agreements, including whether work had been undertaken as outlined in the agreements.

Health Canada's monitoring of the Health Information Roadmap Initiative is weak

6.96 We found that Health Canada did not actively monitor the Health Information Roadmap Initiative to ensure that CIHI complied with the requirements outlined in the grant. In particular, the Department did not ensure that CIHI submitted an evaluation plan. We reviewed CIHI's annual Roadmap Initiative reports to determine if they provided information on the extent to which Roadmap objectives were achieved. Although the reports describe activities that took place, there is little information on the extent to which they contributed to the achievement of Roadmap objectives. We found that no formal evaluation was undertaken and submitted to Health Canada before the grant was renewed for an additional $95 million. Finally, we found that accountability arrangements were unclear—that funding and reporting relationships did not facilitate clear accountability relationships. Overall, Health Canada is not in a position to know if implementation of the Roadmap Initiative is consistent with the funding agreement.

6.97 Recommendation. Health Canada should monitor the Health Information Roadmap Initiative in accordance with the grant requirements to ensure that the requirements are met.

Health Canada's response. Health Canada and CIHI have already jointly developed an evaluation framework. This framework will be used to conduct a formal third party evaluation of the Health Information Roadmap Initiative to be completed by September 2003, in accordance with the timelines laid out in the funding agreement. There has been an ongoing and comprehensive flow of information between CIHI and Health Canada. This information flow and the accompanying dialogue provided a solid basis for the renewal of the Health Information Roadmap Initiative.

Conclusion

6.98 There are no absolute standards for the quality of statistics. Implementing sound data quality practices requires continuous work, monitoring, review, and adjustment. There is always room to improve the quality of statistics.

6.99 Our audit found that Statistic's Canada's quality assurance systems and practices are sound. Health Statistics Division has applied quality assurance procedures to its statistical programs, has undertaken work resulting in measures demonstrating that its programs meet quality standards, and has completed reviews of its quality assurance practices. However, we did find some weaknesses in Statistics Canada's documentation of data quality achieved.

6.100 Statistics Canada has a policy for informing users of data quality and methodology. However, Health Statistics Division's application of the policy is uneven and references to the quality of its health statistics in the Agency's Integrated Meta-database are incomplete.

6.101 Although Statistics Canada takes steps to formally determine the quality of data it receives from most external sources, it does not take steps to formally determine the quality of hospital morbidity data from CIHI. This concerns us because these data are used to construct health outcomes data.

6.102 We found that Statistics Canada's application of quality assurance systems and practices adequately ensured the accuracy of the health indicators data produced for the September 2002 federal, provincial, and territorial health indicators reports.

6.103 For Health Canada, we found that participation on the part of provinces and territories in surveillance databases is voluntary and there is a lack of formal federal/provincial/territorial agreements on data sharing, data standards, and data definitions. The quality assurance systems and practices for these databases are inadequate to ensure the accuracy of the data. Health Canada stated in the federal health indicators report that improvements in data quality are needed. However, we were unable to reach a conclusion on the accuracy of the Health Canada data.

6.104 Audit work led by the Office of the Auditor General of British Columbia found that documentation is inadequate on the quality assurance process for the systems and practices that CIHI uses to provide data for seven health indicators for the September 2002 health indicators reports. As well, the three-year re-abstraction study, which will provide information on the quality of the input data, will not be completed for another two years. Therefore, it was not possible to form an opinion on the accuracy of the specific data provided by CIHI for the health indicators reports.

6.105 The federal government has recognized the importance of developing health indicators to meet information needs. Health Canada needs to actively monitor the Health Information Roadmap Initiative to ensure that accountability requirements are met and that implementation is consistent with the funding agreement.

6.106 Overall, the audit found that Statistics Canada played a key role in providing good-quality statistics that contribute to an informed public debate on health issues and, more specifically, to the September 2002 health indicators reports. Our audit work provides assurance on the quality of these statistics.

About the Audit

Objectives

The objectives of the audit were to determine whether

  • Statistics Canada has put in place appropriate systems and practices to adequately ensure the quality of its health statistics;
  • Statistics Canada, Health Canada, and the Canadian Institute for Health Information have applied quality assurance systems and practices to ensure the accuracy of the health indicators data that they produced for inclusion in the federal, provincial, and territorial health indicators reports; and
  • the implementation of the Health Information Roadmap Initiative is consistent with the funding agreement.

Scope and approach

The audit focussed on Statistics Canada's quality assurance systems and practices as they relate to the Agency's health statistics programs. The audit team carried out work to determine the adequacy of quality assurance work undertaken on health statistics programs. As well, Statistics Canada and the audit team agreed that the Agency would do a self-assessment of data quality for a product, the Canadian Community Health Survey, which was not included in its quality reviews to date. The audit team subsequently audited the self-assessment.

The audit examined how Statistics Canada determined the quality of data received from other organizations and its compliance with its policy on informing users of data quality and methodology. The audit also examined the accuracy of data produced by Statistics Canada, Health Canada, and the Canadian Institute for Health Information for inclusion in the federal, provincial, and territorial health indicators reports released in September 2002. Finally, the audit reviewed the Health Information Roadmap Initiative and Health Information Contribution Program to ensure that the requirements of the agreements were met.

In carrying out this work, we interviewed managers responsible for health statistics at Statistics Canada. We also reviewed Agency publications, reports, and internal documents. As well, we reviewed work at Health Canada related to health indicators.

Members of the audit team participated in work led by the Office of the Auditor General of British Columbia to examine the quality assurance systems and practices in place for the Canadian Institute for Health Information's Hospital Morbidity Database and Discharge Abstract Database, from which the former is derived. This work was undertaken because Statistics Canada uses the Hospital Morbidity Database, and several health indicators rely on it as a source of data.

Criteria

We expected the following:

  • Statistics Canada would have quality assurance systems and practices that are applied across its statistical programs.
  • The adequacy of quality assurance systems and practices in individual statistical programs and products would be assessed systematically. This includes a determination of the quality of data received from other organizations.
  • The results of quality assurance systems and practices would be used in managing quality and in reporting performance.
  • The Agency would have an effective policy on informing users about data quality and methodology to help ensure informed use of its statistical products.
  • Individual statistical programs would appropriately inform users about data quality and methodology when data are disseminated.
  • In producing data for health indicators reports, the quality of data received from other organizations would be determined, data elements would be accurately extracted from the data source, definitions and formulae would be correctly applied, calculations and presentations would be free of errors, and quality limitations would be adequately disclosed.
  • The Health Information Roadmap Initiative grant would be monitored by signatories according to responsibilities outlined in agreements, including determining whether work has been undertaken as outlined in the agreements.

Audit team

Assistant Auditor General: Maria Barrados
Principal: Barry Leighton
Director: Glenn Wheeler

Yves Genest
Denis Jobin
Catherine Livingstone
John McGrath
Albert Melanson
Paul Pilon

For information, please contact Communications at (613) 995-3708 or
1-888-761-5953 (toll-free).