This Web page has been archived on the Web.

2007 October Report of the Auditor General of Canada

Main Points

What we examined

A census is a snapshot of a population's size and its demographic, social, and economic characteristics at a point in time. Statistics Canada, as Canada's national statistical agency, is required to carry out a census of the Canadian population every five years.

We examined whether Statistics Canada applied its established quality assurance systems and practices in managing the 2006 Census of Population. Our past audits found the Agency's quality assurance systems and practices to be sound. We assessed the Agency's efforts to improve the quality of Census data on selected hard-to-count groups, and we looked at how Census data meet the information needs of key government clients. We did not directly assess the quality of the 2006 Census data.

We also examined to what extent the 2006 Census program complied with the government's risk management policy, particularly in its ability to recruit and retain the temporary field staff needed and in managing the risks to the privacy of respondents.

Why it's important

The Census of Population is Statistics Canada's largest survey program. The key clients for census data are governments at all levels, who use it for program planning, analysis, and decision making. Federal transfer payments to the provinces are also based in part on census population estimates. In the 2006–07 fiscal year, these payments amounted to about $62 billion. Private businesses, social institutions, researchers, and academics are other major users of Census data. Population counts from every second (or decennial) census are used to adjust federal electoral boundaries. Statistics Canada recognizes the need to ensure that Census data are of sufficient quality for these uses.

What we found

• Statistics Canada satisfactorily managed the 2006 Census of Population in accordance with its quality assurance systems and practices. It took steps to improve the quality of the information collected on population subgroups identified in the 2001 Census as hard to count. It also consulted with its key government clients for Census information to understand and meet their needs. During the data collection process, data accuracy was balanced with cost and timeliness.
• Statistics Canada did not prepare an integrated and comprehensive document, as it had committed to do, outlining how it planned to satisfy the requirements of its quality assurance systems and practices for the Census. A thoroughly documented data quality management plan could have enabled the Agency to better set out how data quality would be achieved. In addition, while Statistics Canada has assessed the performance of some elements of the 2006 Census program, it has not yet completed an integrated program review. Such a review would support internal program management as the Agency prepares for the 2011 Census as well as external accountability to Parliament and Canadians.
• Statistics Canada took a proactive approach to identifying risks to the 2006 Census. However, it did not fully comply with the requirements of the government's policy on risk management. In particular, despite the numbers of temporary field staff it needed and the known challenges it faced in hiring and retaining them, it did not develop formal and detailed contingency plans to respond in the event that it could not meet those challenges. The difficulties the Agency faced in hiring and retaining the required numbers of field staff prompted its decision to delay the first data release. In addition, the staffing situation could have had an impact on the accuracy of the data for some small geographic areas and sub-populations. Should any such impact have occurred, it would be evident only upon the release of these detailed data. The timing of this audit did not allow us to examine these data.
• The Agency managed risks related to the privacy of respondent information with significant and successful efforts to ensure that the privacy of Census data was protected. Those efforts addressed privacy concerns expressed after the 2001 Census.

The Agency has responded. The Agency agrees with all of our recommendations. Its detailed response follows each recommendation throughout the chapter.

Introduction

6.1 As Canada's national statistical agency, Statistics Canada's mandate is to provide Canadians with statistical information that is relevant, responsive, and of high quality. A key component of this mandate is the Agency's obligation to carry out censuses of population—by law, a census is conducted every five years. Statistics Canada defines a census as the stock-taking of the demographic, social, and economic characteristics of the population at a given point in time.

6.2 The Census of Population is Statistics Canada's largest survey program, and it is the only source of detailed data on the Canadian population for small geographic areas and sub-populations. The budget for the 2006 Census of Population is $567 million. The Agency allocated the equivalent of approximately 3,900 full-time employees to the program and planned to hire approximately 27,000 temporary field staff for the Census.

6.3 Census data have a wide variety of uses, including the following:

• The federal government distributes billions of dollars (approximately $62 billion in the 2006–07 fiscal year) to the provinces and territories based, in part, on Census population estimates. This includes health and social transfers, territorial formula financing, and equalization payments.
• Federal, provincial, territorial, and municipal governments use census data for program planning and analysis activities. Private businesses, social institutions, researchers, and academics also use the data.
• Population counts from every second (decennial) Census are used to adjust federal electoral boundaries.
• Statistics Canada uses the Census database to support several important surveys that it conducts.

6.4 Census questionnaires are sent to all identifiable dwellings, to count and describe the individuals who live in them. Canadians who live overseas (for example, military personnel and diplomats) are also counted. However, because only people living in dwellings are included, homeless people who are in shelters on Census day are counted but those on the street are not.

Changes to questionnaire content for the 2006 Census

6.5 Statistics Canada consults widely on the content of each Census. During the content determination period, the Agency considers the implications of proposed content changes on respondent burden, privacy, and cost. Across successive censuses, the Agency aims to balance the pressure for change with the desire to maintain continuity and comparability.

6.6 In 2006, changes to the Census questions were limited to creating or modifying questions on same-sex couples, ethnic origin, and educational background. Respondents were also asked to allow Statistics Canada access to their income tax files, for income information, and to authorize the release of their personal information after 92 years, for genealogical and historical research purposes.

6.7 The long form questionnaire contained 61 questions and was distributed to 20 percent of dwellings. The short form questionnaire contained 8 questions, all of which appeared on the long form, and was provided to the remaining 80 percent of dwellings.

Changes in methodology

6.8 According to Statistics Canada, the 2006 Census included some of the most significant changes to the program's collection and processing methodologies in over 30 years. The following are some of the prominent changes:

• Dwellings in approximately 70 percent of the country received their Census questionnaire by mail. The remaining questionnaires were delivered, as they had been in the past, by local field staff.
• Canadians with access to the Internet could complete their questionnaires online.
• Almost all completed questionnaires were returned to a single data-processing centre, instead of to local field staff.
• The follow-up for most of the questionnaires that failed completeness tests was done by telephone, from call centres, rather than by local field staff.

In May 2004, a large-scale test was conducted of the new technology, methods, and systems, involving 300,000 dwellings.

Management structure

6.9 Overall responsibility for the 2006 Census resides with the Census Manager who reports to the Assistant Chief Statistician, Social, Institutions and Labour Statistics Field. The Census Manager is accountable to the Census Steering Committee and the Agency's Policy Committee (chaired by the Chief Statistician), and is supported by the Census Project Team, which is the decision-making and managing body for the Census.

6.10 At the next level, project managers are responsible for the projects that make up the Census program. Their work included setting up 36 local census offices, 3 call centres, a national warehouse, and a data-processing centre. Statistics Canada's regional offices in Montreal, Toronto, and Edmonton were responsible for carrying out the 2006 Census in the field.

Time frame for 2006 Census

6.11 Each Census program spans approximately eight years, from start to finish, and overlaps with the one before and after it. Although the first release of data from the 2006 Census occurred in March 2007, planning for the 2011 Census was already under way by then. Exhibit 6.1 lists key dates for the 2006 Census, and Exhibit 6.2 lists the key phases of the 2006 Census.

Exhibit 6.1—Key dates for the 2006 Census of Population

Exhibit 6.2—Key phases of the 2006 Census of Population

Focus of the audit

6.12 The focus of this audit was to assess whether Statistics Canada applied its quality assurance systems and practices to the 2006 Census of Population. We also examined how the Agency applied the government's risk management policy to the census program.

6.13 The timing of this audit allowed us to examine the planning and conduct of the Census up to and including the release of population and dwelling counts on 13 March 2007. Statistics Canada is not scheduled to release the final data from the 2006 Census until 2008. The timing of this audit also allows Statistics Canada to implement our recommendations as it plans for the 2011 Census. It is important to note that we did not assess the quality of the data from the 2006 Census.

6.14 More details on the audit objectives, criteria, scope, and approach are in About the Audit at the end of this chapter.

Observations and Recommendations

Quality assurance systems and practices

6.15 Statistics Canada's product is information. The Agency maintains that if the quality of its data becomes suspect, its reputation as an independent and objective source of trustworthy information will be undermined. The management of data quality must, therefore, play a central role within the Agency. The Quality Assurance Framework (QAF) and companion Quality Guidelines describe the approaches that Statistics Canada takes to manage data quality.

6.16 Statistics Canada prepared its QAF in anticipation of the Office of the Auditor General's 1999 audit of the management of the quality of statistics. The Framework and the Guidelines describe measures that Statistics Canada applies to its survey programs, including censuses of population, and are intended to ensure that the resulting information is "fit for use," meaning that its level of quality is acceptable for its intended uses. The QAF also requires that the performance of the Agency's programs be documented. In our 1999 and 2002 audits of Statistics Canada, we found the Agency's quality assurance systems and practices to be sound.

6.17 Statistics Canada defines data quality in terms of six elements: relevance, accuracy, timeliness, accessibility, interpretability, and coherence (Exhibit 6.3).

Exhibit 6.3—Statistics Canada's six elements of data quality

6.18 According to Statistics Canada, these elements of quality overlap and are interrelated. There is no general model that brings them together to prescribe a level of quality, and they do not apply equally to all survey programs. An acceptable level of quality can be achieved by addressing, managing, and balancing these elements, while paying due attention to program objectives, costs, respondent burden, and other factors that may affect information quality or user expectations.

6.19 In this audit, we assessed whether Statistics Canada managed the 2006 Census according to its quality assurance systems and practices, to ensure that the data are fit for use. We examined how the Agency addressed the six elements of quality as well as how it balanced these elements against costs, user expectations, and respondent burden.

6.20 We also assessed whether Statistics Canada documented its plan for managing and achieving the data quality consistent with the QAF. In addition, we examined whether the Agency documented its plan for assessing the quality of the resulting data. Documented plans support both program decisions and the continuity of operations within the Agency.

6.21 We also assessed whether Statistics Canada reviewed the performance of the Census program as required by the QAF. This information is important because it facilitates external accountability to Parliament and Canadians by allowing for independent review and public debate.

Statistics Canada satisfactorily managed the quality of the 2006 Census

6.22 We found that Statistics Canada applied the requirements of its quality assurance systems and practices to the management of the 2006 Census of Population. This included balancing the six elements of quality with cost, respondent burden, and client needs.

6.23 We found that Statistics Canada satisfied the requirements of the Quality Assurance Framework for all elements of quality, except interpretability. Although these requirements for interpretability were largely met, the requirement to report the response rate in the 13 March 2007 population and dwelling count release was not.

6.24 Our observations concerning relevance, accessibility, and coherence are presented later in the section on key government clients (see paragraphs 6.41–6.44). The following observations concern timeliness, interpretability, and accuracy.

6.25 Timeliness. As required by the QAF, Statistics Canada announced its release schedule for 2006 Census data well in advance of the planned release dates. The dates were earlier than those of the 2001 Census, reflecting the efficiencies anticipated from the changes in methodology. However, in October 2006, the Agency revised the date of its initial release, by one month (from February to March 2007), because difficulties in hiring and retaining field staff had delayed the collection of data. This decision demonstrated Statistics Canada's efforts to balance timeliness with accuracy and cost. Our observations on field staffing are presented later in the chapter (see paragraphs 6.69–6.87).

6.26 Interpretability. Statistics Canada provides a wide variety of information to help users interpret and use Census information. Response rate is an important indicator of data accuracy that users require to interpret survey data. The Agency's QAF and its Policy on Informing Users of Data Quality and Methodology require that this information be provided to users.

6.27 For the 2001 Census population and dwelling release, Statistics Canada presented the response rate as a single number. However, for the 2006 Census population and dwelling count release (13 March 2007), the Agency provided several numbers, including those needed to calculate the response rate, but left it to users to decide which numbers to use and how to use them. Statistics Canada maintains that changes to the methodology, introduced in 2006, preclude reporting a single-number response rate. However, without a clear, unambiguous response rate, users do not have the information necessary to interpret the data.

6.28 Accuracy. Accuracy is the degree to which information correctly describes the phenomena it was designed to measure and is usually characterized in statistical terms. The major causes of inaccuracy include incomplete survey coverage, sampling error, and non-response.

6.29 Statistics Canada demonstrated that it managed the accuracy of the data for key phases of the Census, as required by the QAF. At the design stage of the 2006 Census, questionnaires were tested in both paper and Internet modes of collection. New technologies introduced for the 2006 Census were included in the 2004 Census test. In addition, measures were put in place to follow up on non-response and to deal with missing and logically inconsistent data. At the data collection and processing phases, quality checks were put in place to monitor data accuracy. Information systems were designed to allow management to monitor cost, progress, timeliness, productivity, and quality at various geographic levels.

6.30 Statistics Canada maintains that a key indicator of data accuracy is survey coverage. As part of its efforts to assess the completeness of the coverage of the Census, the Agency conducts a survey of dwellings from which no questionnaire was returned. The survey results are used to estimate how many of these dwellings were occupied and how many people were living in them on Census day and to make adjustments to the Census database. In 2006, after adjustments were made to the database, 935,605 people and 435,789 dwellings were added to the population enumerated in the Census. Other studies that are currently being conducted are used to estimate coverage but not to adjust the results of the Census. Their results are scheduled for release in September 2008.

6.31 Before data are released, subject matter experts within the Agency review them to ensure that they are fit for use. During this review, referred to as certification, the experts consider how well the data align with current knowledge and research as well as with other data sources. Checks against previous censuses are also conducted to ensure historic comparability.

6.32 At the time of our audit, the certification process had been completed only for the population and dwelling counts release (13 March 2007). Statistics Canada's Policy Committee, a senior committee chaired by the Chief Statistician of Canada, reviewed and approved the data and released it, under the Chief Statistician's authority. We found that the certification process for the March 2007 population and dwelling release was applied appropriately.

6.33 However, we noted that this release was not authorized by a formal record of decision. Statistics Canada maintains that the Chief Statistician is accountable for all data released by Statistics Canada and, in that capacity, signs a formal record of decision only when legally required to do so. For the 2006 Census, the Chief Statistician will be required to sign a record of decision only when the population estimates are provided to the Department of Finance Canada in 2008. However, a formal record of decision is important not only for external accountability, but also for internal management purposes.

Efforts are being made to improve coverage of hard-to-count groups

6.34 Statistics Canada maintains that survey coverage is a key element of data accuracy. Under-coverage occurs when the Census misses individuals or dwellings. Under-coverage rates vary across locations and population groups. From 1971 to 2001, overall census under-coverage rates in Canada have increased. However, in the 2001 Census, the overall under-coverage rate was still under three percent. The comparable figure for the 2006 Census is scheduled to be released in 2008.

6.35 In this audit, we examined whether the Agency applied its quality assurance systems and practices to improve the coverage of selected hard-to-count groups. In designing and conducting the 2006 Census of Population, we expected the Agency to

• identify hard-to-count segments of the population, from the 2001 Census and other sources;
• conduct research to understand why these groups were hard to count;
• develop and carry out strategies to improve coverage of these groups in the 2006 Census of Population; and
• determine the effectiveness of the strategies.

6.36 We found that Statistics Canada had identified young adult males, First Nations reserves, members of cultural and linguistic minorities, residents of collective dwellings (for example, seniors' residences), and residents of the rural and remote North as hard-to-count groups. This was done largely through analyses completed following the 2001 Census.

6.37 We found that Statistics Canada had conducted consultations and analyses on why some of these groups—namely, First Nations reserves, residents of collective dwellings, and residents of the rural and remote North—were hard to count. Only limited research was carried out to better understand the under-coverage of young adult males and cultural and linguistic minorities.

6.38 We found that Statistics Canada had developed and carried out strategies to improve the coverage for hard-to-count groups—from generic, public communications to formal and structured efforts that included consultation, targeted communications, and specialized recruitment and training of Census staff. In addition, for the 2006 Census, the Agency improved its procedures for enumerating collective dwellings.

6.39 Statistics Canada made an effort to improve coverage of First Nations reserves through its ongoing liaison efforts, hiring field staff from the reserves, providing communication materials specifically intended to promote the Census message to Aboriginal peoples, and partnering with First Nations and other Aboriginal organizations to facilitate data collection.

6.40 Information on the coverage of these hard-to-count groups will not be available until 2008, when the coverage studies are scheduled for release. This information will assist the Agency in determining the effectiveness of its strategies. In the interim, as part of the planning process for the 2011 Census, Statistics Canada is identifying lessons learned by debriefing Census staff.

Key government clients for Census information were consulted

6.41 The Census of Population aims to respond to the information needs of key government clients, such as the Department of Finance Canada, that rely on population estimates to distribute transfer payments to the provinces and territories, and federal departments and agencies that contribute financially to the Census of Population. The provinces, territories, and municipalities are also important clients who use Census information for service planning and analysis.

6.42 Because census data have many important uses at all levels of governments, we examined the processes that Statistics Canada follows to consult with key government clients and to respond to their information needs (by ensuring that Census data are relevant, accessible, and coherent). We expected the Agency to take the necessary steps to understand the information needs of its key government clients and to meet these needs, given the constraints of cost and respondent burden.

6.43 We found that, as it planned and conducted the 2006 Census, Statistics Canada monitored the needs of its clients and responded to them. The Agency had a number of formal processes in place to understand the needs of its key government clients—in particular, structured efforts to consult clients on the content of the 2006 Census questionnaires, on data access, and on reporting. Through these consultations, Statistics Canada identified possible additional content for the 2006 Census, expanded its product offerings, offered more products via the Internet, and increased data accessibility.

6.44 The key government clients of the Census program that we interviewed informed us that the quality of the Census data is sufficient for their uses.

Statistics Canada balanced data accuracy with cost and timeliness

6.45 Statistics Canada's quality assurance systems and practices call for a balance to be struck among the six elements of quality, as well as program objectives, cost, respondent burden, and other factors that may affect the quality of information or user expectations.

6.46 During the data collection phase, Statistics Canada encountered the following two challenges, which led it to trade off accuracy against cost and timeliness:

• The first challenge was the higher-than-expected number of dwellings requiring follow-up for their failure to return a completed questionnaire, referred to as non-response follow-up.
• The second challenge concerned recruiting and retaining temporary field staff (see paragraphs 6.69–6.87).

6.47 Non-response follow-up workload. The number of dwellings requiring follow-up is driven by the response rate to the initial distribution of questionnaires. The higher the response rate, the lower the number of dwellings requiring follow-up. In 2006, Statistics Canada anticipated that follow-up of approximately 3.1 million non-responding dwellings would be required, which the Agency stated is based on the results from the 2001 Census. According to Statistics Canada, the actual number of dwellings requiring follow-up was approximately 3.5 million, an increase of approximately 400,000 or 13 percent.

6.48 Statistics Canada responded to this challenge in several ways. For example, it transferred some of the follow-up work to the three call centres that had been set up to handle enquiries and complete questionnaires by telephone. The Agency also extended the schedule for this phase of the Census by six weeks, changing the completion date for follow-up from 21 July to 31 August. At that time, Statistics Canada ceased follow-up activities to minimize their impact on the cost of collecting data and on the planned date of the first data release. The decision to cease follow-up also meant accepting lower-than-planned response rates in some locations.

6.49 Accepted response rates. In 2006, Statistics Canada identified the smallest geographic area used for collecting data for the Census as a collection unit. For the 2006 Census, there were approximately 47,500 collection units across Canada. The response rate for a collection unit is calculated by dividing the number of dwellings from which a questionnaire has been returned by the total number of dwellings classified by field staff as occupied. In the 2006 Census, the target response rate was 98 percent.

6.50 The response rate is checked at the collection unit level during data collection, to promote uniformity in the accuracy of Census data across geographic areas and to ensure that enough data are gathered in small geographic areas to support reporting at that level. Statistics Canada performed these checks to focus follow-up activities in areas with the highest non-response rates.

6.51 In all, approximately 55 percent of collection units met the planned response rate target. A further 35 percent were formally approved during data collection even though they were below target, based on the costs and benefits of continuing to try to meet the target. When fieldwork largely ceased, at the end of August, the remaining 10 percent of collection units (also below target) were, in effect, accepted (with an average response rate of approximately 94 percent). This situation may reflect the difficulties experienced in some locations recruiting and retaining field staff.

6.52 Statistics Canada's decision to stop field collection activities at the end of August was a trade off between accuracy and both the cost of completing these activities and the need to meet the data release schedule. The national response rate to the 2006 Census of Population was 96.5 percent, slightly lower than the 98.4 percent achieved in 2001 (consistent with general trends in survey response rates). While 96.5 percent is a very high response rate by survey standards, even small increases in non-response could have an impact on data accuracy for small geographic areas and sub-populations. Should any such impact have occurred, it would be evident only upon the release of the data. The timing of this audit did not allow us to examine these data.

Data quality management planning and performance need to be better documented

6.53 When it was planning the 2006 Census, the Agency described an overarching data quality management plan as being central to achieving an acceptable level of quality. The goal of the plan was to ensure that each step met the quality objectives and adhered to the Agency's policies. In addition, the Quality Assurance Framework emphasizes the need to determine whether the Census design was carried out as planned, whether some aspects of the design were problematic to carry out, and what lessons were learned that could benefit future design.

6.54 Therefore, we expected to find a comprehensive plan that documented how the required data quality would be achieved as well as a document that described how data quality would be assessed. The results of this assessment should be available to the Agency as it finalizes its plans for the 2011 Census.

6.55 However, Statistics Canada did not prepare a comprehensive plan documenting how the quality of the data from the 2006 Census would be managed. In the absence of such a plan, a number of committees monitored and managed data quality throughout the Census. Documentation of such a plan is important to set out how data quality would be managed and achieved. The information gained then informs decisions and supports the continuity of operations within the Agency.

6.56 The QAF also requires each program at Statistics Canada to produce a report that documents its performance, lays out its future direction, and proposes changes related to the management of quality. For the 2006 Census, Statistics Canada did not prepare a comprehensive assessment plan. It will be more difficult to assess this program systematically and coherently without such a plan.

6.57 As previously noted, the Agency made a number of changes to the methodology for the 2006 Census of Population and faced a number of challenges during data collection. In addition, the Census is a costly undertaking and of importance to Canadians. Therefore, it is critical that it carry out a comprehensive and integrated review of the Census of Population program. While Statistics Canada has assessed the performance of some elements of the 2006 Census program, it has not yet completed an integrated program review. Such a review would enable the Agency to determine whether the program was managed according to its quality management systems and practices and to better plan the 2011 Census. It would also serve to provide performance information necessary for external accountability to Parliament and Canadians.

6.58 Recommendation. Statistics Canada should complete an integrated review of the 2006 Census program prior to finalizing its plans for the 2011 Census.

Agency's response. Agreed. Statistics Canada will complete an integrated program review of the 2006 Census. Evaluating results from one census to the next is a standard practice. The review is conducted in a phased approach as each major activity in the Census cycle is completed and is critical to the planning process for the next Census. The evaluation of and lessons learned from the 2006 Census are key inputs to the 2011 Census Planning documents, such as the Treasury Board Submission; Planning Assumptions; Volumetric Assumptions; and Business, User, and Systems Requirements.

6.59 Recommendation. As it plans for the 2011 Census, Statistics Canada should prepare a comprehensive document that outlines its data quality management plan for the Census program. The Agency should also prepare a comprehensive assessment plan to guide it in determining whether the Census program satisfies the requirements of the Agency's quality assurance systems and practices.

Agency's response. Agreed. For the 2011 Census, Statistics Canada will consolidate, in a single comprehensive document, the various practices and procedures in effect that measure and assess the data quality and its impact for the key phases of the program.

Statistics Canada has a well established and effective set of practices and procedures in effect for each major phase of the Census program, in conformance with the Agency's Quality Assurance Framework, Quality Guidelines, and Policy on Informing Users of Data Quality and Methodology.

Risk management

6.60 The Treasury Board's risk management policy requires that program managers proactively identify risks to their programs, take steps to minimize these risks before an incident occurs, contain the effects of and recover from risks that do occur, and learn from these incidents to improve program management in the future.

6.61 The policy also requires that departments and agencies develop formal and detailed contingency plans to prepare for identified risks. These plans—which define the measures to be taken to minimize the impact of a risk once it occurs, including costs, timing, and staffing—are to be kept up-to-date and ready for implementation.

6.62 The tight timelines established by Statistics Canada for the data collection phase of the Census underscore the importance of having fully developed contingency plans in place well before Census Day to minimize the potential impact of risks. Census managers do not have time to consider and develop optimal response strategies once an incident has occurred, when an immediate response is required to minimize adverse consequences.

6.63 Because of its complexity, cost, and importance to Canadians, the Census of Population merits a structured and rigorous approach to risk management. We assessed Statistics Canada's management of the 2006 Census against government policy on risk management. In addition to looking at the Agency's overall efforts, we specifically examined its efforts to address two important risks identified during the planning phase: field staffing and the privacy of respondents' information.

Statistics Canada did not fully comply with the government's risk management policy

6.64 In 2002, Statistics Canada started work on its overall risk management approach for the 2006 Census. The Agency hired external experts to conduct a risk assessment and to develop a risk management strategy with Census managers. This led to the preparation of a risk register that rated the estimated probability and impact of each identified risk.

6.65 The risk register included brief descriptions of response strategies for each identified risk. However, we found that the Agency had not developed formal and detailed contingency plans for any of the 17 ongoing risks that were identified as "high probability" or "high impact."

6.66 The government's risk management policy also requires that, to improve program management, program managers assess how they responded to risks that occurred. We were unable to find documented evidence that the measures the Agency took to respond to identified risks were assessed. As a result, only limited information on the effectiveness of these measures will be available to assist in planning for the 2011 Census.

6.67 Recommendation. Statistics Canada should formally assess and document the effectiveness of measures taken in response to risks that occurred during the 2006 Census.

Agency's response. Agreed. Statistics Canada will formally assess and document the effectiveness of measures taken in response to risks which materialized during the 2006 Census.

6.68 Recommendation. As it prepares for the 2011 Census, Statistics Canada should ensure that formal and detailed contingency plans for risks identified as "high probability" or "high impact" are prepared and kept up-to-date. For any contingency plans activated in 2011, Statistics Canada should assess and document the effectiveness of the measures taken, including their impact on data quality.

Agency's response. Agreed. Statistics Canada will prepare formal and detailed contingency plans for risks with a high probability/high impact designation in advance of the 2011 Census. Statistics Canada will assess and document the effectiveness of the measures taken for any contingencies activated for the 2011 Census.

During the conduct of the 2006 Census, Statistics Canada had a formal risk assessment and management process that proved effective in responding to risks. Assessing the effectiveness of the measures taken, including any impact on data quality is a standard practice and a necessary prerequisite at the end of one cycle and in planning for the next Census.

Planning for the risks of hiring temporary field staff was inadequate

6.69 A program of the magnitude of the Census of Population requires significant human resource planning and management. Primary responsibility for temporary field staffing of the 2006 Census rested with the Field Operations Project, in cooperation with Statistics Canada's three regional offices. The Agency anticipated hiring approximately 27,000 short-term field staff across the country to help conduct the Census.

6.70 As required by the Treasury Board Risk Management Policy, we expected Statistics Canada to have

• identified the risks to the 2006 Census posed by temporary staffing requirements;
• developed detailed contingency plans to address those risks;
• activated these contingency plans, when it encountered situations that threatened the success of the 2006 Census; and
• assessed any measures that it activated to better plan the 2011 Census.

6.71 As discussed earlier, Statistics Canada followed a structured process to identify risks. This included, as early as 2001, identifying risks related to recruiting and retaining the large numbers of temporary field staff required to conduct the Census. Due to local labour market conditions, these risks were particularly high in Alberta as well as in several major urban centres.

6.72 Even though field staffing was recognized, in the risk register, as a high-probability risk, we found that Statistics Canada had not prepared a formal, detailed contingency plan to respond to this risk.

6.73 This particular gap in planning is important for two reasons. First of all, temporary field staff are responsible for a broad range of critical data collection activities. Without sufficient field staff, particularly given the higher-than-anticipated non-response follow-up workload, meeting data collection timelines and quality targets could have been challenging. While Statistics Canada anticipated the need for 27,000 temporary field staff (based, in part, on the planned follow-up workload of 3.1 million dwellings), it employed no more than 21,000 at any one time. Furthermore, only 10,000 field staff were available during the peak follow-up period, two-thirds of whom did not work full-time.

6.74 Secondly, the schedule for data collection was very compressed. As noted previously, once a risk materializes, very little time can be devoted to developing optimal response strategies. To minimize the potential impact of the risk, contingency plans should be ready to implement immediately.

6.75 When it became clear that the 2006 Census was not attracting or retaining the planned numbers of field staff, the Agency (which, as previously noted, did not have a formal and detailed contingency plan) took a number of steps to address the situation, including

• recruiting continually,
• importing field staff to the hardest-hit regions, and
• switching some field staff from piece-rate to hourly pay sooner than planned.

6.76 The shortage of field staff contributed directly to the decision to delay the first release of Census data by one month. It also may have contributed to the lower response rates accepted in some collection units.

6.77 As previously noted, the government's risk management policy requires that, to improve program management, program managers assess their responses to risks that occurred. We were unable to identify documented evidence that the Agency had assessed the measures it took to respond to the recruitment and retention challenges. As a result, only limited information on the effectiveness of these measures will be available to assist in planning for the 2011 Census.

6.78 Recommendation. Statistics Canada should formally assess and document the effectiveness of measures it took to respond to recruitment and retention challenges that it faced during the 2006 Census—including the impact of those challenges on data quality. It should use the results of this formal assessment as it plans for the 2011 Census.

Agency's response. Agreed. Statistics Canada will comply with the specific recommendation to assess and document this process for the 2011 Census. Assessing the effectiveness of the measures taken, including any impact on data quality is a standard practice and a necessary prerequisite at the end of one cycle and in planning for the next Census.

6.79 Recommendation. For the 2011 Census, Statistics Canada should prepare formal, detailed contingency plans for identified risks that are related to the recruitment and retention of field staff and that are rated as "high probability" or "high impact."

Agency's response. Agreed. Statistics Canada will comply with the specific recommendation to prepare formal and detailed contingency plans for identified risks relating to the recruitment and retention of field staff that are rated as high likelihood and/or high impact, in advance of the 2011 Census.

Pay rates for field staff contributed to staffing challenges

6.80 Census management raised concerns about the adequacy of the pay rates of field staff, following the 2001 Census and the 2004 Census test. In 2006, the hourly rate for field staff was $11.88 (in 2001, the comparable rate was $10.50), and the Agency was concerned that sufficient field staff could not be recruited and retained at that pay rate.

6.81 The higher-than-anticipated volume of non-response follow-up work—which is acknowledged by the Agency to be more demanding than other tasks performed by these employees—added to the significant challenge Census managers faced in recruiting and retaining sufficient field staff. Initially, these follow-up activities were compensated on a piece-rate basis, which contributed to a low effective hourly wage, considering the effort required to obtain the cooperation of non-responding dwellings. This situation, which made the pay rates even less attractive to field staff, was addressed as planned by switching from a piece rate to an hourly rate in some locations.

6.82 Recommendation. Statistics Canada should review the rates and basis of pay for 2006 Census field staff to understand their impact on recruitment and retention, and to better plan for the 2011 Census.

Agency's response. Agreed. As part of the standard review process following each Census, Statistics Canada will comply with the specific recommendation to review the rates and basis of Census field staff pay utilized in 2006, in planning for the 2011 Census. The results of this review will be discussed with central agencies in preparation for the funding request for the implementation phase of the 2011 Census.

Unlinked questionnaires created additional work for field staff

6.83 We also examined another issue that arose during data collection. As noted earlier, Statistics Canada set up the Census Helpline; operators in three call centres answered enquiries about the Census and helped respondents complete their questionnaires.

6.84 Helpline operators relied on automated systems to find the dwelling identifier of individual callers on a master address list. If they could not find it, they assigned a temporary identifier to the questionnaire—creating what Statistics Canada calls unlinked questionnaires.

6.85 According to the Agency, the main cause of this situation was that the operators were unable to find the dwelling identifier in the master address list, because the system required exact matching. If the operators did not enter the address exactly as it appeared in the master address list, they could not get the dwelling identifier and link the questionnaire.

6.86 Approximately 400,000 unlinked questionnaires were created in the 2006 Census—instead of the anticipated 40,000. Statistics Canada maintains that many of the unlinked questionnaires were resolved quickly, and that approximately half of these questionnaires were resolved before they became part of the non-response follow-up workload. The remaining questionnaires, some of which were difficult to resolve, were referred to field staff, adding to their follow-up workload (which was already high).

6.87 Despite efforts that field staff and the head office made to resolve the unlinked questionnaires, approximately 80,000 questionnaires remained unlinked at the end of the collection phase. This number was eventually reduced to approximately 18,000. Statistics Canada has made a commitment to reduce the number of unlinked questionnaires for the 2011 Census.

Risks to privacy were successfully managed

6.88 It is critical to the success of the Census that the privacy of Canadians' personal data be protected. Their trust in Statistics Canada's processes and systems is essential for adequate response rates, a key determinant of data accuracy. We assessed the efforts the Agency made to address the risk to the privacy of respondents when it designed and conducted the 2006 Census of Population. We found that Statistics Canada had made considerable efforts to ensure that the privacy of Census data was protected and that it was successful in managing these risks.

6.89 After the 2001 Census, partly in response to concerns raised by the Office of the Privacy Commissioner (among others), Statistics Canada made changes to Census methodology for 2006. It provided mail and Internet options, so Canadians could respond privately, without the presence of a local Census employee. The Agency also followed up on incomplete returns by telephone from a central telephone facility, which provided some degree of privacy to respondents.

6.90 As required by the Treasury Board's Privacy Impact Assessment Policy, Statistics Canada conducted a Privacy Impact Assessment to determine if there were any privacy, confidentiality, or security issues associated with the 2006 Census. The Assessment identified a number of privacy concerns, assessed their risk, and suggested specific measures to address them. The Agency concluded that some of the remaining risks were negligible, and it was prepared to accept and manage the others. As was the case for risks discussed earlier, no formal and detailed contingency plans were developed for these remaining privacy risks.

6.91 In 2003, Statistics Canada hired two private sector firms to develop several new systems for the 2006 Census. This provoked some expressions of concern by Canadians and the Office of the Privacy Commissioner about the privacy of information on individual Canadians that is gathered during the Census. The Agency's response was to reduce the scope of these contracts, removing any involvement of these firms in data-processing operations.

6.92 Statistics Canada also contracted three independent security audits of the systems developed by these contractors, which confirmed the security of the systems. The Agency then asked an external task force to review the audit work. The task force concluded that the data gathered using the contractor-supplied systems would be secure and that it would be practically impossible for these contractors to access Census data.

6.93 Finally, the Office of the Privacy Commissioner conducted an on-site review of the processes for the 2006 Census and declared that it was satisfied that reasonable precautions had been taken to ensure the integrity and confidentiality of the data once it came into Statistic Canada's possession.

Conclusion

6.94 Statistics Canada satisfactorily applied its quality systems and practices to the management of the 2006 Census of Population to ensure that the data are fit for use. This included efforts to improve the quality of the information collected on hard-to-count population subgroups and to understand and meet the information needs of key government clients. During the data collection process, data accuracy was balanced with cost and timeliness.

6.95 However, Statistics Canada did not prepare a comprehensive document outlining how it planned to achieve the data quality required by the Quality Assurance Framework. It also did not create an integrated document laying out its plans to assess, after the Census, whether or not the required level of quality had been achieved. While Statistics Canada has assessed the performance of some elements of the 2006 Census program, it has not yet completed an integrated program review. Such a review would support both internal program management and external accountability.

6.96 Statistics Canada's approach to the 2006 Census did not fully satisfy the requirements of the government's policy on risk management. In particular, Statistics Canada failed to develop detailed formal plans and contingency plans to respond to the challenges it faced in hiring and retaining the required numbers of temporary field staff. This failure may have contributed to the delay of the first data release. In addition, the staffing shortages may have had an impact on data accuracy for some small geographic areas and sub-populations. Should any such impact have occurred, it would be evident only upon the release of these detailed data. The timing of this audit did not allow us to examine these data.

6.97 Statistics Canada made significant efforts to ensure that the privacy of the 2006 Census data was protected, and it was successful in managing these risks.

6.98 As Statistics Canada prepares for the 2011 Census, it has the opportunity to further ensure that the requirements of its quality assurance systems and practices are respected and that risks are adequately identified and planned for.

About the Audit

Objectives

The objectives of this audit were to determine whether

• Statistics Canada applied its established quality assurance systems and practices to the 2006 Census to ensure that the data are fit for use; and
• Statistics Canada's management of the 2006 Census complied with government policies on risk management.

Scope and approach

Specific issues addressed under the first objective included Statistics Canada's management of the coverage and response rates of certain hard-to-count groups (identified after the 2001 Census) and its efforts to meet the needs of key government clients.

Specific issues addressed under the second objective included Statistics Canada's management of the risks related to the recruitment and retention of field staff and to the privacy of respondents.

The scope of this audit included activities that Statistics Canada undertook between 2002 and Census Day (16 May 2006), to prepare for the 2006 Census, as well as the data collection and processing and the initial dissemination. The Census of Agriculture, which is carried out at the same time as the Census of Population, was not examined.

Our audit did not include the following:

• Direct assessment of the quality of the 2006 Census data.
• Assessment of information technology systems. In 2004, Statistics Canada conducted a large-scale test of the IT systems to be used in the 2006 Census, which confirmed the capability of the overall approach to meet the Agency's requirements. In addition, during our November 2006 audit of IT-enabled projects, we found that Statistics Canada's Census Online project met all of the Office's criteria for well-managed IT projects.
• Financial management. In January 2003, Consulting and Audit Canada completed an Independent Review of the Census of Population's Costs and Planning Assumptions. That document reached positive conclusions on the costing and financial management practices followed in the 2001 Census, and on the planning and costing assumptions, and the governance framework for the 2006 Census.
• Assessment of security-related personnel screening, accuracy and timeliness of pay processes, and management controls for quality of service delivered by Census help-line operators. Statistics Canada's Internal Audit Division told us that an audit of the 2006 Census—related to selected security, administrative, and quality practices—was to be carried out and reported to Agency senior management in fall 2007.

For this audit, our key methodology was to review a wide variety of files, documents, and reports pertaining to the Census. Staff at Statistics Canada helped to identify and provide us with relevant documents.

We interviewed officials at Statistics Canada, including managers responsible for the key Census projects, and staff of the three regional offices. A key objective of these interviews was to identify documentation relevant to this audit. We also interviewed representatives of federal departments and agencies, including those that contribute financially to the Census, and representatives of provincial and municipal governments.

Criteria

Our criteria for this audit were as follows:

• We expected Statistics Canada to demonstrate and document how the management of the 2006 Census balanced the six elements of quality against cost, client needs, and respondent burden.
• We expected Statistics Canada to have identified lessons learned from the 2001 Census and the 2004 Census test and to have considered them when it planned the 2006 Census.
• We expected Statistics Canada to have processes in place to monitor the 2006 Census as it was carried out, including mechanisms to identify challenges and take corrective action.
• We expected Statistics Canada to have identified risks to the current census in advance and taken steps to minimize them.
• We expected that Statistics Canada would, on encountering a situation that threatened the success of the 2006 Census, activate contingency plans and recovery measures, and assess these measures to better plan the 2011 Census.

Audit work completed

Audit work for this chapter was substantially completed on 31 May 2007.

Audit team

Assistant Auditor General: Mark Watters
Principal: Glenn Wheeler
Director: Colin Meredith

Joanne Butler
Doreen Deveen
Sophie Hebert
Anupheap Ngoun
Ruth Sullivan
Jacqueline Wickett

For information, please contact Communications at 613-995-3708 or 1-888-761-5953 (toll-free).

Appendix—List of recommendations

The following is a list of recommendations found in Chapter 6. The number in front of the recommendation indicates the paragraph where it appears in the chapter. The numbers in parentheses indicate the paragraphs where the topic is discussed.

Definition:

Non-response follow-up—The process used to identify and contact dwellings that have not returned completed Census questionnaires. (Return)