This Web page has been archived on the Web.

1983 Report of the Auditor General of Canada

Synopsis

16.1 Statistics Canada, operating since 1918 under the authority of successive Acts, is a centralized statistical agency and plays a dominant role in Canada's statistical system. The Agency faces a practically unlimited demand for more and better statistics. With limited resources, the demand to produce more data while maintaining the quality of existing data and not placing excessive burdens on businesses and individuals poses a continuing challenge.

16.2 When we began the audit, the Agency was just coming out of a troubled period. There had been four different Chief Statisticians in the previous 10 years, and the Agency had come under attack in both the press and the House of Commons for the quality of its products and its management practices. The Agency has now stabilized and there is a renewed sense of direction and purpose.

16.3 We directed our audit work toward management practices in five key areas which we considered essential to the functioning and effectiveness of Statistics Canada: credibility; meeting user needs; co-operation of respondents; confidentiality; and management controls.

16.4 Based on our review of the Agency's systems and procedures, we observed the following with respect to each audit area.

16.5 Credibility. We reviewed the measures taken by the Agency to protect its objectivity and to ensure the reliability of its statistics. With respect to its regularly published statistics, the Agency is taking appropriate steps to preserve its objectivity. With respect to its analytical studies, on which the Agency is placing greater emphasis, a strengthening of controls is needed to ensure that its objectivity is maintained. With respect to the reliability of its data, we found that the Agency had a sound international reputation. The knowledgeable users we contacted were generally satisfied with the reliability of the Agency's products. We were concerned, however, that even knowledgeable users may not always be aware of the limitations of the statistics they are using. The Agency needs to improve some of the underlying processes which contribute to the reliability of its statistics. For example, Statistics Canada lacks a formal quality assurance function; implementation of its policy on quality measurement and disclosure has been uneven; and it has yet to correct persistent underestimations in certain key economic surveys.

16.6 Meeting user needs. With respect to meeting user needs, we found that the Agency's product line is certainly relevant. We found no instances where the Agency was spending significant sums on products that had no identifiable users. However, improvements are needed in the way in which Statistics Canada identifies product gaps, sets objectives and ensures that there is a balance in its programs. The frequency and timeliness of products need to be reviewed periodically in a systematic fashion to ascertain if improvements can be made in certain cases or costs can be saved in others. The Agency's dissemination program should be strengthened through the development of a comprehensive marketing plan.

16.7 Co-operation of respondents. Over the past five years, the Agency has taken significant steps in reducing the response burden placed on individuals and businesses. Our interviews with representatives of key respondent groups showed that the business community was generally satisfied that Statistics Canada's demands for information were no longer excessive. In all subject matter areas we examined, program directors were aware of the need to minimize demands on respondents. We also found evidence that alternative means of obtaining data were being reviewed and used by directors.

16.8 Confidentiality. The Agency is taking satisfactory measures to protect the confidentiality of the responses it receives. Although no system is fail safe, there is a strong awareness at the Agency of the importance of maintaining privacy over individual data.

16.9 Management control. While improvements have been made, there remain substantial opportunities to strengthen management control and reduce costs. Managers are not systematically monitoring and reporting on the performance of their programs. As a result, the Agency lacks systematic information on which to evaluate and improve the key factors affecting the costs and effectiveness of its program: the required reliability of statistics; the frequency of production; the timeliness of release; and the best means of disseminating results.

16.10 Clerks responsible for processing statistical surveys are subject to operational peaks and valleys. Management was not measuring the efficiency of clerical staff in a meaningful way. Furthermore, in many cases, management was staffing divisions for peak-level production in order to meet deadlines without an adequate assessment of the costs of doing so. We found considerable variations in the efficiency of clerical resources at various points in the operating cycle. We tested 130 clerks in two divisions and found that they were operating at an average efficiency level of only 56 per cent throughout the year. We believe that productivity gains could be made through a systematic study of the Agency's current workflows and procedures.

16.11 Classification levels have a major impact on salaries. In 1980, the Agency commissioned the Audit Services Bureau of Supply and Services Canada to conduct an audit of its classification activity. The audit, subsequently accepted by Treasury Board, found that 44 per cent of the 100 positions selected for audit in the National Capital Region were misclassified (38 per cent over-classified and 6 per cent under-classified) and that 47 per cent of the official job descriptions were not consistent with the duties being performed. The audit report also indicated significant weaknesses in the Agency's monitoring and control of its classification system. Statistics Canada has not yet put in place mechanisms that would prevent these problems from continuing to occur. As well, the potential over-classification of a large number of clerical positions throughout the Agency remains unresolved.

16.12 Generally, in the area of management control, our observations underscore the need to strengthen corporate cohesiveness. There is a tendency for individual directors to operate independently without recognizing the need for trade-offs and co-ordination of effort. All the recent independent studies of the Agency as well as senior managers themselves have tended to confirm this. The processes for setting priorities, planning, allocating resources, and holding managers accountable all need further significant improvement.

16.13 In our opinion, the present autonomy given to individual directors should be supported by sufficient central direction and accountability to ensure that directors are working toward common goals. In this manner, the Agency can initiate corrective action across program and organizational lines.

16.14 There is evidence that present management is committed to making improvements. A project on operations integration, which is designed among other objectives to improve the efficiency of staff engaged in processing surveys, has recently commenced. A series of management committees has been formed to oversee activities and develop Agency policies in a number of areas. In addition, a comprehensive management plan has been prepared that seeks to make significant improvements to Agency practices in corporate planning; quality assurance; communications; human resource management; internal audit; program evaluation; and performance measurement. Because of difficulties in staffing, the plan has taken over two years to develop. It was formally approved for implementation in January 1983. Initial results from implementation of corrective action are promising. However, the target date of December 1984 for completion of the plan is ambitious.

16.15 Many of the areas needing improvement have been known to the Agency for several years, as outside reports and internal studies clearly show. Present management has made progress in re-establishing a sense of stability and direction to the Agency after its setbacks in the late 1970s. However, real and lasting changes to underlying management processes are still needed. They pose a significant challenge and will require continued commitment from the management team.

Background

16.17 Statistics Canada is made up of four fields - see Exhibit 16.1. The Economic Statistics Field and the Social Statistics Field are the two program areas.

Exhibit not available

16.18 The mandate of Statistics Canada is very broad, requiring a coverage of all relevant aspects of the structure and functioning of Canadian society. It has a responsibility to collect and disseminate information that affects the social and economic lives of practically all Canadians. It must do so in an unbiased manner, whether the information reflects well or badly on the government of the day.

16.19 The Agency faces a practically unlimited demand for more and better statistics. However, its resources are limited and have been growing only modestly in recent years - see Exhibit 16.2. The demand to produce more data, while maintaining the quality of existing data and not placing excessive burdens on businesses and individuals, poses a continuing challenge.

Exhibit not available

16.20 When we began the audit, the Agency was just coming out of a troubled period. There had been four different Chief Statisticians in the previous 10 years, and the Agency had come under attack in both the press and the House of Commons for the quality of its products and its management practices. In 1979, the President of the Treasury Board, who was also the Minister responsible for Statistics Canada at that time, commissioned two studies to look into the affairs of the Agency. A national consulting firm was asked to study the Agency's management practices, and a team, headed by Sir Claus Moser, a former head of the government statistical service of the United Kingdom, was asked to examine the Agency's methodology. Both these studies reported their findings in 1980. In addition, the Comptroller General had undertaken an examination of the management practices of the Agency as part of the government-wide initiative for improved management practices and controls (IMPAC) project. We carefully considered the findings of these studies during our audit.

Audit Scope

- Credibility: the Agency's objectivity and accuracy of its products.

- Meeting User Needs: the relevance of its products and the way in which they are disseminated.

- Co-operation of Respondents: control over demands for information directed to businesses and individuals.

- Confidentiality: control over the confidential responses received by the Agency.

- Management Control: practices affecting economy, efficiency, and measurement of program effectiveness.

16.23 We did not assess the quality of the Agency's statistical methodology. This was done by the Moser team in its 1980 study, and we took the results of that study into consideration. Our own examination was more concerned with controls exercised by management over the quality of statistical programs.

Observations and Recommendations

Credibility

16.25 Objectivity. We found the Agency to be impartial and objective with respect to the statistics it publishes regularly. It uses methodology based on accepted scientific concepts that is free from bias in the sense of arbitrarily favouring any particular group or ideology. We noted a strong sense of independence and professionalism in the staff we interviewed. Furthermore, the release dates for most widely-used statistics are published well in advance and adhered to. This avoids the appearance of bias in the timing with which data are released.

16.26 We found one development which will require adequate monitoring to protect the Agency's reputation for objectivity. Statistics Canada is placing renewed emphasis on analytic studies. Such studies are conducted in order to contribute to understanding economic and social conditions. In addition, through trying to analyse and draw conclusions from the data, the statistician becomes aware of where the data are insufficient and need to be expanded or modified. This provides opportunities for assuring the continued relevance of its statistical information base. Furthermore, statisticians who are directly involved in collecting the data have an intimate knowledge of their strengths and weaknesses.

16.27 The well-known risk in doing analysis, however, is that the conclusions reached are open to the criticism of being biased to a particular point of view. Some of the Agency's studies have, on occasion, been criticized for this. Criticism and debate can be healthy. However, the credibility of the Agency depends on its being viewed as impartial and unbiased.

16.28 The risk of bias in analytical work can be minimized through the exercise of proper professional judgement by senior managers and a process of peer review. Whereas the Agency does monitor the quality of its analytical work through informal peer review, its current procedures need strengthening. At present, there are no formal Agency guidelines on what constitutes appropriate peer review for analytical work. Such guidelines would address the need to co-ordinate the review independently of the author, select objective critics and document the process.

16.29 To protect its reputation for impartiality, the Agency should strengthen its requirements for review of analytical studies prior to release. Compliance with these requirements should be monitored.

Agency's comments: Statistics Canada has received very little criticism concerning the objectivity of its analytic studies. The Agency's basic position is that an undue emphasis on the risk of criticism would constrain the healthy and necessary evolution of its ability and mandate to serve the public interest through analysis of the data it collects. Thus, the Agency will continue to place appropriate emphasis on analytical work, and will seek to minimize the risks, as it has done in the past, through appropriate choice of topics for analysis, explicit separation of assumptions and interpretations from facts, and the use of scientifically appropriate techniques. The means of ensuring that these requirements are satisfied are peer review and professionally competent supervision. These have long been in place within the Agency in one form or another, but will be more explicitly articulated through the establishment of more formalized criteria and processes for the review of analytical studies.

16.30 Reliability. Complete reliability in statistics does not exist and is an unrealistic goal. A more practical goal is to try to make statistics sufficiently reliable for the major uses made of them. Statistics Canada faces a considerable challenge in attempting to do this within a fixed budget, in a reasonably timely fashion and without undue demands on its respondents.

16.31 The Agency has several activities that contribute to the reliability of its products. As an example, methodological research in the methodology branches of the two statistical fields has produced sophisticated sampling designs and techniques. There is ongoing research into alternative means of data collection. The Agency has developed a training program to turn new university graduates into skilled methodologists over a period of years. Program directors put significant effort into editing and imputation systems that are designed to check the integrity of data. The methodology used by directors is being documented; this allows it to be consistently applied.

16.32 The Moser team, in its 1980 study of the Agency's methodology, said, "As regards data quality we are impressed by much of what we have seen... . As regards timeliness, accuracy, and overall adequacy, most key series we have considered stand high in the international ranking". It referred in particular to the Labour Force Survey, the Systems of National Accounts and the Consumer Price Index.

16.33 Our contact with several knowledgeable users indicated that in general they were satisfied with the reliability of the statistics which the Agency is publishing. We were concerned, however, that even sophisticated users may not always be aware of the limitations of the data as described in more detail later in the chapter.

16.34 Despite its earlier comments, the Moser team remarked that it was concerned about the running down of the Agency's "methodological stock" and specifically referred to matters affecting data quality, such as the deterioration of benchmarks and sample frames from which the survey samples are drawn. The Agency has initiated certain corrective actions to strengthen its methodology function. However, improvements continue to be needed in the following areas.

16.35 Methodology. The Agency provides support to program managers through a pool of methodologists in each of the Economic and Social Statistics Fields. These methodologists are highly skilled and in some cases recognized internationally. However, trained methodologists are in short supply relative to the demand for their services so that there are programs which, according to their directors, have not received the methodological attention requested.

16.36 We found that there were no Agency standards or guidelines to provide direction to staff on acceptable practice in the design of methodology. Such guidelines would cover, for example, sampling design, the use of standard concepts, and requirements for designing and testing questionnaires. In 1980 the Agency produced a draft document entitled Survey Design Standards and Guidelines. Among other things, the standards sought to

"- establish practices for survey design activity in Statistics Canada and help ensure that all significant factors affecting the quality of data and their intended use are considered at the design stage; and

"- provide a set of standards to be observed and guidelines to be considered by all new and existing survey activity within Statistics Canada."

16.38 The 1980 Standards and Guidelines draft was not completed or released. We were informed that the matter is under renewed consideration by the Agency's Methodology and Standards Committee.

16.39 The Agency should develop methodology guidelines covering all aspects of statistical design to provide a framework for the design of sound methodology by its staff.

Agency's comments: In previous work on the development of methodology guidelines Statistics Canada found that, because of the great diversity of particular statistical programs, guidelines would have to be very general in nature. Their real usefulness, therefore, appeared to be questionable. The audit report does not suggest that the Agency's statistical work has suffered from the absence of a formal set of guidelines. The Agency ensures sound methodology through such means as building up in this area a staff of international caliber, developing a training program for new university graduates, documenting existing methodology, establishing methodology branches and conducting methodological research in each of the two subject-matter fields, and creating a variety of concrete mechanisms for the co-ordination and stimulation of methodology research and development work. Nevertheless, as part of the Agency's ongoing efforts to ensure that it is making the best possible use of "state-of-the-art" methodology, the practicality of the recommendation will continue to be kept under consideration.

16.40 Measurement and disclosure of quality. A recent Agency Strategic Overview stated that "an essential responsibility in statistical information development is to make every reasonable scientific effort to assess and articulate the degree of uncertainty associated with the information". It is essential that the user be provided with sufficient information to determine the appropriateness of the data for a specific use. Reliability itself is a multi-dimensional concept that is difficult to quantify. Although we recognize that quality measurements are costly and time-consuming, they are vital to the Agency's credibility and the usefulness of its products.

16.41 In 1977, the Agency issued a policy requiring that the quality of statistics be measured and disclosed. This policy has not been uniformly implemented or monitored. Although some publications, such as The Labour Force , include extensive information on survey methodology and the reliability of published results, in other instances the Agency is either not adequately measuring or not disclosing the quality of its statistics where it would be reasonable to do so.

16.42 For example, with respect to external trade statistics, the Agency provides some information on the source of its data but provides no information to users on the quality of the published statistics. While the Agency believes that its estimate of the total level of imports, for example, is reliable, it has known for some time that the classification of imports by commodity in its publications may contain significant errors because of deficiencies in the administrative records used to compile the data. Inadequate warning is provided to users on the quality of the data at the commodity level. The data are used extensively by the Department of Finance, the Anti-Dumping Tribunal, the Department of Industry, Trade and Commerce, and the Department of External Affairs in setting tariffs and in international negotiations.

16.43 The Agency should develop and disclose measures of quality for its statistics in accordance with its 1977 policy. Adherence to the policy should be monitored.

Agency's comments: The Agency is strongly committed to the intent of the 1977 policy and will continue its efforts to extend the policy's area of application beyond those major areas of the Agency's output which now satisfy its requirements. As worded, however, the recommendation could not be fully implemented, since "measures of quality" for many statistics - particularly those of a composite nature - are impossible to produce. The 1977 policy explicitly recognized this, and called for a full description of available information related to possible quality limitations, including, of course, quality measures when they are available.

16.44 One of the important factors affecting the cost of statistics is their required level of reliability. In response to a question by the Standing Committee on Miscellaneous Estimates, the Agency stated that "... in order to minimize both costs and the burden on individual respondents, the smallest sample size which will provide the required level of reliability is derived." We were not able to determine that in planning for or managing data series, management was setting explicit goals, where possible, for the required level of reliability of data which could be related to their main uses. For example, in the products we examined, we could find no explicit targets for acceptable response rates to questionnaires sent out, or acceptable precision in statistics relative to specific uses being made of them. This is unsatisfactory. Statistics could be produced which are not sufficiently precise and would therefore affect the Agency's credibility. Alternatively, statistics could be produced which are more precise than required and thus cost more than necessary.

16.45 The Agency should establish reasonable goals for the reliability of its statistics in relation to the major uses being made of them and the costs of gathering them. Achievement of the goals should be monitored and reported to senior management.

Agency's comments: The Agency recognizes and concurs with the underlying intent of the recommendation, but considers that this objective is best met by less mechanistic approaches.

The substantive difficulties are two-fold. First, any particular statistical series or compilation is typically used in many different contexts by many different users, and hence with different minimum expectations of reliability. The Agency does not, and cannot, know the full extent and complexity of these uses and, in any case, has no objective basis for assessing the relative importance of the various uses. A second, technical, problem derives from the numerous factors affecting the reliability of a particular statistical series, the inability to measure many of them, and hence, the impossibility of generating any single, overall measure of the reliability of the series.

The Agency assesses the reliability of its output in relation to user needs and costs through market feedback based on regular consultations with its major users, including systematic program evaluations. In addition, the Agency on an ongoing basis monitors its own product quality and operating costs, using a variety of measures appropriate to the circumstances (response rates, estimates of variance, edit failure rates, etc.) and feeds the results of these assessments into the ongoing production process. Instances of statistics being "over-engineered" from the quality standpoint, and hence unnecessarily consuming resources, are indeed rare (no current examples were identified during the audit review), and the Agency in the past has discontinued any statistical programs whose benefits ceased to outweigh their costs.

16.46 Quality assurance. As stated earlier, individual program managers are making extensive efforts to ensure the quality of the data they are publishing. However, at present, the Agency does not have a quality assurance function, independent of the line manager, designed to periodically assess and report on the adequacy of methodology and other measures taken by program directors to ensure that the quality of statistics is maintained. The current environment of limited budgets, autonomy given to program directors, absence of methodology guidelines and shortage of trained methodologists makes the need for a quality assurance function all the more essential. Management has recently recognized this need. Its management plan calls for development of a quality assurance process by the first quarter of 1984.

16.47 Management should ensure that its planned quality assurance process is accompanied by an independent monitoring function which can inform senior management of its findings.

Agency's comments: The Agency agrees with this recommendation. The formal establishment of a quality assurance process, scheduled to take place in 1984 in the context of the Agency's management plan, is planned to be organized so as to ensure its independence from production activities and the related accountability structure.

16.48 With respect to quality assurance, one of the key advantages in operating a central statistical agency such as Statistics Canada is the ability to integrate and compare statistics from a wide range of programs. By doing this, the Agency can become aware of inconsistencies in the results of various programs and take corrective action if necessary. The System of National Accounts is one of the prime tools for integration used by Statistics Canada. In preparing the National Accounts, the Agency draws together results from its economic surveys. Based on this information, underestimates have been identified by the Agency in the results from two of its major surveys; namely, retail trade and labour income.

16.49 The survey of retail trade provides an estimate of the value of retail trade through a survey of retail establishments. However, by comparing the results from this survey with other economic indicators, the division which prepares the National Accounts has concluded that the survey of retail trade is underestimating the value of retail trade. To compensate for this, the Agency has been making upward adjustments to the results from the retail trade survey before making use of them in preparing the National Accounts. The initial upward adjustment for 1971 was 2.2 per cent but, since 1975, the annual adjustment has been over 12 per cent. For 1982, the adjustment was 13.5 per cent or $13 billion.

16.50 The division which conducts the retail trade survey has known for at least seven years that its monthly survey is underestimating retail trade, but it questions the magnitude of the understatement involved. The opinion of this division is that the underestimate is much lower than 13 per cent. This difference of opinion has not been resolved. Furthermore, the Agency does not disclose the existence of an underestimate to readers of its retail trade publication.

16.51 One contributing problem is that the Agency has not conducted a census of retailers since 1971. A census, or full count of retailers, would provide a benchmark against which to measure the accuracy of survey results in subsequent periods. The Agency was refused funding for a new census of retailers in both 1976 and 1981. After several years of going without an adequate benchmark, the Agency is now assessing alternative measures to improve the quality of its retail trade survey. These measures involve greater use of income tax records. It is too early to tell if the new measures will correct the understatement problem in the survey.

16.52 A second factor which may be contributing to the underestimation of retail trade is the incompleteness or inaccuracy of the lists of businesses, or "sample frame", from which the samples for the retail trade survey are drawn. If the business list is inaccurate or incomplete, this affects the reliability of surveys based on it. The retail trade survey uses the Agency's Business Register as its sample frame. The Agency recognizes that the Business Register, which is based on taxation records, is incomplete because by design it excludes businesses without employees and is subject to delays in updating. As a result, several divisions have supplemented their survey lists derived from the Business Register with information from other sources.

16.53 The accuracy of the Business Register has not been evaluated, although the Agency plans to do so. The accuracy of the separate divisional lists is unknown. Statistics Canada spends $3 million a year on the Business Register. The cost of the separate divisional lists is unknown. Given the importance of accurate sampling frames to the quality of all business surveys conducted by the Agency, an assessment and rationalization of the Business Register and associated divisional lists are essential.

16.54 With respect to labour income, the employer-based survey of employment used until early 1983 resulted in initial estimates of labour income that were $2 billion or 1 per cent lower on an annual basis than that used in calculating GNP. While this difference is not large in percentage terms, it is significant in terms of the sensitivity of labour income to many economic forecasters, including the Department of Finance. The Agency has known for several years that the results from its employer-based survey may be understated. There were user complaints as early as 1974. In 1976, the Agency undertook to solve the problem by designing a new survey which was to be operational in 1981 but is only now being introduced. It is too early to assess whether the new survey will be successful.

16.55 The Agency should resolve the existing discrepancies between the National Accounts and its two major underlying surveys. Problems identified through the integrative mechanism of the system of National Accounts should be resolved in a timely manner.

Agency's comments: As noted in the report, the Agency has already implemented major programs to address the problem of underestimation in the retail trade and labour income survey data.

With respect to the retail trade estimates, it has long been evident that corrective action depended largely upon the availability of an appropriate benchmark - which indeed is one of the recommendations of the present report. However, the Agency's attempts to gain resources for a comprehensive Census of Merchandising and Services, which would have provided such a benchmark, were unsuccessful in both 1976 and 1981. Consequently, the Agency has undertaken for the 1982 reference year a stop-gap measure consisting of a survey of the retail trade sectors covering all large retailers, supplemented by tax information for the smaller retailers. This approach, albeit less effective than a comprehensive census, will provide timely benchmark levels, as well as more up-to-date business/personal expenditure ratios to assist in the appropriate allocation of expenditures to the consumer sector of the National Accounts.

With respect to current labour income estimates, corrective steps have already been initiated through a redesigned monthly establishment-based survey of Employment, Payrolls and Hours, from which the income estimates derive. A completely new survey is now fully operational but as the report indicates, it is too early to assess its results.

The Agency has no difficulty in endorsing the second part of the recommendation. To the extent that problems identified through the integrative mechanism of the National Accounts are not "resolved in a timely manner", it is not an indication of tardiness on the part of the responsible managers, but rather of the major dimensions of the problems, and of the difficulties of resolving them within a short timespan.

16.56 Appropriate steps should be taken by the Agency to ensure that it is able to maintain sound and economical sampling frames for economic statistics.

Agency's comments: The Agency is taking appropriate steps to maintain sound and economical sampling frames for economic statistics, and will continue to pursue these vigorously. Experience in Canada and elsewhere has demonstrated that Business Registers based on major administrative systems (such as the annual Revenue Canada lists of employers) provide the soundest and most economical approach to the development of sampling frames for economic statistics, and such an approach is used in Canada to provide the foundation of the central Business Register. This taxation-based source, while the best available, is subject to unavoidable problems: it does not cover on a current basis businesses having no employees, it is subject to changes resulting from new administrative arrangements, and is subject to a built-in lag in reporting business "births" and "deaths". A number of separate divisional registers, all of which are derivations of the central register, are purposely maintained for one of two reasons: to overcome, for a specific application, the known limitations of the central Business Register, or to benefit from the economies of processing smaller lists.

16.57 The Agency should take appropriate action to protect more fully the quality of its merchandising and services statistics by ensuring the availability of adequate benchmark data.

Agency's comments: The recommendation appropriately identifies merchandising and services statistics as a specially critical area of concern within the general requirement for periodically calibrating the current statistical program through more comprehensive benchmark estimates. As will be clear from the comment on an earlier recommendation, the Agency has repeatedly sought the funds required to conduct a Census of Merchandising and Services, but without success. Given the urgency of the problem, the Agency is taking alternative steps which require fewer resources but will provide less comprehensive information.

Meeting User Needs

16.59 Relevance. The Agency's mandate is very broad. Essentially, it is "to provide information needed to develop social and economic policies and programs and to understand Canadian society". An added dimension is that Statistics Canada is a national agency with responsibility for collaborating with departments and agencies in all levels of government to best meet the statistical needs of users. At any point, the mix of Statistics Canada's products and services reflects its interpretation of that mandate.

16.60 Our work showed that the Agency's product line was certainly relevant. The product line of over 500 statistical programs has evolved over a long period, and there is evidence that it has been adjusted at the margin in response to user requirements. Most of the Agency's budget goes to producing statistics required by law or well established government needs. Examples are the Census of Population, the Consumer Price Index, the System of National Accounts, and unemployment statistics. We found no instances where Statistics Canada was spending significant sums on products which did not have identifiable users.

16.61 The Agency's core users are extensively consulted. We noted a strong commitment on the part of the Chief Statistician to be responsive to government needs. The Agency's relatively recent access to Cabinet documents and its representation at meetings of three government envelopes - Economic and Regional Development, Social Development, and Services to Government - facilitate frequent senior level contact with other federal government departments. This improves the Agency's ability to anticipate the information requirements of the federal government.

16.62 Our interviews with several key users, including those in federal and provincial governments, showed that they found the Agency's products on the whole both satisfactory and useful, though gaps in coverage both of subject matter and of detail do exist.

16.63 In an environment of infinite demand, gaps in coverage will always exist. On the other hand, almost any information produced is likely to have some application for some users. The key question is this: is the Agency using its resources in the most effective manner? The answer rests with how the Agency sets its priorities, how it evaluates options for service, and the extent to which it critically evaluates its product line. Steps which the Agency can take to improve the way it currently sets priorities are:

- Collecting user feedback in an organized manner, evaluating the feedback, and ranking demands. At present, this is done by program directors and senior line managers as part of their ongoing management responsibilities. While the assembled information is used in program planning and priority-setting within the two major Fields, there has, until recently, been little co-ordinated Agency-wide assessment of options. This could result in Fields pursuing incremental priorities in their own areas without adequate reference to the balance in the Agency's overall program.

- Achieving overall balance in serving various users. Careful attention must be given by senior management to ensuring an appropriate balance between serving government needs at all levels versus private sector needs. Broad priorities must also be established between different subject matters, for example, between investing in more business-related statistics versus more health-related statistics. Finally, management needs to communicate clear priorities in the area of improving the quality and usefulness of existing programs versus investing in new programs.

At present, there are few explicit guidelines in these areas, and decisions in many cases do not appear to be based on explicit objectives or a long-term plan.

- Requiring directors to assess and report the balance in the extent of coverage, timeliness, accuracy, and frequency within specific programs. These factors are the key elements in determining the costs and the usefulness of a statistical program. Generally, the more comprehensive and detailed the statistics, the more they cost and the longer they take to produce. Similarly, there is often a trade-off between accuracy and timeliness in statistics.

At present, there is no requirement for directors to assess their programs on an ongoing basis. We found that directors who were assessing their programs were doing so with varying degrees of thoroughness. The result is that there is no organized reporting to senior management to inform them of potential imbalances in programs.

- Establishing a National Statistics Council. To help in setting priorities, both the Moser and the national consulting firm's studies, referred to earlier, recommended that a National Statistical Council be established. Such a council, composed of high level external advisers representing a variety of interests and users, could advise the Chief Statistician on a wide range of matters including priority setting. A council of this type exists in Australia, and our review of its reports showed that such a council can be of great help to a Chief Statistician. We understand that the formation of a council has been considered by the Agency.

16.65 The Agency should continue its recent initiatives to improve the decision-making processes involved in setting goals and achieving balance within and among its individual programs.

Agency's comments: The Agency will indeed continue to follow the intent of this recommendation, through a variety of means. Nevertheless, it bears emphasizing that an explicit set of priority rankings is neither practicable nor productive because the complexity and non-commensurability of the Agency's programs make it impossible to establish operationally useful criteria. Furthermore, as noted by the audit report, an overwhelming proportion of the Agency's products are fixed by statutory requirements or respond to well-established and largely unchanging governmental and other public needs. The Agency's planning process, therefore, operates within a relatively narrow margin of discretion.

16.66 Statistics Canada should continue to explore the formation of a National Statistics Council and reach a decision about its feasibility in Canada.

Agency's comments: National Statistical Councils, in one form or another, exist in several countries; the report cited the Australian experience, for example. On the other hand, several statistically developed countries, among which are the United Kingdom and the United States do not have them. The issue has been considered carefully in Canada on a number of occasions during the past decade, but to date the potential advantages of such a body have not been considered sufficient to offset the difficulties associated with addressing competently the very great diversity of Statistics Canada's program activities under the umbrella of by single body of part-time external experts. The Agency has long had an extensive network of consultative arrangements with both government and industry which have been quite successful in doing much of the work that might be expected of a formalized council. The Agency has elected for the time being to continue with these arrangements and extend them where appropriate on discrete and definable areas of statistics (e.g., price measurement).

16.67 Frequency and timeliness. Frequency refers to how often the statistics are produced (e.g., monthly or quarterly) and timeliness refers to how soon after the end of the period being measured they are released. Both these factors significantly affect costs.

16.68 We found no explicit evidence that the Agency is making an organized assessment of the frequency with which it delivers its products, although we were advised that this was given some attention during the past year's planning exercise. A review of the Agency's publications shows that more than 90 per cent are produced on an annual or more frequent basis. Opportunities may well exist to save costs through a systematic examination of the need for this level of frequency. For example, is it necessary to produce restaurant, caterer and tavern statistics monthly, or statistics on honey production semi-annually? Could savings be made in a cost-effective manner by publishing these and other statistics less frequently?

16.69 With respect to the timeliness with which statistics are released, we observed that time schedules are established and met and that users seem generally satisfied. The Agency has made efforts to make data available to users within a reasonable period of time, although in many cases the data would be more useful if they were available faster. Certain key statistics, such as the Consumer Price Index and the unemployment rate, are published very promptly. For example, unemployment rates based on a survey of 56,000 households each month are processed and released no later than the eleventh day of the following month. Others, such as the travel accommodation publication, are released as much as one year or more after their reference period. There is a danger that, in some cases, such late data may be irrelevant for many users.

16.70 On the other hand, pressure from users tends to be only one way - for more timely data. The danger exists that, without controls, certain programs will respond to this pressure out of proportion to the cost to the Agency. Shortening production periods in order to accelerate data release has implications for staffing requirements and efficiency levels. We found little evidence that senior management was systematically evaluating and challenging the timeliness of its products in relation to costs and the needs of various users.

16.71 The frequency and timeliness with which statistics are produced should be regularly reviewed by management and assessed in relation to user needs and costs.

Agency's comments: The formal process of review is being made more systematic through the mandate of the Program Evaluation function and through management review carried out in conjunction with planning done within the context of a new Operational Planning Framework. Program evaluations, as well as ongoing management review, have focused and will continue to focus on frequency and timeliness, since these are two of the fundamental aspects of virtually all statistical programs.

16.72 Dissemination. One of Statistics Canada's key responsibilities is to inform Canadians of its findings. The Agency carries out its dissemination responsibility through a variety of means, such as printed publications; a computerized data bank called CANSIM; a telephone user advisory service; seminars and workshops for interested users; and press releases to the media.

16.73 Sophisticated users of the Agency's products seem well served. These include government departments and large private corporations. The Agency's program directors keep in close touch with these users and try to satisfy their needs.

16.74 However, the Agency also has a responsibility to make available its broad range of products - prepared at taxpayers' expense - to a wider audience of users in cost-effective modes. To carry out this responsibility effectively, the Agency needs an overall dissemination strategy. This is not in place at the present time. Such a strategy would include:

- objectives and priorities of a dissemination program;

- the relative roles and responsibilities for dissemination;

- the most effective means of meeting the information needs of various client groups;

- alternative modes of dissemination, such as the optimal mix of hardcopy and electronic products; and

- pricing of services in relation to costs and revenues.

16.76 Most directors choose to disseminate information through printed publications. These publications represent only a fraction of the information available and, although informative, are often difficult for users to work with. A study recently conducted by the Agency concluded that insufficient attention was being given by program directors to the costs of their publications and alternatives to printed publications such as microfiche and computer tapes.

16.77 In the area of computerized dissemination, the Agency's CANSIM represents a significant achievement. CANSIM is a computerized data bank of statistical information which can be accessed on-line through a terminal on the user's premises. Recently, the Agency has developed a new public information service, called TELICHART, which enables Telidon terminals to access CANSIM data and display statistics in colour graphics.

16.78 Although CANSIM has been in operation for more than 10 years as the Agency's main computerized dissemination vehicle, it has not been evaluated for its effectiveness versus costs. To date, the placement of data on the CANSIM data bank has been slow to evolve, despite a 1974 Agency policy requiring directors to place as much data as possible on CANSIM. The present contents of the CANSIM data bank have not been evaluated to determine if they represent an optimal mix and quantity of data across a range of subjects.

16.79 The Agency should develop and maintain a comprehensive marketing strategy which addresses the objectives and priorities of a dissemination program, roles and responsibilities for dissemination, the most effective modes of dissemination, and pricing of services in relation to costs and revenues.

Agency's comments: The Agency agrees with the intent of this recommendation and has made substantial progress toward its full implementation.

The report correctly observes that "sophisticated users of the Agency's products seem well served", and thus this recommendation is mainly addressed to the wider audience of less sophisticated users which the Agency also has an obligation to serve.

The Agency has been pursuing this objective for some time; for instance, the centralized marketing support staffs have been regrouped into market-oriented segments; much more active use is now being made of news releases; and the progressive dissemination of 1981 Census data is being more closely tied in to the needs of communities and interest groups.

Co-operation of Respondents

16.81 A major effort in reducing response burden came in 1978 when, in co-operation with the Office of Paper Burden, the Agency identified opportunities for substantial reductions. For example, the Agency estimates that, as part of the government paper burden reduction exercise, it reduced the number of questionnaires mailed by 14 per cent and the time required to fill in forms by 22 per cent. This was achieved through either eliminating surveys or reducing sample sizes.

16.82 In all subject matter areas we examined, program directors were aware of the need to minimize demands on respondents. We also found evidence that alternative means of obtaining data were being reviewed by directors. There is evidence of continuing progress in reducing response burden. For example:

- The Agency has established two research groups - Tax Record Access and Administrative Data Development - to explore the possibilities of using administrative data, such as tax returns, gathered by other organizations as an alternative to collecting similar data by surveys.

- Changes to the Corporations and Labour Unions Returns Act will result in tax records being substituted for surveys of small business. This will enable the Agency to drop 96,000 businesses from its list of respondents. This is expected to take effect this year.

- The Agriculture Division, among others, has consolidated and eliminated surveys and substituted data from other sources (e.g., Milk Marketing Board) for surveys. The Agriculture Division is also using satellites to determine potato crop yields. This innovative data collection method is supplemented by visits to a relatively small number of farms to verify the satellite information. Research into expanding this method to other crops is under way.

16.84 The Agency should establish and monitor indicators of response burden in a manner which will allow senior management to periodically assess the impact of its data gathering activities.

Agency's comments: The Agency agrees with this recommendation. It has in the past prepared measures of response burden, the results of some of which are noted in the report. Such calculations will be repeated periodically.

As the report clearly indicates, the business community presently appears to be satisfied that Statistics Canada's requirements for information are no longer imposing an excessive burden. This is almost wholly the consequence of the Agency's own efforts during the late 1970s. Nevertheless, Statistics Canada regards the minimization of response burden as a continuing challenge, and will continue its efforts to reduce respondent burden, particularly through the further exploitation of administrative records as a substitute for the direct gathering of data.

Confidentiality

Management Control

- policy setting;

- planning and resource allocation;

- program evaluation;

- internal audit;

- accounting control;

- management of clerical resources;

- classification levels; and

- other.

16.88 The Agency should establish and maintain a comprehensive manual of policies.

Agency's comments: The Agency agrees with this recommendation. During the past two and one-half years, senior management in the Agency has been addressing fundamental aspects of policy setting. In November 1982, after a 15-month period of pilot trials, a network of some 15 management committees was established, covering responsibilities for each of the Agency-wide functions such as Personnel, Planning and Finance, Operations, Systems and Information Processing, etc., and including a Policy Committee chaired by the Chief Statistician. Each committee is authorized to establish policies and practices in its area of concern, and is required to disseminate its policy decisions. With this process in active operation, it is now practical and useful to start consolidating these decisions into a comprehensive manual.

16.89 Planning and resource allocation. Planning in a statistical agency is an intrinsically complex operation involving judgements about relative priorities in many different areas. Nonetheless, there are deficiencies in the Agency's planning and resource allocation mechanisms. Management has prepared plans to address the existing deficiencies in:

- communication of the planning process;

- integration and consistency between strategic overviews and long and short-term plans;

- integration of separate planning processes for areas such as human resources, EDP and marketing;

- accountability for use of resources (budget variances in dollars are reported but not analysed as to how they have affected the Agency's goals);

- use of the Management Information System; and

- adequacy of monitoring the performance of programs as well as the criteria and procedures to guide such measurements.

16.91 Program evaluation. The Program Evaluation Division was officially established in 1973, but was used until 1980 for a variety of tasks not related to program evaluation.

16.92 Program evaluation activity, as it is currently understood in the government, began in the Agency in 1980 with a planned cycle of five pilot evaluations. A policy on evaluation was developed in 1981 and a Director of Program Evaluation was appointed in 1982. With the appointment of a Director, significant attention has been given to setting up the Division and to putting in place the necessary procedures and plans to carry out corporate program evaluations. The development work in this area is now largely complete.

16.93 As of March 1983, the Division had completed four of five planned pilot evaluations. The pilot evaluations were purposely limited in scope in order to gain experience and establish credibility. Such a strategy was a reasonable and appropriate way to proceed. The pilot evaluations conducted to date have not uniformly covered a full range of potential issues, such as consideration of methodology, appropriateness of dissemination procedures, and the extent to which user needs were met in the program. However, the experience gained through the pilot studies has now given the unit the capability of conducting more comprehensive evaluations if they are directed to do so by senior management. The Agency has indicated that future evaluations will cover a comprehensive range of evaluation issues.

16.94 In its evaluation policy, the Agency has indicated that it intends to evaluate all its programs within a five-year cycle. In light of this, and management's high expectations with respect to the quality and scope of evaluations, the level of resources currently available to the Division will need to be closely monitored if these objectives are to be met.

16.95 Management should closely monitor the level of resources assigned to the program evaluation function in light of the expectations placed on it.

Agency's comments: The Agency agrees with this recommendation. Great importance is attached to the effectiveness of the program evaluation function, the work of which is under the continual scrutiny of the Program Evaluation Committee, chaired by the Chief Statistician. The resources presently allocated to the program evaluation function are based on experience obtained in five pilot studies and on the comprehensive plan which has been developed during the past year. However, should it be necessary, the resources will be adjusted in the light of further experiences.

16.96 Internal audit. We reported major deficiencies in the internal audit function in 1976 and in 1978 to the Chief Statistician. There has been no material improvement in the situation since then. Management has recently appointed a new Director of Internal Audit and has a project in its management plan to build a stronger internal auditing function.

16.97 Accounting control. We found that overall accounting control at the Agency was satisfactory.

16.98 Management of clerical resources. The nature of statistical programs is such that there are peaks and valleys in the processing cycle. When questionnaires are returned on a monthly survey, for example, there is a peak workload during which the questionnaires have to be processed and the monthly statistics prepared. During the remainder of the month, activity diminishes substantially. Another important factor affecting the peaking of workloads is the demand placed on the Agency for timely statistics.

16.99 In view of the characteristics of much of the clerical work at the Agency - its repetitive nature as well as the potential problem of having staff alternately very busy or idle - we examined the Agency's management of the efficiency of its clerical staff.

16.100 We found that clerical staff in the two major program fields were generally organized to serve individual statistical programs instead of being pooled to serve more than one program. We also found that program directors, concerned with meeting deadlines, appeared to be staffing their programs to meet peak level production without assessing the costs of having staff idle during low volume periods. In addition, program managers were not systematically monitoring the efficiency of their clerical staff. In the absence of management information on the efficiency of clerical staff, we conducted our own audit tests in this area.

16.101 Our tests covered approximately 130 clerical employees in two divisions of the Agency. One division was recommended to us by the Agency as representing a division with a high concentration of clerical staff processing a high volume of work. The other division was chosen by us because it was in another field and also had a reasonably high concentration of clerical staff. The objective of our test was to ascertain if significant opportunities existed to improve clerical efficiency.

16.102 We examined the nature of the work performed by the clerks and ascertained that it was of a routine nature such as sorting documents, coding, and keying in data. We prepared engineered work standards for the tasks performed using a widely accepted work measurement technique which allows for fatigue, coffee breaks and other customary non-productive time. Finally, we examined time records and production volumes over the full year January 1982 to December 1982. Our tests were designed to measure the average efficiency of clerks producing output which met quality levels acceptable to management.

16.103 Our observations showed the effect of the peak and valley conditions we mentioned earlier. When work was available toward the end of the week, month or other cycle, we observed that clerks achieved acceptable efficiency levels. At low points in the cycle, however, clerks were typically operating at much lower levels of efficiency.

16.104 On the average, throughout the year, we found that the clerks we tested were operating at only 56 per cent efficiency. This represents a considerable underutilization of clerical staff in the units we tested. We did not measure all the clerks in the Agency. In our 1978 annual Report, we cited 80 per cent as being a minimum acceptable level of clerical performance for comparable job types and working environments.

16.105 There is evidence that the Agency has recognized the need to improve clerical efficiency and has begun to explore the possibility of integrating its clerical operations to save costs. A Director General in charge of Operations Integration and Regionalization, supported by an Operations Committee, has recently been appointed for this purpose.

16.106 When conducting our measurement of efficiency, we accepted the existing methods of processing work as a given, and did not attempt to define a more efficient set of procedures. We did observe, however, that there were inefficiencies in certain cases in the way in which the workflow was designed. For example, we noted in one unit that work stations were laid out so that documents had to be transported between floors several times before processing could be completed. We also noted that clerks in some units experienced idle time in the mornings while waiting for work to arrive. These situations could be improved by better workflow design; for example, by changing the routing or storage of documents to avoid unnecessary transportation, and by creating a small daily overflow of work so that clerks can be kept continuously occupied instead of waiting for work to arrive each morning.

16.107 At present, individual directors are responsible for designing their own workflows and clerical procedures without the benefit of advice from specialists in workflow design. Many government departments involved in large scale clerical processing, similar to that of Statistics Canada, have found it useful to employ specialists in workflow design and clerical efficiency to improve the productivity of their staff.

16.108 The Agency should examine its workflow and procedures using appropriate expertise to improve the productivity of its clerical operations.

Agency's comments: Increasingly in the future, the newly established Operations Integration and Regionalization Branch will be playing an important role in encouraging, facilitating, and co-ordinating undertakings designed to increase overall productivity, including clerical operations. Nevertheless, it remains a basic responsibility of divisional management to keep its workflow and operational procedures under review in order to identify opportunities for improving productivity. Significant improvements in the quality, quantity and timeliness of outputs relative to resource inputs have been achieved in the recent past, and emerging technological developments in electronic data processing and office automation will continue to provide fresh opportunities for further improvements. In the planning and implementation of such improvements, divisional management will supplement its own specialist skills with those from the various centralized functional support services within the Agency, and from outside sources when necessary.

16.109 Opportunities to improve the present level of clerical efficiency should be actively pursued. The practice of staffing for peak production levels should be examined and, where appropriate, more cost-effective ways of meeting production demands should be explored.

Agency's comments: The Operations Integration and Regionalization Branch will lead and co-ordinate the Agency's efforts to improve its operational efficiency by means which encompass, but go far beyond, attempts to obtain increased productivity from clerical staff.

The reported level of clerical efficiency in the portions of the two Divisions examined, at 56 per cent, was roughly comparable to the average level of 60.8 per cent found in tests conducted in 10 departments in 1979 by the Auditor General. However, the tests of clerical efficiency conducted during the audit were limited in scope, in that they deliberately did not attempt to deal with overall operational efficiency, and were not intended to be a representative sample of all of the Agency's clerical operations. The Agency therefore does not believe that the test results form an appropriate basis for an overall assessment of the efficiency of its clerical staff. The Agency also does not believe that the "practice of staffing for peak levels of production" is as widespread or inefficient as the report implies, nor does it believe that unwarranted trade-offs between timeliness and costs are being made by program managers.

16.110 The Agency should establish clerical performance measurement policies and, where appropriate, implement performance measurement systems.

Agency's comments: The Agency fully supports the intent of the recommendation, although it does not agree with the observations on which the recommendation is based. The recommendation regarding performance measurement systems is being addressed through the Agency's Management Practices Action Plan.

16.111 Classification levels. Classification levels have a major impact on salaries. In 1980, the Agency commissioned the Audit Services Bureau of Supply and Services Canada to conduct an audit of its classification activity. The audit, subsequently accepted by Treasury Board, found that 44 per cent of the 100 positions selected for audit in the National Capital Region were misclassified (38 per cent over-classified and 6 per cent under-classified) and that 47 per cent of the official job descriptions were not consistent with the duties being performed. The audit report also indicated significant weaknesses in the Agency's monitoring and control of its classification system. The Agency has not yet put in place mechanisms that would prevent these problems from occurring.

16.112 Of the specific positions which were identified as misclassified in the audit sample, most have been acted upon. However, the few that have not been resolved involve over-classifications of clerical statistical positions. Any change to the classification levels of these positions would have an impact on what the Agency estimates to be 1,200 similar positions throughout the organization. With the agreement of the Treasury Board Secretariat, corrective action on these positions had been deferred, subject to a departmental study into the classification standard which was to have been completed by July 1982. The project is not yet complete, and therefore the classification status of a large number of clerical positions throughout the Agency remains unresolved.

16.113 The Agency should complete the corrective action on potential over- classifications identified by the 1980 audit since the positions not yet corrected affect a large number of similar positions throughout the Agency.

Agency's comments: As noted in the report, the Agency has already acted upon most of the positions identified as misclassified in the 1980 audit. The situation arising from the potential misclassification of some of the Agency's clerical positions will be dealt with as soon as the Agency and the Treasury Board Secretariat have agreed upon an appropriate course of action.

16.114 The Agency should establish the control systems necessary to ensure adequate management of the classification system.

Agency's comments: The Agency has developed and is in the process of implementing appropriate procedures to monitor the classification system.

16.115 Other. In the area of management control, our observations underscore the need to strengthen corporate cohesiveness. There is a tendency for individual directors to operate independently without recognizing the need for trade-offs and co-ordination of effort. All the recent independent studies of the Agency, and senior managers themselves, have tended to confirm this.

16.116 In our opinion, the present autonomy given to individual directors should be supported by sufficient central direction to ensure that directors are working toward common goals. In this manner, the Agency can initiate corrective action across program and organizational lines. Without the ability to do this, the Agency's management might well continue to be unable to correct many long-standing deficiencies. For example, the process for setting priorities, planning, allocating resources and holding managers accountable all need significant improvement.

16.117 There is evidence that present management is committed to making improvements. A number of management initiatives in key areas have been referred to throughout this report. As indicated earlier, a comprehensive management plan has been prepared which seeks to make significant improvements to Agency practices in corporate planning; quality assurance; communications; human resources management; internal audit; program evaluation; and performance measurement. The individual projects referred to in our observations have target completion dates ranging from the third quarter of 1983 to the fourth quarter of 1984 with the full plan being scheduled for completion by December 1984. Because of difficulties in staffing, the plan has taken over two years to develop. It was formally approved for implementation in January 1983. Initial results from implementation of corrective action are promising. However, the target date of December 1984 for completion of the plan is ambitious.