This Web page has been archived on the Web.

1983 Report of the Auditor General of Canada

Synopsis

5.1 The Department of Finance prepares regular forecasts of Canadian economic activity. These forecasts are a basis for policy advice to ministers and are key inputs into federal budgets. In addition, the forecasts of gross national product and other key variables are used in calculating interim payments to the provinces under the Federal Provincial Arrangements and Established Programs Financing Act. In 1983-84, these payments will total over $24 billion.

5.2 Our objective in examining the forecasting process in the Department of Finance was to assess the reasonableness and adequacy of management controls over the preparation of forecasts of the Canadian economy in light of the important uses made of these forecasts. Our examination focused on the forecasting activities of the Fiscal Policy and Economic Analysis Branch. We reviewed the procedures involved in generating the regular quarterly economic forecasts, the development of econometric models used to create these forecasts, and the supporting electronic data processing (EDP) services.

5.3 There was a clear and consistent understanding of the forecasting process on the part of those involved in the forecasting activities of the Department of Finance; however, the process is informal and relatively unstructured. For example, there are no written procedures describing the steps to be followed in preparing a forecast. The forecasting process as a whole is almost completely dependent on the professional calibre and continuity of the Department's staff at all levels. The absence of formal procedures and of any review to ensure that all steps in the process have been adequately performed introduces a fragility and potential inefficiency into Finance's forecasting activities. Moreover, it provides little assurance to management that its directions have been carried out and that, apart from its final review of the forecast itself, the forecast is an accurate reflection of management's initial directions. Failure to carry out all procedures could cause errors affecting the quality of the forecasts.

5.4 We noted that the Branch has started to address some of the deficiencies we observed, such as the need for better documentation of the steps involved. Although there are no specific target dates for completion, these initiatives should help to strengthen the forecasting process.

5.5 Although our audit did not attempt to assess the reliability of the Department's forecasts, there is little evidence of any systematic, periodic evaluation of forecast results by the Department. Periodic comparisons of forecasts as a whole to economic performance would assist management in refining the forecasting process and in assessing the reliability of the forecasts.

5.6 There is a need for management practices and procedures governing the development of new and existing forecasting tools to reflect essential project control mechanisms, such as clearly defined and documented objectives, well understood research strategies, and performance reviews following each development segment.

5.7 The security over EDP systems and installations is weak. Unauthorized access to the latest forecast of the Canadian economy is possible. This could result in unauthorized release of the Department's outlook, with a subsequent impact on financial markets. The Department has made little progress in addressing many of the deficiencies identified in this area by the RCMP in 1979-80. The Department does not have a designated EDP co-ordinator or EDP policy. This could be a contributing factor in deficiencies noted in the co-ordination of EDP activities and also in EDP security.

Background

5.8 The Department of Finance generates regular, quarterly forecasts of Canada's economic performance. These forecasts are used within the Department as a basis for the federal government's economic and fiscal policies and for economic advice to Cabinet. Although the Department's forecasts are not generally circulated outside Finance, the Budget has traditionally included the latest departmental forecast. This budget forecast is regarded by other government departments and many private sector analysts as an authoritative forecast of the Canadian economy over the near future. As a result, the forecasts influence their own analyses and strategic decisions. The Department of Finance's forecasts thus have a significant effect on expectations of the future direction of the economy.

5.9 The forecasts prepared by Finance also have a tangible impact on the federal government's cash flow, since forecasts of key variables are used in calculating fiscal transfer payments and tax advances under the Federal-Provincial Arrangements and Established Programs Financing Act. In 1983-84, the federal government will transfer an estimated $24 billion to the provinces under this legislation. The forecasts affect the interim payments under this Act. The final transfer in respect of any year is determined by actual historical data.

5.10 The forecasting activities within Finance are concentrated in the Fiscal Policy and Economic Analysis Branch. The Branch is composed of four policy and analysis divisions, one of which includes a data services support group, the Data Services Unit (see Exhibit 5.1). In each division, there is some econometric modelling capability to facilitate policy simulation or forecasting, or both, in relation to specific areas of interest, such as energy or tax policy. The primary current forecasting group, however, is within the Economic Analysis Division, and it is responsible for the co-ordination and preparation of the quarterly outlook for the Canadian economy. Other forecasting and modelling activities within the Branch are, in many cases, linked with this effort in that they provide input to the short-term forecast, use short-term forecast outputs or support the forecast in other ways.

Exhibit not available

5.11 The quarterly economic forecast is essentially a forecast of the main National Income and Expenditure aggregates as well as other important variables, such as the Consumer Price Index, unemployment and the balance of payments. From 1970 to 1980, the Economic Analysis Division co-ordinated preparation of the forecast, but there was no complete departmental econometric model. In mid-1980, the Branch introduced such a model, the Quarterly Forecasting and Simulation Model (QFS), which built on existing econometric blocks. This model now forms the core of economic forecasting activity within the Department of Finance. The use of computer resources enhances the speed, accuracy and efficiency of these activities.

5.12 The QFS model is essentially a set of econometric equations. It is based on historical and current data obtained from sources external to the model, such as the Statistics Canada database, assumptions and internally-generated data. The products of this type of modelling activity are forecasts or estimated values of key variables of the Canadian economy, such as Gross National Product, unemployment, and balance of trade.

5.13 Econometric models such as the QFS are tools to assist in better forecasting and simulation. The final product, a set of forecasts, is highly dependent on the judgement of those using the models. In forecasting, there is no such thing as a perfect model and it should not be expected. Department personnel accept, however, that it is their responsibility to develop the best possible forecasting tools with the resources available. Improving econometric modelling is a continuing project in the Department.

Audit Scope

5.14 We examined management controls within the Department of Finance over the preparation of forecasts of the Canadian economy. In particular, we were interested in the controls over those figures used as the basis for calculating interim advances and transfers to the provinces. These activities are concentrated within the Fiscal Policy and Economic Analysis Branch and, in particular, within the Economic Analysis and Fiscal Policy Divisions. We traced key linkages to other branches in the Department, such as Federal-Provincial Relations, Social Policy and Administration.

5.15 Our audit concentrated on the Department's main forecasting tool, the Quarterly Forecasting and Simulation Model (QFS), and, to a lesser degree, on the linked tax models and the Public Debt Model. The audit covered three major aspects of the Department's forecasting activities:

    - the forecasting process: procedures for the review of data inputs, the method of generating a forecast of the Canadian economy, and the evaluation of forecast outputs;
    - the forecasting tools: procedures for initial and ongoing development and maintenance; and
    - electronic data processing: acquisition, security and organization.
5.16 We examined the procedures used to produce, verify and evaluate the short-term forecast and the management controls over these procedures. We did not assess the quality of the forecast or the theoretical basis underlying the main econometric model, the QFS.

Observations and Recommendations

The Forecasting Process

5.17 The forecasting process is a co-operative, interdivisional effort, orchestrated by the forecast co-ordinator of the Economic Forecasting Unit. The process requires substantial co-ordination of activities in various divisions within the Fiscal Policy and Economic Analysis Branch. The forecast begins when analysts prepare and submit assumptions to branch management for approval. These assumptions are the Department's best estimates of aspects of the economy which are not produced by the forecast itself, such as the outlook for the United States economy, and energy prices. This step is followed by updating data banks and running the model to produce a forecast. The forecast is then circulated throughout the Branch to allow analysts to review the results and make any necessary modifications. The model is then run again to produce a second, improved forecast. This process continues until a final forecast meets division and branch management approval. It is then sent for approval at the senior departmental management level.

5.18 Exhibit 5.2 depicts the phases of the forecasting cycle. At the time of our audit, Finance was not able to provide us with any written procedures describing the elements of the forecasting cycle and responsibilities for data input, processing, and review and distribution of forecast outputs. Individuals in the Branch had a clear and consistent understanding of each step of the forecasting process and their respective roles. Responsibilities, approvals and review procedures were, however, communicated verbally. There was no documented structure for ensuring that each phase was carried out at the appropriate time and that forecast inputs and outputs were reviewed and approved. There was no clear written delineation of responsibilities nor was there evidence indicating that procedures described to us were followed. Completion of these procedures depends wholly on the diligence, integrity and competence of those persons responsible for the forecasts.

Exhibit not available

5.19 Failure to perform all necessary procedures at the required time could precipitate errors that could affect the quality of the forecast. Moreover, forecasts are typically prepared under significant time pressure; written procedures could contribute to a more efficient operation in that individual analysts would have identified and commonly understood responsibilities. They would also help orient new staff members, who currently must learn the process through observation, experience and guidance from others, as well as help protect the continuity of the process. Finance has prepared a draft of such procedures since we completed our audit.

5.20 The Department should ensure that:

    - its written procedures cover review and approval of assumptions, data inputs, forecast outputs and their distribution;
    - instructions for the forecasting process provide sufficient detail to clearly establish key processing steps and individual responsibilities;
    - there are checkpoints identified such that management can be assured that these procedures have been followed; and
    - these procedures are implemented.
Department's response: Action has already been taken on drafting procedures for the forecasting process.

Evaluation of the Forecasts

5.21 Standard practice within the econometrics field requires testing econometric models for their predictive capabilities before they are implemented as regular forecasting tools. Once implemented, models are then usually evaluated on both an ongoing and a periodic basis to determine if their structure remains valid and their predictive capability is acceptable over time.

5.22 In the Department of Finance, the individual sector analysts, in conjunction with their section chiefs, are responsible for comparing forecasts of specific variables to actual results as part of the preparation of input for a forecast. The forecast's key variables are also compared with forecasts by other groups, such as the Bank of Canada, before being finalized. However, no records are kept of this analysis.

5.23 The QFS model is relatively new and went through substantial testing before being introduced. Despite recent major changes to the model, we found little documentation to show that the predictive capability of the model as a whole had been formally assessed on a regular basis since its introduction. A more formal assessment would help to identify areas where development should be concentrated. As well, it would give Finance further assurance on the quality of its forecasts to supplement the ongoing scrutiny by analysts and senior managers.

5.24 The Department should develop and implement formal procedures to evaluate regularly the predictive capability of its forecasting models, including a periodic internal review of their continuing integrity.

Development and Maintenance

5.25 Although the development proposal for the QFS was preceded by an internal review, there is no formal requirement that major development proposals be assessed formally for feasibility, including consideration of objectives, compatibility with existing work, resources and timing. In the case of the QFS, the preliminary assessment looked into alternative options for acquiring an in-house modelling capability and its impact on the Branch's performance in forecasting and simulation, but did not assess the impact in terms of ongoing costs. There is no documentation available to indicate that development of the other branch models had been assessed for feasibility and approved.

5.26 Major projects should be preceded by a feasibility study, including objectives, proposed strategy, compatibility with existing work, resources and timing. Formal approval should be given prior to development.

Department's response: The newly re-established Branch Model Development Committee will be responsible for advising on the implementation of major development projects.

5.27 Ongoing econometric research to upgrade the forecast tools, the QFS in particular, is directed by Branch section chiefs and their assistants who meet after each forecasting cycle to discuss problems, identify projects, set priorities and assign responsibilities. For all development activity, whether it is to create a new model such as the QFS or to refine existing models, there is a reliance on a peer review process whereby individual analysts throughout the Branch are expected to critique each other's work. In the case of QFS's initial development which was concentrated in the summer of 1980, there was a rigorous structure for the co-ordination of this peer review process under a Branch Model Development Committee (BMDC) which was established specifically for this purpose. This Committee was responsible for the ultimate theoretical soundness of the QFS, and it directed the assembling of existing econometric blocks, identified gaps and steered research toward eliminating them. This Committee was dissolved after the QFS was implemented. The peer review process for other research and development activities is not structured, and there is no documentation to show that the results of such research are reviewed and approved by management prior to implementation. Such a review would help ensure that individual projects are compatible with other analyses and in line with Branch objectives.

5.28 Although individual analysts had a common and consistent knowledge of the steps to undertake in model-based research, there were no written procedures describing standard items to be addressed in econometric research, such as objectives, data sources and limitations, required testing and approvals. The Department has recently prepared a list which includes some of these items. At the time of the audit, the model co-ordinator in the Economic Analysis Division reviewed and tested all proposed refinements to the QFS prior to implementation, but the extent of testing was at the co-ordinator's discretion.

5.29 We believe that written procedures would be useful in providing a guide and a checklist for analysts to follow. They would not, of course, be a substitute for an individual analyst's imagination, initiative and competence, upon which successful econometric research depends.

5.30 Management should prepare written procedures for econometric research projects, including documentation, testing, implementation, approvals and post-implementation review.

5.31 Consideration should be given to establishing a structure similar to the Branch Model Development Committee to co-ordinate and review econometric research.

Department's response: It has been decided to re-establish the Branch Model Development Committee. It will be constituted at the Director level and chaired by the Director of Economic Forecasting. It will be responsible for, among other things, advising on the implementation of major development projects as recommended earlier.

Documentation

5.32 Documentation supporting the QFS model, its testing and operation was minimal at the time of our audit. The same situation applied to the tax and public debt models, although documentation was being prepared for the tax models. There were no central archives of all technical papers supporting the QFS model's development process after the initial development period. Existing documentation essentially consisted of technical papers held by each analyst, the model and its latest forecast maintained on computer, and a brief overview of the model's structure, prepared in March 1982. At the time of our audit, the most complete, recent description of the model and its properties was a paper for the comparative models seminar initiated by Finance and the Bank of Canada.

5.33 The current lack of adequate documentation concerning the development of the forecasting tools, their testing and ongoing review may limit management's ability to maintain and revise efficiently its modelling and forecasting capabilities. In addition, inefficiencies in operating such models are likely to arise, particularly when staff turnover or absences occur. In the event of staff leaving the Department, inadequate documentation would make it difficult for management to maintain continuity. The Department has started to prepare some documentation. This would enhance the efficiency and stability of the forecasting process.

5.34 Complete documentation of the forecasting models should be developed and maintained in a central location, supported by a chronological file of previous development papers.

Department's response: Action has already been taken on preparing comprehensive documentation on the Department's QFS macroeconometric model and a first draft, prepared in April 1983 was made available to the Audit team.

In response to the six recommendations in paragraphs 5.20, 5.24, 5.26, 5.30, 5.31 and 5.34, the Department has decided to establish an Economic Forecasting Division within the Fiscal Policy and Economic Analysis Branch, primarily through the re-allocation of person-years from elsewhere in the Department. The establishment of this Division, which will be headed by a Director of Economic Forecasting, will provide greater focus to the management and co-ordination of the forecasting process and to the development, documentation and effective dissemination of forecasting methodologies. The specific recommendations are being carefully examined in the context of defining the mandate of this division, and appropriate action will be taken to formalize existing procedures and to develop additional practices as recommended.

Electronic Data Processing

5.35 Electronic data processing contributes significantly to an effective and efficient econometric forecasting operation. For this reason, we included EDP in the scope of the audit. Some of the issues which we addressed, however, are Department-wide in nature and are not limited to the forecasting process alone. EDP expenditures for 1982-83 for the Department as a whole were estimated at $2.1 million. Of this, $1.6 million (77 per cent) was used by Fiscal Policy and Economic Analysis in its forecasting, simulation and analytic work, and its provision of data services to the rest of the Department.

5.36 Policy, organization and acquisition. There is no published departmental EDP policy, as required by Treasury Board. Such a policy would cover the provision, administration and control of EDP services and would identify organizational roles and responsibilities. This is a factor in a least some of the problems identified by the RCMP's inspection report, discussed below. Similarly, there is no officially designated EDP co-ordinator, as recommended by Treasury Board. The creation of such a position could reduce the likelihood of co-ordination problems and duplication of effort between the Fiscal Policy and Economic Analysis Branch and the Administration Branch. The former, for historical reasons, is the centre of the Department's EDP expertise, while the latter provides EDP services to Treasury Board and the Office of the Comptroller General as well as to the Department of Finance. In December 1982, the Administration Branch proposed a strategic plan for information processing in Finance. The proposal addressed organizational, technological and control issues, but no decisions have been taken on the proposal.

5.37 We found that the process for acquiring computer hardware, software and services has functioned adequately in the past in that the manager of the Data Services Unit within the Fiscal Policy and Economic Analysis Branch has assumed the role of EDP co-ordinator for the relatively few acquisitions that the Department has made. However, the Administration Branch is considering acquiring in-house computer equipment. Because neither the Administration Branch nor the Data Services Unit has clear and total authority for acquisition functions, the existence of the two separate systems groups with overlapping responsibilities and unclear mandates could contribute to a lack of co-ordination of EDP services.

5.38 A departmental EDP policy should be developed covering all aspects of EDP activities within the Department, including acquisitions, security, and operational and administrative procedures.

5.39 The Department of Finance should formally designate an EDP co- ordinator having the responsibilities set out in the Treasury Board Administrative Policy Manual.

5.40 The present organizational structure for providing EDP services should be re-examined with a view to co-ordinating the responsibilities for departmental EDP activities.

Department's response: In response to the recommendations 5.38, 5.39 and 5.40, internal proposals for ensuring that responsibilities for departmental EDP activities are effectively co-ordinated, that an EDP co-ordinator is formally designated and that a departmental EDP policy is developed, are currently being reviewed by the Department's Management Advisory Committee.

5.41 Security. In August 1980, the RCMP submitted a security evaluation report to the Department on its data processing operations and activities. The evaluation covered all security-related aspects of the computer operation, including organizational, administrative, personnel and environmental considerations, as well as communications, software, and operations. The security evaluation report, the summary volume of which was sent direct to the Deputy Minister, indicated that the facilities and systems in place did not satisfy the requirements for secure processing.

5.42 The problems identified by this report constituted a serious threat to the security of the Department's data processing. They also indicated a lack of compliance with Treasury Board policies. Treasury Board also requires that, after completion of such an evaluation, departments respond within six months on the actions taken to correct identified problems. In October 1981, 14 months after receiving the evaluation report, the Department sent a status report to Treasury Board. In July 1982, the Department sent an update, indicating that 43 of the 79 recommendations were implemented or required no further action. All 36 of the outstanding recommendations at that date related to requirements for developing EDP security policies, procedures and standards. These were still outstanding at the time of our audit.

5.43 The Department should review the outstanding security recommendations and take any necessary action to correct the deficiencies noted.

Department's response: The recent acquisition of modest in-house data processing capability has increased the Department's ability to respond to the outstanding security recommendations, and these are being reviewed.

5.44 Our audit also showed that 6 of the 43 recommendations noted as implemented or as requiring no further improvement still needed further action. The most significant of these concerned procedures to ensure that former employees cannot access confidential information stored on the computer system. This is important because the system on which the economic forecasts are stored is protected through a system of account numbers, passwords and computer control software. Although the model structure itself appeared adequately protected from tampering, individual sector analysts may access the model and its solution to read or copy the model logic or solution. At the time of our audit, there was no formal procedure to delete the account numbers or passwords of former employees, and there was no automatic mechanism for notifying the manager of Data Services of an employee's departure so that appropriate precautions might be taken. Failure to delete passwords of former employees allows unauthorized access and release of the forecast. For example, an individual with access to a compatible terminal remote from the Department would have access to Finance's latest forecast solution.

5.45 The Department should implement procedures to ensure against unauthorized access to sensitive information.

Department's response: Administrative procedures have been put in place to protect against the possibilities of unauthorized access to sensitive data by former employees.

5.46 Physical security is also loosely controlled. On more than one occasion, we observed that terminal rooms were left empty, unlocked and open to access by unauthorized personnel. This lack of security provides potential for damage to machines, interference with processing, and access to confidential material.

5.47 The Department should strengthen its physical security arrangements.

5.48 The Departmental Security Officer is responsible for maintaining lists of locations of computer hardware for physical security purposes. Because computer terminals have been installed without his knowledge, he informed us that he could not ensure that proper precautions were taken to protect new installations against unauthorized access.

5.49 As part of the formal acquisition process, the Departmental Security Officer should be informed of intended equipment acquisitions so that appropriate security and environmental precautions can be taken.

Department's response: Administrative procedures have been put in place to comply with this recommendation.

In response to the recommendations in paragraphs 5.43, 5.45, 5.47 and 5.49, a complete resolution of the issues raised in these recommendations will have to await the development of departmental policies and structures for EDP operations, as well as the pending relocation of the Department to new facilities.