Performance Audit Manual

4.1 Every facet of a performance audit requires professional judgment and individual initiative. After an entity (or portion of an entity), sector or functional area has been chosen, decisions need to be made about:

• what and how much to audit;
• what audit approaches, methodology and technology to employ to assess performance; and
• what staff skills, disciplines and experience to assign to the audit.

4.2 The Office's credibility, the cost-effectiveness of the audit, and the quality of the reports of the Auditor General and the Commissioner of the Environment and Sustainable Development depend on sound judgment being exercised throughout the entire audit process.

4.3 The approach taken to arrive at a conclusion against each audit objective is an iterative one, and information is gathered and assessed; decisions are made whether to proceed to the next stage or whether additional input and consultation are necessary. The approach allows teams to identify at an early stage if an audit will not be cost-effective or if the approach needs to be revised. Audits can be modified or cancelled before significant costs are incurred.

4.4 Figure 2 illustrates the approach.

4.5 Consultation is an integral part of the process to assist in specific judgments or decisions in the audit. For example, the audit Principal is expected to consult, at the critical decision points during the audit, with the Assistant Auditor General (AAG)/Commissioner of the Environment and Sustainable Development (CESD), functional responsibility leaders (FRLs), Product Leader for Performance Audit (PL-Performance Audit), Subject Matter Experts (SMEs), the Quality Reviewer, the Audit Advisory Committee, departmental management and other support groups in the Office, as appropriate.

4.6 The following general characteristics, therefore, summarize the basic approach:

• professional judgment and individual initiative;
• consultation at the key decision points;
• an iterative process to maintain a focus on matters of significance and interest to Parliament;
• staff and audit methodology tailored to reflect the characteristics of the entity or functional area; and
• audit costs kept in balance with the significance of the issues being examined and their interest for parliamentarians.

Click the thumbnail below to view an enlargement.

Planning the performance audit

4.7 Prior to starting field work, a process of setting priorities, developing strategic and long-range plans, submitting chapter proposals, rationalizing resources and assessing anticipated audit worth has taken place. In selecting audits through this process, the audit management uses their preliminary knowledge of the subject area to form a reasonable basis for believing that the audit can be completed in accordance with the performance audit policies.

4.7.1 Early in the planning phase of an audit, the Performance Audit Management Committee (PAMC) receives and approves a chapter proposal. This one-to-two-page proposal:

• Describes the program/areas/issues to be audited (including if possible, the dollars and FTE's involved) and the objectives of the survey phase;
• Includes which AG focus area(s) the audit addresses;
• Presents the timing for the planning phase of the audit (including the planned start date, date of the first Advisory Committee meeting, and the date of the survey report);
• Presents the estimate of hours and contract and travel dollars budgeted for the entire audit and the portion of each allocated for the planning phase; and
• Identifies the mandatory FRLs and any other FRLs/SMEs that will be consulted during the planning phase.

This process results in the approval to begin the overview/survey phases of an audit.

4.8 DELETED. (October 2004)

The overview stage: understanding the subject of the audit

4.9 Audit teams acquire a sound knowledge of the audit subject (department, agency, sector or function) prior to commencing detailed planning of an audit. Irrespective of the size and nature of the subject, it is important for the audit team to understand "the big picture". Forming audit conclusions or reporting weaknesses without this overall knowledge may result in unproductive audit work or misleading findings. The audit team should have up-to-date knowledge of:

• significant legislative authorities;
• organizational arrangements;
• the environment in which the entity operates;
• key personnel;
• spending levels and revenues;
• the entity's clients;
• the objective, mission and expected results;
• major operations, including in the field;
• the accountability arrangements;
• the major control systems;
• major risks facing the entity, including control risk (the risk that significant error will not be prevented, detected or corrected by the related internal controls) and inherent risk (the susceptibility of the subject matter by its nature to significant error where there are no related controls);
• environmental issues in the context of sustainable development and the entity's Sustainable Development Strategy; and
• prior deficiencies/known weaknesses. (May 2005)

By including it in the list above, special emphasis is given to environmental issues to ensure that audit teams integrate fully the amendments made to the Auditor General Act in 1995. This is also linked to the identification of these issues and risks as part of audit teams' one pass planning. The "4th E Practice Guide" contains guidance and tools to assist auditors to identify and assess environmental issues and risks in their performance audit work.

4.10 Figure 4 illustrates important features and complexities of a typical department or agency.

4.11 This knowledge provides the basis for describing the entity, making initial scoping decisions and defining lines of inquiry. It is also used to confirm which FRLs are mandatory and determine which SMEs or PLs to consult.

4.12 An audit team with considerable experience in auditing the department or agency may have cumulative knowledge to satisfy these requirements without engaging in a formal overview stage. In situations where the organization has not been audited recently or where there has not been continuity in the audit team, an overview study may be necessary.

4.13 Where a government-wide or sectoral audit is being carried out, an overview is required, and a submission is made to the Executive Committee at the end of the overview stage. Consultation should take place with the audit teams responsible for the entities or functional areas affected by the audit to obtain the necessary background information. The amount of knowledge necessary for these types of audits depends on the nature of the examination being conducted.

4.14 When an overview is done, an overview report is produced that discloses the subject of the audit, areas to explore during the survey stage and the reasons they were selected, an initial estimate of costs and milestone dates for the audit, a list of the FRLs, PL-Performance Audit and SMEs, and entity or functional Principals that will be consulted during the audit, and the skills needed to carry out the audit.

The survey stage

4.15 The purpose of the survey is to gain sufficient knowledge of the subject area for confirming that the audit can be conducted in accordance with the performance auditing policies, and to develop an examination plan that will provide a basis for the orderly, efficient and cost-effective conduct of the audit.The plan is prepared by the audit Principal, reviewed by the Quality Reviewer and FRLs, PL-Performance Audit and SMEs, as appropriate, possibly reviewed by the Audit Advisory Committee if appropriate, and approved by the AAG/CESD.

4.16 The survey is a broad-based appraisal of the operations subject to audit, without carrying out detailed verification. The auditors gather information in order to fine-tune initial decisions about scope, cost, timing and skills, and to propose audit objectives, areas for in-depth review, criteria, and examination approach. In finalizing these decisions, the audit team designs an audit to reduce the risk of making erroneous observations, faulty conclusion and inappropriate recommendations in the report to correspond with the level of assurance provided by the work.

4.16.1 When an overview is not formally done or is combined with the survey phase, the chapter proposal reviewed by the Performance Audit Management Committee can be used as the survey plan that the team will follow in conducting its survey phase.

Like the chapter proposal, the survey plan is a short document that:

• Describes the program/areas/issues to be audited (including if possible, the dollars and FTE's involved) and the objectives for the survey phase;
• Includes which AG focus area (s) the audit addresses;
• Presents the timing for the planning phase of the audit (including the planned start date, date of the first Advisory Committee meeting, and the date of the survey report);
• Presents the preliminary estimate of hours and contract and travel dollars budgeted for the entire audit and the portion of each allocated for the planning phase; and
• Identifies the mandatory FRLs and any other FRLs/SMEs that will be consulted during the planning phase.

4.17 A wide variety of procedures and techniques are used to gather the necessary information. These may include:

• interviews with management;
• review of authorities, policies, directives, Cabinet documents, etc.;
• review of entity's Performance Report and Report on Plans and Priorities; and Sustainable Development Strategies;
• review of entity's Internet site;
• review of management and accountability reports;
• review of result commitments;
• observation of facilities;
• walk through of major systems and control procedures;
• analysis of the relationship between resource utilization and results;
• consider related environmental issues, including discussion, as needed, with FRL-Environment (Guidance can also be found in the "4th E Practice Guide");
• assessment of risks facing the entity, including control risk and inherent risk;
• consultation with advisors and outside organizations to identify best practices and opportunities for improvement;
• previous audits and studies and audits conducted by others;
• survey of the use of technology; and
• review of spending trends.

4.17.1 As noted in 4.9, audit teams should have up-to-date knowledge of major controls systems and risks (both control risk and inherent risk) facing the entity prior to detail planning of the audit. For audits in which the team intends to rely on controls, a more in-depth knowledge and assessment of the controls is required, and the audit team should obtain sufficient appropriate audit evidence through tests of those controls.

4.18 The time spent at the survey stage of the audit will usually result in a more organized and cost-effective audit. There is no universal approach to ensure effective decision making during this stage. The audit Principal and the team need to develop a thorough understanding of the audit subject, and exercise significant judgment. The resulting examination plan should provide a clear focus to guide the audit to a successful conclusion. This is critical to the identification of the issues that will be reported. Performance audit reports contain conclusions about complex government operations, and their relevance and impact is heavily influenced by decisions made during the survey phase.

4.19 Advice and concerns received are documented. The examination plan is approved by the AAG/CESD. A summary examination plan is then prepared and submitted to the Performance Audit Management Committee for approval and is posted on the INTRAnet.

4.20 An important tool used in all phases of the planning process is risk assessment. Risk is defined as the probability that an event or action may adversely affect the organization, such as exposure to financial loss, loss of reputation, or failure to deliver the program with economy, efficiency, cost-effectiveness or taking into account the environmental implications. A risk assessment requires the auditor to ask the following type of questions:

• What can go wrong?
• What is the probability of it going wrong?
• What are the consequences?
• Can the risk be minimized or controlled?

The survey report

4.21 The results of the survey are documented in a survey report. The report includes footnote references to the relevant analysis and documentation supporting the report that are maintained in the permanent files or working papers, as appropriate. The survey report contains those aspects of the audit on which the audit team consults with the Audit Advisory Committee and others. The report should at least include:

• preliminary audit objectives;
• overview and background of the organization or function under audit;
• environmental issues considered and reasons for their inclusion or exclusion in the proposed audit scope;
• an indication of how the audit links to Office priorities/focus areas and to risks identified in the one-pass planning;
• survey findings for the matters of potential significance;
• proposed audit scope and a general description of the proposed audit approach;
• draft audit criteria and their sources;
• timing of the audit, including the likely timing of Audit Advisory Committee meetings; and
• reporting strategy, including the response to any direction from the Executive Committee.

4.22 The audit Principal is responsible for the preparation of the survey report. The AAG/CESD documents his/her approval for distribution of the report before presenting it for consultation to the Audit Advisory Committee.

The Examination Plan

4.22.1 Following consultation with the Audit Advisory Committee, appropriate FRLs, PL-Performance Audit and SMEs, and the Quality Reviewer, the plan for the completion of the audit is finalized and documented in an examination plan. Audit procedures with appropriate level of detail to address the audit objectives and criteria are addressed in the examination plan. Where the team considers it necessary, it may use audit programs to set out more detailed audit procedures. The examination plan should contain the following:

• audit objectives;
• final audit scope, major considerations and rationale for the scoping decisions, reasons for any scope limitations, and how the audit addresses the risks identified in one-pass planning and in the planning stage of the audit;
• environmental issues included in the final audit scope;
• specific issues including environmental issues, selected for audit and the objectives for each issue/project;
• specific issues raised in the audit and the objectives of each question/project;
• audit criteria and their sources;
• for each project, description of the audit approach and methodology (that is, nature, extent and timing of evidence to be collected and analyzed taking into account the identified risks and tests for reliance on controls), including opportunities to quantify the results and the approach for the use of secondary evidence;
• for each project, identification of audit staff, including regional and functional staff and the qualifications of contractors engaged for their special knowledge or skills;
• estimated cost for each project in terms of hours and contract and travel dollars, including where necessary an assessment of variance from initial approved audit budget;
• timing of the audit, milestones/control points and timing of Audit Advisory Committee meetings; and
• timing, estimated costs and resources to complete the reporting phase and all its requirements. (October 2004)

4.22.2 The audit Principal is responsible for the preparation of the audit examination plan. It is the audit Principal's responsibility to ensure that the Quality Reviewer is consulted and receives the information needed to perform his/her review prior to the approval of the plan by the AAG/CED. The AAG/CESD documents his/her approval of the plan before the examination work begins. It is the audit Principal's responsibility to ensure that the mandatory Functional Responsibility Leaders and the Quality Reviewer are consulted and receive the information needed. (October 2004)

4.22.3 The examination plan should confirm the original audit budget or result in a revised budget being submitted for approval. In the case of a large variance, the AAG/CESD should submit the revised budget to the Executive Committee and to the Comptroller for information. This will allow for the timely reallocation of resources.

4.23 The examination plan is the basis for a summary examination plan submitted by the team to the PAMC.

4.24 Any major revisions that are made subsequent to the audit objectives, scope, budget requirements, reporting strategy, cost or timing of the audit should be approved by the AAG/CESD and the PAMC.

Audit objectives

Audits should have clear objectives that can be concluded against.

4.25 Audit objectives are normally expressed in terms of what questions the audit is expected to answer about the performance of an activity; for example, results achieved, economy or efficiency. Ideally, audit objectives would be consistent with the achievement of results of the entity, sector or functional area. In general terms, the objectives of a typical performance audit are compatible with the Office's Strategic Plan.

4.26 The audit objectives are to be carefully considered and clearly stated. They must be defined in a way that will allow the audit team at the end of the audit to conclude against each of the objectives. Future audit effort will be directed toward answering the questions raised in the objectives. The audit objectives should therefore be defined as precisely as possible in order to avoid unnecessary and expensive audit work. Any changes to the audit objectives, and the major considerations and rationale for such changes, should be brought to the attention of the AAG/CESD, the audit advisory committee, the Quality Reviewer and the PAMC.

4.27 In many cases, the audit work also includes providing valuable and necessary information to Parliament. Such non-audit objectives (... to provide an overview of...) for which a conclusion cannot be reached and is not expected, should be separated from audit objectives (...to determine whether...is efficient), for which a conclusion can be reached.

4.28 Historically, the Office has relied on direct reporting. Direct reporting is done in a situation when there is no assertion by the auditee, and the performance auditor audits the subject matter and reaches a conclusion on it. In an attestation engagement, on the other hand, the auditee makes an assertion and the auditor expresses an opinion on the assertion.

4.29 There is a move to performance reporting on a government-wide basis. All departments and agencies are expected to produce a Report on Plans and Priorities and a Departmental Performance Report (DPR). This provides a greater opportunity for the Office to audit DPRs in an attest mode.

4.30 However, this process is just beginning and understandably progress is still minimal. Therefore, the Office will continue to rely on direct reporting for some time.

Audit scope

Audits should have a clear scope that focuses the extent, timing and nature of the audit. Audits should select issues on the basis of their relevance to the Office's mandate, significance and auditability.

4.31 During the early planning stages, the activity to be audited is often defined in broad terms. Very seldom is it practical or cost-effective to audit everything. Scoping the audit involves narrowing the audit to a relatively few matters of significance that pertain to the audit objective, can be audited with the resources available, and are critical to the achievements of the intended results of the audit subject. There are three underlying principles in establishing the scope of the audit:

• relevance to the mandate
• matters of significance
• auditability

Relevance to the mandate

4.32 The Auditor General Act provides the Auditor General considerable latitude in deciding what to audit. The fact that certain matters are specifically identified in the Act for inclusion in reports indicates that they are matters of interest to parliamentarians. The mandate of the Office and the interests of parliamentarians are key factors in assessing the relevance of matters to audit.

4.33 The merits of political policy are beyond the scope of our audits. Refer to paragraph 1.6 on the relationship between the audit function and government policy.

Matters of significance

4.34 The five focus areas set out in the Office's Strategic Plan provide an important tool when considering audit worthiness of potential areas of audit. They state that we focus on significant issues and matters that will add value and serve Parliament and the well-being of Canadians. Identifying matters of significance for audit involves answering the following type of questions:

• Does the subject have an important impact on results?
• Is it an area of high risk?
• Does it involve material amounts?
• Does the audit have the potential to result in improved performance, accountability or value for money? Will it make a difference?
• Is it an issue with visibility or of current concern? Is it of interest to parliamentarians and Canadians? Is the timing opportune for the audit and to meet the needs of the client?

4.35 The purpose of the scoping exercise is to allow the concentration of audit resources and effort on a relatively few areas that can have a significant impact on the performance and results of the subject being audited.

4.36 The identification of matters of significance is usually carried out by taking a top-down approach. Most organizations have a hierarchy of objectives and planned results, reflected in their Program Activity Architecture (PAA) and controls. The activities, procedures, controls and transactions tend to mushroom as one moves down the hierarchy. In larger organizations, there may be hundreds or thousands of procedures and controls at the lower end of the hierarchy. A top-down approach allows a global perspective to be taken of what is important. (Figure 4 illustrates the complexity of a typical department.)

4.37 One of the outputs from the scoping exercise is the identification of matters of potential significance or issues for in-depth audit. Typically, the five or six matters most critical to the success of the activity being audited, or those that present the greatest risks or opportunity for improvement, are chosen for detailed audit. Relentless attention by the auditor is needed to identify and focus the audit on the critical operations.

Auditability

4.38 Auditability relates to the audit teams ability to carry out the audit in accordance with professional standards and audit policies. A variety of situations may arise that may cause the audit team to decide not to audit a particular area even though it is significant. In reaching such a decision, the audit team should have concluded that:

• The nature of the activity is inappropriate; for example, it may not be practical to attempt to audit the technical considerations of a research facility.
• It does not have or cannot acquire the required expertise.
• The area is undergoing significant and fundamental change.
• Suitable criteria are not available to assess performance.

4.39 The scope statement should describe the parts or functions of the organization/program that are the subject of the audit and to which the audit conclusions apply as well as the time period covered by the audit.

Audit criteria

Audits should have suitable criteria that focus the audit and provide a basis for developing observations and conclusions.

4.40 Auditors need a means of measuring or judging the performance of the matters subject to audit. The standards used for this purpose are referred to as audit criteria.

4.41 Audit criteria are reasonable and attainable standards of performance and control against which compliance, the adequacy of systems and practices, and the economy, efficiency and cost effectiveness of operations can be evaluated and assessed.

4.42 Suitable criteria are criteria that are appropriate to the particular characteristics of the audited organization. They focus, wherever possible, on the results expected to be achieved by the operation, system, control, etc. The assessment of whether or not criteria are met results in audit observations.

4.43 Criteria should be developed for each of the lines of audit inquiry. They are to be relevant, reliable, neutral, understandable and complete. The aggregate of the observations allows the audit team to form a conclusion against each audit objective.

4.44 The sources of the criteria determine the amount of effort needed to assure the suitability of the criteria. Criteria are either generally accepted or not. Potential sources are listed in paragraphs 4.45 and 4.48 in descending order of relevance. (May 2006)

4.45 The sources of generally accepted criteria are:

• laws, regulations and central agencies requirements; and
• standards developed by recognized professional organizations that follow due process. (May 2006)

4.45.1 Criteria based on the law or regulations can be accepted by the auditor. In these circumstances, the auditor needs only to ensure that they are related to the audit objective. The same is true of central agencies authoritative requirements (such as policies) that departments and agencies must comply with and that are issued under the authority of their Acts and ministers. This category of criteria does not include the guidelines and tools developed by the central agencies for use by departments and agencies at their descretion. These are discussed in paragraph 4.48.3. (May 2006)

4.45.2 Criteria developed by recognized professional organizations that follow due process also come from a source generally accepted. Due process means that criteria have been developed through consultation, appropriate challenge and vetting and that they reflect consensus among professionals. (May 2006)

4.46 When there are no generally accepted criteria consistent with the objective of the engagement, criteria that are not generally accepted should be used. (May 2006)

4.47 DELETED. (May 2006)

4.48 Criteria that are not generally accepted, i.e., that lack authoritative support, can also be used after their suitability has been established by the auditor through sufficient research and validation. The sources of these criteria are:

• standards developed by recognized professional organizations that do not follow due process;
• criteria used by the Office or others in similar engagements unless they come from sources in paragraph 4.45;
• standards established by the audited organization; and
• standards and practices used by other organizations carrying out similar activities. (May 2006)

4.48.1 Criteria developed by recognized professional organizations that do not follow due process could be used. A number of reasons may justify the absence of due process without detracting their relevance and overall suitability. However, as could be expected they will not be as authoritative and will require due care on the part of the auditor to ensure that their legitimacy is well established and accepted by the entity audited. (May 2006)

4.48.2 Over the years, the Office has developed and tested criteria for a large number of entities and activity areas. These may apply well to more current audits. However, the fact that these criteria have been used in the past does not, by itself, make the criteria authoritative. It is the responsibility of the auditor to re-assert the source and suitability of the criteria. (May 2006)

4.48.3 Primary sources of criteria for performance audits are the controls, standards, measures, result commitments and targets adopted by the management of the organization. Where the entity has adopted meaningful and specific measures for assessing its own performance, the auditor should carry out a review of those relevant to the audit to ensure that they are reasonable and complete. Guidelines and tools developed by the central agencies and adopted by the audit entity should be treated in this way. The audit team can consult with professional bodies or other organizations carrying out similar activities or operations to test the quality of the standards or to identify best practices. Where the entity's own measures are found to be suitable, they can be adopted as the audit criteria. Central agencies' guides and tools should not be confused with the requirements mentioned in paragraph 4.45. It is up to the entities to use them or not and consequently they cannot be treated as being as authoritative as policies. (May 2006)

4.48.4 The auditor can rely on performance data of other organizations, inside or outside of the government, that have similar activities or operation, best practices determined though benchmarking or consultation; and standards developed by the auditor through the analysis of a task or activity. (May 2006)

4.48.5 Benchmarking and the development of standards through analysis of individual activities and/or comparison with similar activities in other organizations are costly activities and would not normally be undertaken by the auditor. Where the audit survey indicates the potential for significant improvement to operations or savings, the auditee could be encouraged to carry out such activities. In extreme circumstances, the auditor can seek advice on the advisability of carrying out such tasks from senior management of the Office and the Audit Advisory Committee. (May 2006)

4.49 The audit Principal should discuss the audit objectives and the criteria to be used, as well as management's responsibilities for the subject area, with senior officials in the audited organization and obtain written comments, if possible, on the suitability of the criteria and the team's understanding of management responsibility in the context of the audit approach. Auditors may use the "Entity Plan Summary" template to help prepare the Summary plan. If there is disagreement with management on the team's understanding of management responsibility or on the suitability of the criteria and the conflict cannot be resolved, the Principal should consult the AAG/CESD and the Audit Advisory Committee before proceeding with the audit. Under no circumstances is the audit to be carried out using criteria that would result in biased or misleading audit results. (May 2006)

4.50 DELETED. (May 2006)

4.51 If there is disagreement with management about criteria or management responsibilities, this is to be disclosed in the chapter with an explanation of why the audit team believes management is responsible for the subject matter and/or why the team used the criteria despite management's objection.

4.51.1 The source of the criteria should be disclosed in the chapter. If the criteria were developed specifically for the audit, the discolusre statement should provide indications on the process followed to validate them. (May 2006)

4.52 As the audit progresses, additional information may result in certain criteria not being necessary to achieve audit objectives. In these circumstances, further audit work related to the criteria is not necessary.

Audit approach: a focus on results

4.53 Having defined the audit objective, scope and criteria, the audit team needs to design an audit approach that will produce the most meaningful audit result for the client, in a most cost-effective manner. This applies equally to direct reporting and attestation audits.

4.54 Parliamentarians have indicated a preference for information that is results-oriented and at a high level. To the extent possible, audits should be designed to provide information that points to areas of interest to parliamentarians.

4.55 In the past, many audits were driven by control and process concerns rather than added-value considerations. Because of this, the Office has shifted the emphasis of its audits, in recent years, to focus more on results. This requires that audits, irrespective of the approach, identify, wherever possible, the effect or potential effect of audit findings. A focus on results should be kept regardless of whether the scope of the audit is a program, an operation, a system or a control. When carrying out an audit of a component of a program, the auditor needs to understand its relationship to the intended results of the program.

4.56 An audit that does not provide the "so what" of the issue will likely receive an indifferent reception from parliamentarians and the management of the audited organization. It may also cause the auditor difficulty in concluding against audit objectives.

4.57 This section briefly describes two main audit approaches:

• Auditing results directly. This approach focusses initially on outputs and outcomes.
• Auditing the control systems. This approach focusses initially on systems and controls.

4.58 Developing a practical and effective audit approach brings out the diversity and complexity of performance auditing. There is vast room for innovation in the application of new techniques and, over the years, the Office has employed computer-assisted techniques, operations research, simulation and modelling, statistical sampling, surveys and a variety of other advanced methods to collect audit evidence. For any specified audit, a combination of approaches may be used.

Auditing results directly

4.59 Departments and agencies are now required to define results commitments in their Program Activity Architecture (PAA) and to report goals and actual performance in the Estimates documents tabled annually. These provide excellent points of reference for results-oriented auditing.

4.60 The concept of a results-oriented approach can apply irrespective of whether the scope of the audit is a program, an operation, a system or a control.

4.61 This type of audit focuses on assessing the results achieved in relation to those intended. The audit does not initially examine the details of the methods or processes but looks at the outputs or outcomes themselves. The approach is particularly appropriate where there are suitable criteria available to measure the quality, quantity and cost of the outputs. If the result is satisfactory, the risk of there being serious flaws in the design or implementation of the activity or process is minimal. Where the auditor finds the result to be unsatisfactory, the activity and the control system are examined to the extent necessary to identify the specific causes of the problem.

4.62 The types of problems that may be identified include:

• services that are not in accord with the program mandate;
• unit costs that exceed departmental standards, or costs of comparable activities in other sections of the government or in outside organizations, and
• goods or services that do not meet standards of quality or quantity.

4.63 Where the audit objective is to examine the achievement of program objectives, the auditor exercises caution that the audit does not question the merits of political policy. The Office has no desire to enter into a political policy debate — that is the job of the politicians.

An example of a results-based audit: Succinctly stated, an objective of the audit was to assess the cost-effectiveness of the payroll system. The audit was focussed on the reasonableness of the annual cost of paying an employee, and whether the employee was paid on time and in the right amount. The examination of the accuracy of the pay and the timing indicated that these aspects of the payroll were satisfactory. Criteria on costs were based on the costs of paying provincial government employees. This information was obtained with the assistance of several provincial auditors general. Comparison of actual costs with the criteria indicated potential for very large savings in the federal system. Further analysis indicated that greater use of technology would substantially reduce costs. The department reported that its subsequent actions to modernize the payroll system resulted in millions of dollars in annual savings.

Auditing the control systems

4.64 This approach is designed to determine if the organization has adequate control systems to provide reasonable assurance that the intended results are achieved. The word control is taken in its widest interpretation and embraces all of the elements of management that are required to achieve an intended result. The audit is designed to carry out analysis, review and testing of the key components of the control system to ensure that it is appropriately designed and implemented. If the control system is effective, it provides a strong indication that the results will be satisfactory.

4.65 Normally, only high-risk components of the system would be reviewed in depth. Controls are chosen for audit on the basis of their significance to the achievement of key results. Where major deficiencies are identified, the auditor takes further steps to identify the cause of the problem and its effect or potential effect on intended results. The approach provides a solid foundation for making recommendations to improve the systems and practices and for identifying unnecessary controls.

4.66 Flow charts are often used to analyze the system. The disadvantage of this approach is that in a large, complex organization the cost of detailed systems analysis is high. It is also frequently difficult to identify what impact a control deficiency will have on results.

4.67 In both approaches described above, the auditor may examine the actual transactions, events, records or documents. The basic methodology is to define the population to be tested, select a sample, and then examine the transactions against the standard or criteria. Testing is directed toward results whenever possible. For example, a sample of purchases could be tested to determine whether a department is paying too much overall.

4.68 Sampling may be the primary approach for gathering evidence. Direct testing is particularly useful where the auditor wants to assess the extent of some event or characteristic in the population, to quantify the effects of a deficiency. Where the auditor wishes to project the results of tests as a generalization of the whole population, formal sampling techniques can be used. If the auditor does not have a strong background in sampling techniques, expert advice can be sought.

The examination stage

4.69 The purpose of the examination stage is to gather sufficient appropriate audit evidence to allow the auditor to support all of the statements made in the audit report. (Note: In reference to audit evidence, the word `sufficient' has the meaning of `enough'.)

4.70 The audit team designs audit tests and procedures to obtain evidence in the most cost-effective manner. The examination plan sets out the tests and procedures. The examination plan provides:

• a guide for conducting and co-ordinating the work of the examination stage;
• a framework for assigning work and assessing and establishing budgets for the remainder of the audit;
• a basis for supervising work; and
• a means for transferring knowledge to junior staff.

4.71 As noted earlier in this manual, performance auditing can be described as an iterative decision-making process. The gathering of evidence is in line with this overall process. The auditor gathers information, evaluates it for its appropriateness, and determines if it is sufficient to support observations about entity performance, conclude against audit objectives and make useful recommendations. If not, additional evidence may need to be gathered.

4.72 The evidence-gathering process involves the following steps:

1. designing the audit procedures or tests (examination plan);
2. carrying out the audit procedures or tests/gathering evidence;
3. analyzing evidence and drawing conclusions which may also involve evaluating performance against the audit criteria; and
4. making decisions about whether additional information is required and can be obtained (go back to step 1) or whether sufficient appropriate evidence exists.

4.73 It is not unusual for audits to be redesigned during the examination stage as teams encounter unforeseen difficulties in gathering sufficient evidence of appropriate quality. Auditors have to be alert to any signs that the evidence-gathering process may not be achieving the level of assurance required for the performance audit assignment and take appropriate corrective action.

Audit evidence

Audits should have sufficient appropriate evidence to support the contents of the audit report.

4.74 Evidence is information that is collected and used to provide a factual basis for developing observations and concluding against audit objectives. Evidence provides grounds for believing that a particular thing is true or not by providing persuasive support for a fact or a point in question. As such, it is evidence that must support the contents of an audit report, including any descriptive material and, more importantly, all observations and conclusions leading to recommendations.

4.74.1 Audit observations, conclusions and recommendations included in the report must be able to withstand critical examination. They must, therefore, be supported by sufficient appropriate evidence. In determining whether evidence of sufficient quantity and appropriate quality has been gathered, the auditor needs to be satisfied that in their professional judgment there is no risk of making erroneous observations, faulty conclusions, or inappropriate recommendations.

4.74.2 Although decisions about whether there is sufficient appropriate evidence are ultimately matters for the auditor's professional judgment, there are several considerations to bear in mind in making these decisions.

Appropriate evidence

4.75 For evidence to be appropriate, the information must be relevant, reliable and valid.

• Relevance refers to the extent to which the information bears a clear and logical relationship to the audit criteria and objectives. For example, when determining whether small and medium-sized enterprises have easy access to a specific government program, gathering data from large enterprises would likely yield irrelevant information. If information is not relevant, it cannot be evidence.
• Reliability concerns whether there is a likelihood of coming up with the same answers when either the audit test is repeated or information is obtained from different sources. This means that a measurement or evidence gathering process is more reliable when repeated measures or performances of the process produce the same result or a consistent result that is minimally affected by measurement errors (random distribution of measurement errors).
• Validity has to do with whether the information actually is what it purports to be in relation to content, origin and timing. An audit rarely involves the authentication of documentation/information, nor is the auditor trained as or expected to be an expert in such authentication. However, the auditor has to consider the validity of the information to be used as audit evidence, for example, photocopies, facsimiles, filmed, digitized (scanned) or other electronic documents, including consideration of controls over their preparation where relevant.

4.75.1 The following rules of thumb have proven helpful in judging the appropriateness of evidence:

• documentary evidence is usually better than testimonial evidence;
• audit evidence is more reliable when the auditor obtains consistent evidence from difference sources or of a different nature (e.g. testimonial evidence that is corroborated by other sources is better than testimonial evidence alone);
• original documents are better than photocopies;
• evidence from credible third parties may be better than evidence generated within the audited organization;
• the quality of information generated by the audited organization is directly related to the strength of the organization's internal controls (the auditors should have a good understanding of internal controls as they relate to the objectives of the audit); and
• evidence generated through the auditor's direct observation, inspection and computation is usually better than evidence obtained indirectly.

Sufficient evidence

4.76 The concepts of sufficient (quantity) and appropriate (quality) in relation to evidence are interrelated. The quantity of evidence is sufficient if when taken as a whole its weight is adequate to provide persuasive support for the contents of the audit report. In exercising professional judgment, auditors should ask themselves whether the collective weight of the evidence that exists would be enough to persuade a reasonable person that the observations and conclusions are valid, and that the recommendations are appropriate. Important factors to consider in making these judgments include:

• the quality of the evidence (its relevance, reliability and validity);
• the level of materiality (in dollar terms) or the significance of the observation or conclusion (in general, the higher the level of significance or materiality, the higher the standard that evidence will have to meet);
• whether an audit level of assurance (high) or a review level of assurance (moderate) is required (for example, a higher level of assurance is required for evidence to support observations than is required to support contextual information included in the report);
• the risk involved in making an incorrect observation or reaching an invalid conclusion (as an example, if any risk of legal action against the auditee results from reporting an observation, the standard of evidence demanded will be high); and
• the cost of obtaining additional evidence relative to likely benefits in terms of supporting observations and conclusions (as in most things, diminishing returns apply in gathering audit evidence -- at some point, incurring the cost of obtaining more evidence will not be justified by changes in the persuasiveness of the total body of evidence).

4.76.1 It is often the case in performance audits that important `facts' are not singular but instead are made up of a collection of interrelated facts. In reaching a conclusion, the auditor has to take into account that the strength of the `collection' may be as important as the strength of the individual facts for the assessment of the quality and quantity of evidence.

4.77 A challenge that auditors frequently face is how to provide sufficient appropriate evidence that something does not exist (for example, that the audit entity provides no training to a particular category of staff). Not finding something begs the question of where and how hard one has looked. In these circumstances it is particularly important for auditors to use multiple sources of evidence, that is, to corroborate, and to document their approach to looking for the evidence. In the example above, testimonial evidence by some staff that they had never received training would not constitute sufficient appropriate evidence that the entity provides no training. However, confirming such testimony by consulting the training records of all staff in the category and interviewing human resources professionals in the entity would provide much stronger evidence.

4.77.1 In this instance and in others, "clearance" of the audit report by the auditee does not replace the need for sufficient appropriate evidence. Such evidence must be on hand before the audit report is drafted, so that the report's observations, conclusions and recommendations are evidence-based. The purpose of sending a draft report to the auditee is to obtain confirmation (not evidence) that the facts in the report are accurate, and that the report presents a fair perspective.

4.77.2 If, despite best efforts, an audit is unable to meet the standard of obtaining sufficient appropriate evidence, a limitation exists on its scope. In these circumstances, the available evidence and its limitations may be reported, but observations and conclusions should not be drawn from the evidence. If the Office decides to report the matter, it would be reported as a qualification to the conclusion that a certain part of the subject matter can not be evaluated due to lack of evidence. When, in the judgement of the Office, a `qualification' is not sufficient due the significance and extent of the limitation in the evidence, the audit report will express a denial of conclusion. A denial states that a conclusion can not be made on the subject matter.

Documentation of evidence

4.77.3 One of the OAG's Performance Audit conduct policies requires the audit team to maintain appropriate documentation and files.

4.77.4 The documentation of evidence is a vital aspect of performance auditing and it should be completed before the DM transmission draft is issued. Good documentation of evidence helps ensure that:

• an adequate and defensible basis exists for the audit's observations, conclusions and recommendations;
• the observations, conclusions and recommendations can be explained in response to internal or external enquiries (for example, at a hearing of the Standing Committee on Public Accounts);
• an effective link exists between successive audits; and
• an appropriate basis exists for quality control in carrying out an audit and for subsequent third-party reviews.

4.77.5 In so far as evidence is concerned, audit files and working papers should contain information about the approach and work undertaken to achieve the audit objectives. The key documentation covering the entire audit process includes:

1. The Executive Committee's approval of the selection of the audit.
2. The survey report with footnote references to the relevant analysis and documentation supporting the report.
3. The examination plan that outlines the audit objectives and criteria, and a description of the audit and methodology (how evidence will be collected), including opportunities for quantification.
4. When necessary, an explanation of the major deviations from the original examination plan as approved by the AAG and brought to the attention of the Performance Audit Management Committee.
5. The examination report (internal draft) presented to the Advisory Committee on completion of the examination phase for advice on the proposed observations, conclusions and recommendations. In presenting the report, the audit team identifies those points for which the team has gathered sufficient appropriate evidence and those points for which the team is still gathering evidence.
6. The audit report sets out the logic between the criteria, the analysis that was carried out, the observations that were made, and how these led to conclusions against the audit objectives. Supporting the audit report are the substantiation binders that contain extracts of the most pertinent and persuasive evidence for the audit report (chapter) content. (Further guidance on substantiation is provided in Chapter 3.)

Beyond these requirements, the auditor needs only prepare a short summary for each audit objective explaining how the methodology was employed, the nature and extent of evidence collected, and the analyses to which it was subjected.

4.78 Auditors need to exercise professional judgment in documenting evidence, but a guiding principle in exercising this judgment is that the audit files and working papers must either include the evidence, or such descriptions of the evidence examined as to allow audit team managers and other to examine all of the evidence that supports the audit report and to come to the same conclusions as the auditors. In addition to being complete, accurate and clear, the files and working papers containing the evidence need to be structured in a logical way to provide for ready access to the audit evidence.

4.78.1 Although the documentation will usually include most of the evidence itself, it is not always necessary to copy and file every document examined or to list detailed information from all such documents. For example, when evidence includes the auditee's records, it may be enough to note that a particular document has been examined and provide the information required to support the identification and location of the document (e.g., file number, date, location) should a subsequent need arise to gain access to it. In carrying out its audit activities, the Office is required to comply with the National Archives of Canada Act in its management of records. Further guidance on maintaining audit files and working papers is available in the "Guidance for Managing Audit Records in the OAG".

Sources and types of evidence

4.79 There are three broad sources for the information that constitutes audit evidence.

• Information gathered by the auditors (primary evidence). Information can be gathered by the auditors directly by such means as interviews, surveys and direct inspection or observation. In these cases the auditors themselves have control over the methods employed and the quality of the information gathered. However, it must be emphasized that the auditors need the necessary skills and experience to apply the methods competently.
  
  
• Information gathered by the auditee (secondary evidence). Auditors can use information gathered by the auditee — including the reports of internal audit and program evaluation groups, as well as information found in other auditee files, databases, reports and documents. Auditors should determine the quality of this information by evaluation and corroboration as well as by tests of the effectiveness of the auditee's internal controls over the quality of the information. Auditors can reduce tests of information quality if they find that the auditee's internal controls are effective.
  
  
• Information gathered by third parties (secondary evidence). Audit evidence can also include information gathered by third parties. In some cases this information may have been audited by others, or the auditors may be able to audit the information themselves. In some cases third-party information cannot be audited, but its quality will be known (for example, many performance audits make use of data available from Statistics Canada). In still other cases, establishing the quality of third-party information may be impractical or impossible. The extent to which third-party information can be used as evidence will depend on the extent to which its quality can be established.

Further guidance on the use of secondary evidence from audit entities and third parties can be found in "Using Secondary Evidence Information in Performance Audits - A Summary Guide". (March 2006)

4.79.1 Audit evidence derived from the above sources can take a variety of forms. It may be:

• Physical — typically obtained by the auditor's direct inspection or observation, and supported by field notes, photographs or videotapes wherever possible. Examples include observing processes such as customs inspections or fisheries patrols, and inspecting assets to establish their existence and condition. An inherent risk of observation is that the observer's presence may alter what occurs in the setting, and as a consequence the evidence collected can be less valid. Thus the observer should aim to disturb the setting as little as possible. However, in some cases ad hoc visits to perform physical inspection or to observe certain conditions may be warranted and beneficial. (October 2004)
  
  
• Documentary — obtained from sources such as files, performance reports, databases, minutes of meetings, organization charts and correspondence. Documentary evidence can be obtained from the auditee or from third-party sources, and includes both electronic and hard-copy information.
  
  
• Analytical — obtained by the auditor by manipulating other types of evidence through analytical techniques such as computations, comparisons and content analysis of qualitative data. Examples of such techniques include comparisons using ratios, regression analysis of quantitative data, coding and systematic analysis of qualitative data, chronology analysis, and benchmarking against relevant standards.
  
  
• Testimonial — obtained from others through their oral or written statements in response to enquiries made by the auditor. Examples include interviews with staff of the auditee and surveys (either telephone or mail) of clients of a program or service.
  
  Enquiry has always been one of the significant techniques used in undertaking an audit. Careful preparation and briefing beforehand and debriefing and documentation afterwards improve the interview's effectiveness. How auditors structure enquiries depends on the circumstances (what kind of information is sought, who and how many are being asked, etc.). Every question should make sense to the interviewee, use vocabulary that is common to the interviewee and the auditor, and elicit the information required. A sound knowledge of the business and of the particular area of responsibility of the interviewee adds to both the preparation for an interview and to the interpretation of the responses as to their reliability and significance. Evidence from individual interviews should be corroborated with evidence from other people or other sources, whenever it is appropriate.
  
  Sharing the results of the interviews among the audit team is crucial. Good documentation, presented in an easily digestible and accessible form, can add significantly to the efficiency and effectiveness of the audit evidence from enquiry.
  
  The audit team should prepare formal minutes for all meetings involving entity staff that the audit team intends to rely on for evidence purposes.
  
  It is a matter of judgment and the responsibility of the PX to provide guidance to the audit team as to whether or not a particular meeting will require sign-off of minutes for evidence purposes. However, it is recommended that the audit team, at the beginning of the audit, informs the auditee that the record of certain meetings might be used as audit evidence and the approval of minutes by the entity will be necessary.
  
  In particular, if the meetings occur as part of the audit process (planning, examination or reporting phases) and is considered evidence, the minutes of meetings should be prepared, reviewed and sent to the senior entity participants at the meeting for sign-off within five working days of the meeting. The following approach will allow the audit teams to:
  
  
  • raise the quality of the evidence, particularly in those instances where no other source is available;
  • provide a written record of what documents have been received, provided, and promised, which will permit teams to better monitor the receipt of information; and
  • provide the interviewees with confirmation that their statements have been understood and recorded correctly.

  The audit teams should use the following suggested format for notes and/or minutes.

  • The date and location of the meeting.
  • The date of minute preparation.
  • The approximate start and end times.
  • The names and positions of each of the attendees.
  • A summary of the purpose of the meeting.
  • The questions asked and the answers to them.
  • A list of actions agreed to at the meeting.
  • A list of all documents identified, offered, or requested at the meeting.
  • Signature of the senior entity representative.
  • Signature of the senior OAG participant at the meeting and other OAG attendees if necessary.

  Before being sent to the entity participants, the minutes should be reviewed and agreed to by the senior OAG participants at the meeting.
  
  The audit team should ensure that the entity participants do not unduly delay sign-off of the minutes. In cases where sign-off delays or refusals have been encountered, the matter should be referred to the PX who will discuss the matter with the entity liaison officer. If the matter still remains unresolved, it should be referred to the AAG who will consider raising the matter with the organization.
  
  At times, a meeting which was not judged as providing evidence may subsequently become important for evidence purposes. In this case, the senior OAG participant at the meeting should as quickly as possible formalize the minutes and seek their clearance by the entity.
  
  Verbal assents or telephone conversations do not constitute a valid sign-off. The audit teams need to obtain a signature from the senior entity participant on the minutes submitted or use the e-mail system by following the E-mail Use Policy and Guidelines. (Testimonial: October 2004)

4.79.2 Although evidence analysis follows evidence capture in chronological terms, the audit team members need to know what specific analytical techniques they will use before they start to design their strategy for capturing evidence. Otherwise, the auditors may find that the evidence collected is not susceptible to the appropriate forms of analysis. The audit team should use computer-assisted techniques for gathering and analyzing evidence whenever their use will increase the efficiency or quality of the audit.

4.80 When gathering information during the examination stage, the auditor thinks forward to the reporting stage, and the need to communicate the audit message in a persuasive manner. The auditor needs to look for opportunities to use case studies or visual aids, such as photographs, as these often provide a convincing way to illustrate an issue in the audit report.

4.80.1 The Office emphasizes the importance of quantification as a means to demonstrate the significance of its observations and recommendations. Experience has shown that a focus on quantification has to be built in at the planning stage.

Relying On and Using The Work Of Others

4.81 In the interests of audit efficiency, auditors should rely on the work of internal audit whenever possible in areas relevant to the audit. When the work of internal audit is the main or sole evidentiary support for particular observations, conclusions and recommendations, auditors should evaluate and corroborate the specific internal audit work on which they intend to rely. The purpose will be to determine whether the work meets the Office's standards with respect to sufficient appropriate evidence, such that an adequate basis for reliance exists.

4.81.1 Auditors can determine the quality of internal audit work by assessing the reputation, qualifications and independence of the internal audit team, as well as by reviewing audit reports, audit programs and audit working papers. The nature and extent of the evaluation and corroboration will depend on the significance of the internal audit work in relation to the Office's audit objectives and the extent to which the auditors will rely on it.

4.82 Where auditors use the work of internal audit (e.g., by including reference to findings from internal audits in the Auditor General's Report) the audit team should evaluate and corroborate the supporting evidence to assure the validity of the findings. Normally, when such matters are included in the report, the source of the findings is clearly indicated.

4.82.1 With the increasing emphasis on results-focused performance audit, as well as results-focused management across government, there is a growing potential for auditors to use the data and findings generated by departmental program evaluation and performance measurement groups.

4.82.2 As in the case of reliance on internal audit, auditors should evaluate and corroborate the work of program evaluation and performance measurement specialists (as well as others — such as experts on scientific and technical matters) to determine the appropriateness of audit evidence obtained from these sources. This assessment will be based on such factors as the knowledge, experience, professional standing and independence of the professionals concerned. Where appropriate, auditors will also need to have a good understanding of the standards, methods, data sources and significant assumptions that have been used by these professionals.

4.82.3 Other specialists whose work is relied upon may be members of the Office staff or contractors used by the Office to provide expertise in an area relevant to the audit. The specialists who are members of the audit team or of other groups in the Office are covered by the requirements for audit teams in Chapters 2 and 3 of the Manual in relation to objectivity, independence, knowledge, competence, supervision and review of work. For the individuals contracted by the Office, the audit team should assess their knowledge, competence, integrity and independence in the relevant area of expertise, and evaluate and corroborate the reasonableness and significance of the specialist's work and findings for the audit.

Developing audit observations

Audits should involve objective evaluation of the evidence against the criteria to develop observations and conclusions.

4.83 The audit team gathers evidence in order to support a description of an activity or program under review and make an assessment of the actual performance of an activity or process against the audit criteria. Where the auditor finds that performance does not meet the criteria, further investigation should be carried out to gain assurance that any resulting observations and conclusions are significant, fair and well-founded, and that recommendations have the potential to result in important improvements to performance, value for money or accountability.

4.83.1 Gathering additional evidence and/or discussing the matter with auditee management, may be necessary to:

• determine whether the deficiency is an isolated instance or represents a generic or systemic problem;
• assess the impact or potential impact of the deficiency on results. Whenever possible, the effect of the problem should be quantified to illustrate the "so what" in the audit report;
• identify the cause of the deficiency to gain assurance that recommendations will be appropriate;
• determine whether the problem can be fixed by the auditee, or whether it results from circumstances beyond its control;
• gather further evidence (cases, statistics, photographs, etc.), where appropriate, to illustrate the nature and importance of the issue;
• determine who is affected by the issue (for example, other units in the organization, central agencies, third parties); and
• determine auditee management's awareness of the issue. If management is aware of the issue and has corrective action under way, the issue may have less significance for reporting purposes. Certainly it will change how the matter is reported.

4.84 The comparison of evidence against criteria, and further investigative work into the nature and significance of the issue, will result in the development of observations. Audit observations confirm satisfactory performance or disclose the level, nature and significance of deviations from criteria, who is responsible, and may disclose the cause and effect of the problem. In reaching their decisions on observations, auditors may need to look at the collection of interrelated facts and evidence assessing them against the corresponding criteria as well as considering them individually.

4.85 The observations, in turn, are the basis for forming conclusions against each of the audit objectives. The auditor should assess the significance of the observation in relation to the audit objectives. In concluding against an audit objective the auditor will use their professional judgment. The audit conclusions and the major considerations and rationale for them are approved by the AAG/CESD and reviewed by the Audit Advisory Committee.

Developing recommendations (May 2005)

Audits should include recommendations to point to the direction in which positive changes can be made for the most serious deficiencies reported. Recommendations are not required for each audit finding. (May not apply in all audit notes).

4.86 Recommendations generally relate to strategic issues. Recommendations address areas where there are significant risks to the entity if deficiencies remain uncorrected. Audits include recommendations where the potential for significant improvement in operations and performance is demonstrated by the report findings. The most serious deficiencies, not each audit finding, are to be addressed.

There may be circumstances where making a recommendation is not the best way to achieve the intended result. In those circumstances, exceptions to the audit policy should be justified on a case-by-case basis and approved by the AAG. The audit can still make a major contribution in such circumstances by bringing a highly professional analysis of the situation to the attention of the audited entity and Parliament. Where corrective action is underway, it is good practice to point out that such actions are underway.

4.87 Writing a good recommendation is not an easy task. Good recommendations meet several criteria. The criteria relate to the audit process, the format, and to the substance of the recommendation.

Recommendations that lend themselves to follow-up are:

• fully supported by and flow from the associated observations and conclusions;
• aimed at correcting the underlying causes of the deficiency;
• addressed to the organization with the responsibility to act on them.

Clear recommendations are:

• succinct, straightforward and contain enough detail to make sense on their own;
• broadly-stated (i.e. stating what needs to be done while leaving the specifics of how to entity officials);
• positive in tone and content.

Action-oriented recommendations are:

• presented in the active voice;
• practical (i.e. able to be implemented in a reasonable timeframe, taking into account legal and other constraints);
• cost-effective (i.e. the costs of implementing them will not outweigh the benefits), and they will not increase the bureaucratic burden;
• results-oriented (giving some indication of what the intended outcome is, ideally in measurable terms);
• able to be followed-up (i.e. able to determine whether it has been acted upon);
• consistent and coherent with the other recommendations in the chapter and mindful of recommendations made in prior chapters.

4.88 To enable the auditors to develop action-oriented and practical recommendations and to provide entity officials with the time required to prepare a response and develop an action plan, the audit team should seek management's views, as early as possible, normally at the end of the examination phase. There should be consultation with entity representatives as to the risks they are facing and managing. Entity officials, including the Deputy Head, should be briefed on recommendations.

Recommendations should be included as an agenda item for the advisory committee meeting held at the end of the examination phase.

In preparing to consult entity officials and the audit advisory committee members, the audit team should brief the Assistant Auditor General. To that end, the team could document in a working paper on each serious deficiency identified, the criteria applied, the observations made and the identified causes that lead to the recommendation. For each proposed recommendation, the audit team needs to consider the effect that offering such a recommendation may have on the auditor's objectivity in subsequent audits of the same entity.

4.89 An area of high sensitivity is a recommendation for changes to legislation. If observations are pointing to the need for changes to legislation, the matter should be discussed with Legal Services.

Entity responses to recommendations

4.90 We encourage and will publish responses to each recommendation in a chapter, indicating whether there is:

• agreement with the recommendation and a commitment to undertake action;
• agreement with the recommendation and an explanation as to why action cannot be taken at this time; or
• disagreement, with a brief explanation.

The auditor's may point to the direction in which positive changes can be made; however, detailed plans and implementation of changes are the responsibility of management. Entities should be encouraged to include a timetable and specific steps to describe how the recommendation will be addressed.

4.91 When recommendations are made to entities other than those for which the audit Principal is responsible for, he/she should advise the entity Principal of recommendations that apply to their entity.

The responses provide the Office and the Public Accounts Committee with a basis for follow-up of the audit.

4.92 The response is typically provided to us by a Deputy Head or delegate who is acting on behalf of a Deputy. When more than one entity has been the subject of the audit, one joint response, or multiple responses with each entity being clearly identified are acceptable. In general, government-wide recommendations are to be avoided. However, if it is not possible, a lead department or agency is to be identified and that role should be accepted by the entity. The recommendation should specify what entity will lead the implementation. Such recommendations can then be responded to by the lead organization on behalf of the government.

4.93 We have established limits on the content and publication of entity responses, and entities should be encouraged to comply with the following:

• Responses are to be short and clear, normally no more than two paragraphs. Where appropriate, we will publish an overall entity action plan that responds to our observations and recommendations.
• We do not normally publish entity responses when there are no recommendations or when the audit is a follow-up of previous work and there are no new recommendations.
• Responses must be received at least seven weeks before tabling day in order to be published with the report. We also need the entity response seven weeks prior to tabling in order to meet commitments to brief the appropriate officials of central agencies.
• We do not print entity responses or comments in the Main Points or throughout the chapter. Nevertheless, we do briefly describe the entity commitment (or non-commitment) to take action in the Main Points.
• We discourage global comments as a regular feature of entity responses.

4.94 Audit teams should ensure entity officials are aware of the limitations to responses to recommendations, and encourage them to comply. If exceptions to these limits are requested, they are to be discussed with the AAG. We may, from time to time, wish to include a global response to a study to make the government position available to the reader. Also, entities may wish to publish an action plan to correct the deficiencies noted in the report. This would be acceptable if it assists the accountability or provides more information about the benefits to be achieved by the recommendations, and are limited to one page. The final decision on an entity response in these instances rests with the Office and must be approved by the Performance Audit Management Committee (PAMC).

4.95 The Principal's draft chapter should be presented to the entity as required by the agreed schedule. In the case of audit notes and follow-up, this time frame may be shortened but sufficient time must be given to the entity to consider and respond to the issues.

4.96 The audit report stands on its own merit. We do not respond to the comments of the entity in the report. However, we will not publish an entity response or comment that we know is materially wrong or misleading. Where we disagree with an entity position, we will make our position clear in subsequent Public Accounts Committee hearings. If there is substantial disagreement between the entity and the auditor we will highlight this in the Main Points of the chapter.

4.97 DELETED

Audit conclusions

Audits should have necessary and sufficient observations to support conclusions made against each audit objective.

4.98 The process of dividing the audit into component parts does not obscure the need to conclude in relation to the overall audit objectives. Planning decisions have identified lines of inquiry for the audit. Audit evidence has been gathered and performance in the critical areas has been assessed against criteria. Actual performance has been found to be satisfactory or deviations from the criteria have been identified. Further investigations of the deviations from satisfactory results or good practice have led to the development of observations.

4.99 Audit observations confirm satisfactory performance or disclose the level, nature, and significance of deviations from criteria, who is responsible, and the cause and, if determinable, effect of the problem on the subject matter of the audit.

4.100 The auditor should assess the significance of the observations in relation to the audit objectives. At the extreme ends of the performance spectrum — fully satisfactory performance or highly unsatisfactory performance — concluding against the overall objective may not pose a problem. In these cases the audit report would contain an unqualified (positive) conclusion or an adverse conclusion, respectively. An adverse conclusion is used when the significance and extent of the deviations from satisfactory performance are pervasive. In the majority of cases, however, the auditor will have to use judgment in forming a qualified conclusion. Qualified conclusions are made when there are significant deviations from satisfactory performance for one or more aspects of the subject matter. A qualified conclusion contains an "except for" statement, either stated explicitly or implicitly, to disclose the deviations in relation to the audit objectives.

4.100.1 The audit conclusions and the major considerations and rationale for the conclusions are reviewed with the AAG/CESD and the Audit Advisory Committee.

The audit report

Audits should result in a report that meets the Office's Reporting Policies.

4.101 Having completed the field audit work, developed the audit observations, and concluded against each audit objective, the auditor is in a position to draft a report that must meet the performance audit reporting requirements.