Performance Audit Manual

Selecting the right area for audit

2.1 The starting point in the performance audit planning process is deciding what to audit from the myriad of government activities. This is a complex and challenging exercise that requires good knowledge of the entity's business or sector of activity and a high level of professional judgment. It is, however, one of the most important steps in the process, if we are to meet the requirements of our mandate cost-effectively and make a difference for Canadians. If the selection of audit topics is not done well, all the audit work that follows will have little chance of producing satisfactory results.

2.2 The Office's Strategic Plan sets out a vision that provides an important starting point in deciding what to audit. As well, our annual Report on Plans and Priorities sets out the strategic or end outcomes that we are trying to achieve; in general, better managed government programs and better accountability to Parliament and the public, and, specifically:

• public confidence in government institutions,
• good governance and an accountable government,
• progress toward sustainable development,
• effective and efficient programs, and
• credible performance reporting by departments to Parliament.

2.3 The planning process involves several layers of activity that interact in a complex manner before an audit begins. These include the identification of external trends and business risks that apply to all audits; the defining of entities, functional areas and sectors to be examined over time; and the choice of entities, programs or activities to be examined. The audit selection process is driven by three criteria: the significant risks associated with the area; relevance of proposed audit activities to the Auditor General's focus areas and mandate; and auditability.

2.4 Risk-based audit planning focuses on how well an entity is managing major risks rather than simply focusing on areas of suspected weaknesses. It begins by establishing a good understanding of entity objectives, expected results, and stewardship responsibilities. Teams then identify the important factors that may affect the entity, or entities within a functional area, and assess how well the entity is responding to key challenges, opportunities and critical success factors that affect the achievement of objectives and the fulfillment of stewardship responsibilities for public funds and assets. Significant risk includes the notions of materiality, importance to the achievement of government results, and current parliamentary or public interest.

2.5 Relevance addresses whether the area being considered advances the Auditor General's five focus areas and falls within the mandate of the Office. Areas that would fall outside of our mandate are political policy decisions and the administration of programs delivered by other levels of government.

2.6 Auditability defines whether the area is amenable to audit. Auditability risks include:

• Availability of adequately qualified internal and external resources
• High political sensitivity
• Complexity of the proposed area
• Experience of the audit team

The Office planning process

2.7 The Office has established an extensive process of consultation, analysis and planning in order to ensure that relevant matters of significant risk are audited in a timely fashion and that the requirements of the mandate are met. It is not possible to audit every aspect of government, and a process is needed to select the areas of greatest risk. The following is a brief outline of this process.

2.7.1 The planning process or cycle starts with periodic environmental scanning to identify external trends and long-term risks and challenges that the Office may face. Throughout the year, groups identify proposed audits through one-pass planning and other risk-based analyses. As one-pass plans are completed for entities and functional areas, the results are presented to Executive Committee for review and discussion. Each spring, all of the proposed audits are consolidated and Executive Committee approves those that will be reported over the next 12-18 months and agrees on a schedule of planned audits for the next several years. In the fall, the operational plan for the upcoming fiscal year is approved. This plan provides for the financial and people resources required to carry out the next year's audits.

2.7.2 The key components of the planning process, as well as the role of the Executive Committee, the Strategic Planning and Professional Practices group (SPPP), and the Comptroller are outlined below. The Executive Committee has overall responsibility for audit planning and is supported in this role by SPPP and the Comptroller.

One-pass plans

2.7.3 One-pass plans provide a systematic, integrated and risk-based approach to long-range audit planning. These plans differ from our previous performance audit-specific planning in several important ways:

• There is greater emphasis on business risks that are critical to the success of the organization or functional area, and how they are being managed, as the basis for selecting audits.
• Risks are integrated for all mandate areas, not just for performance audits.
• The one-pass planning process is consistently applied from team to team.

2.7.4 One-pass plans ensure that available resources are focused on the areas of greatest risk. They provide assurance to the Auditor General and to Parliament that the Office is exercising due diligence in applying the discretion provided in the Auditor General Act section 7 for selecting matters for audit. The elements of the one-pass planning process are:

• Identify entity/functional objectives and stewardship responsibilities
• Prepare the entity risk profile or challenges/risks to achieving functional objectives
• Prepare the entity control profile (may not be applicable for functional areas)
• Align risks with Office mandate and focus areas
• Identify potential audits and assign priorities
• Assess gaps in audit coverage
• Report to Executive Committee

Auditor General's focus areas

2.7.5 The Auditor General identified five areas that she wishes to focus on during her term. These focus areas form an integral part of our Strategic Plan. They will assist us to both plan and report on the results of our audit work. The focus areas are:

• Accountability to Parliament
• Effective public service
• Aboriginal issues
• Well-being of Canadians — health, safety, environment, and social and national security
• Legacy and heritage

Executive Committee spring audit selection meeting

• At the spring meeting, the Executive Committee rationalizes proposed audits with other work and the resources available (financial and people) on an Office-wide basis to achieve a balanced performance audit program and consensus for the next year's work.
• A schedule of planned audits for future years is also developed with a view toward advancing the Auditor General's focus areas in a satisfactory manner.
• The input for this meeting comes primarily from focus area plans, one-pass plans and other risk-based analyses of entities and functional areas that have been prepared.

Executive Committee fall operational planning meeting

• Each fall, led by the Comptroller and SPPP, the Executive Committee meets to update audit plans and to consider and approve the budget for the upcoming fiscal year.
• The initial funding for performance audits that will be tabled in the next fiscal year, and April and May of the following year, is approved.
• Following this approval, individual budgets for the audits are reviewed and confirmed or revised at the end of the survey phase and any necessary reallocations take place at that time.

Executive Committee

2.7.6 The Committee's responsibilities are to:

• review one-pass plans for entities and functional areas (proposed audits are approved at the spring audit selection meeting);
• establish a reporting strategy over time by scheduling the number of chapters, the length of the chapters and by co-ordinating and grouping report themes;
• approve individual audits, review audit summaries (synopsis of approved examination plans), track chapter progress, and provide a forum to discuss concerns; and
• resolve report issues and issues arising between audits.

Strategic Planning and Professional Practices group

2.7.7 SPPP supports the performance audit planning process by:

• co-ordinating environmental scanning work;
• scheduling presentation of one-pass plans to the Executive Committee;
• organizing the spring audit selection meeting including preparation of supporting material such as risk analyses and proposed audits; and
• maintaining one-pass planning methodology for risk analysis and reporting.

Comptroller

2.7.8 The Comptroller supports the performance audit planning process by:

• preparing budget information for the Executive Committee spring audit selection meeting, in particular, performance audit capacity projections for the planning year and two subsequent years;
• presenting an operational plan and budget for approval at the Executive Committee fall operational planning meeting; and
• maintaining the chapter-tracker database and presenting regular reports to Executive Committee on the status of approved performance audits.

Types of performance audits

2.8 The Office has designed a number of performance audit approaches in order to make the audit products more relevant to Parliament. Government activities and projects often cross departmental lines. Reporting on the activity or project as a whole is normally more useful than commenting on a segment carried out by a specific entity. The types of performance audits are:

• entity or program audits, which provide a substantive review of the whole or part of the operations of a department or agency;
• government-wide audits, which focus on government-wide issues or functional areas, such as human resource management, in a number of departments selected by the Office;
• sectoral audits, which focus on program areas delivered by a number of entities, for example, search and rescue operations;
• audit notes, also called other audit observations, are an alternative mechanism for reporting matters of significance that come to our attention during the course of our work on any Office products. Audit notes usually report on a single subject which must be within the mandate of the Office; and
• follow-up audits, which report on government actions in response to previous recommendations and issues of the Office and that are of continued interest to Parliament, and/or that pose a significant risk. (May 2005)

Roles and responsibilities of the key players

2.9 Many groups and individuals in the Office contribute to the cost-effective completion of a performance audit and a high-quality audit report. They provide expert advice, guidance, legal counsel, challenge and review, methodology, high-technology audit tools, and assistance in editing, translation and presenting the report. Their roles and inputs are noted throughout various sections of this manual.

2.10 The final audit chapter is the result of the joint effort of all these individuals. This section briefly sets out the roles and responsibilities for a typical audit engagement of the audit Principal (team leader), team directors, team members, the AAG/CESD and the Quality Reviewer.

The audit Principal

2.11 The audit Principal has overall responsibility for auditing the entity, managing the entire audit cycle and a team of auditors, and ensuring the quality of audit products produced by the team. The responsibilities include:

• maintaining an adequate team knowledge of the organization(s) or function;
• maintaining effective departmental relations;
• managing all aspects of the audits addressing her/his entities, and co-ordinating with other teams on audits affecting their entities;
• leading the audit team, delegating responsibilities, monitoring progress and reviewing performance;
• managing budgets and timely completion of audits;
• seeking counsel and expert advice throughout the audit;
• reviewing draft audit reports and chapters;
• advising the AAG/CESD and the Executive Committee on progress of audits and emerging problems;
• involving the AAG/CESD, the Quality Reviewer and the audit advisory committee on all important audit matters and documenting decisions;
• providing assurance to the AAG/CESD on audit quality, including documenting her/his approach to ensuring: (1) the adequacy of the evidence to support major observations, conclusions and recommendations prior to issuing the PX Draft, and (2) the completion and review of the substantiation binder to support all observations, conclusions and recommendations prior to issuing the Transmission Draft chapter;
• ensuring compliance with all performance audit policies; and
• recommending to the AAG/CESD that the Principal's draft and Transmission draft be forwarded to the entity;
• recommending the chapter be approved for publication by signing the Final Approval Form.

Directors

2.12 Depending on their capabilities and experience, directors are delegated by the Principal (responsible for managing individual audits) some of his/her authority. Their responsibilities may include:

• initiating the audit planning process and developing detailed audit plans;
• determining audit objectives, identifying entity components significant to the overall audit objectives, defining an audit approach, determining criteria and documenting how the audit covers the risks identified in one-pass planning;
• carrying out overviews, surveys and audit examinations;
• preparing overview and survey reports, examination plan and chapter drafts;
• preparing briefing packages on the audit for advisory committees, the Performance Audit Management Committee, the Executive Committee, the Auditor General and others, as well as preparing accountability reports, press releases, and chapter communications strategy;
• supervising the work of auditors and other team members; and
• supervising the preparation of audit files, substantiation binders, and accountability and other documents, and documenting her/his review of the files/binders prior to the issuance of the Transmission draft chapter.

Team members

2.13 Carry out the responsibilities assigned to them by their supervisors. As such they are expected to support their supervisors in fulfilling their responsibilities, including:

• delivering quality products;
• being alert to possible non-compliance with policies and practice expectations;
• identifying other audit opportunities; and
• providing continuous improvement feedback on the Office's quality management.

The Assistant Auditor General / Commissioner of the Environment and Sustainable Development

2.14 The AAG/CESD oversees all aspects of the audit. The duties include:

• giving advice and counsel to the Principal and to the audit team;
• rationalizing Group workload and resources;
• being involved in major audit decisions on entity relations, scope of audit, access problems, complex and contentious issues, reporting strategy, reviewing and challenging the report chapter, reviewing related files as necessary, clearing the report with senior entity officials and documenting on the Final Approval Form his/her concurrence with the Principal's recommendation that the chapter be approved for publication.
• communicating expectations vis-à-vis quantification, and reviewing survey plans to ensure this is addressed;
• seeking the advice and input of the Quality Reviewer assigned to audits within her/his Group;
• providing signoff that any advice received from the Quality Reviewer was dealt with in a mutually satisfactory manner;
• chairing audit advisory committees;
• providing assurance to the Performance Audit Management Committee (PAMC) on audit quality;
• ensuring that all performance audit policies are followed; and
• approving that the Principal's draft and the Transmission draft be forwarded to the entity.

Quality Reviewer

2.15 A Quality Reviewer is named for each performance audit, study and audit note as soon as they are approved. Quality Reviewers provide an additional element of independence and objectivity in key risk areas: audit planning and reporting. In order to maintain their independence, Quality Reviewers provide advice but do not make decisions. The Quality Reviewer is a member of the audit Advisory Committee or Audit Notes Committee. S/he is not considered to be part of the audit team.

The Quality Reviewer provides advice on the following risk areas:

• significant audit risks identified by the audit team;
• completeness of the planning process;
• suitability of the criteria used for evaluating the subject matter;
• suitability of the audit approach, particularly in higher risk areas;
• sufficiency and appropriateness of evidence, particularly in relation to high risk findings;
• handling of contentious issues that may arise during the audit;
• nature and extent of consultation by the audit team;
• significance of any disagreements between the entity and the audit team relating to matters discussed in the chapter; and
• appropriateness of the conclusions.

These responsibilities are carried out primarily through discussion with the audit team and through review of selected working papers.

2.16 Quality reviews are carried out by an AAG/CESD or PX or, in the case of some audit notes, possibly a DX. The PX Practice Development, at the request of the Performance Audit Executive Working Group (PA-EWG), prepares a list of Quality Reviewers for upcoming reports. The list identifies for each chapter at least two possible candidates for consideration by the PA-EWG. The PA-EWG ensures that a quality reviewer is appointed for each audit and study and notifies the affected AAGs, PXs, QRs and Chapter Tracking of the assignments. The Audit Notes Committee ensures that a quality reviewer is appointed for each audit note. This is typically a member of the Audit Notes Committee. (May 2005)

In developing the list of potential Quality Reviewers, the PX Practice Development should take the following criteria in consideration:

• a Quality Reviewer is to be an AAG or PX with sufficient experience in conducting successful performance audits; and
• a Quality Reviewer is not to be assigned to more than one chapter per report. (May 2005)

In addition to these two criteria, key attributes of a quality reviewer are:

• familiarity with the subject matter or entity;
• independence and objectivity; and
• availability. (May 2005)

Performance Audit Management Committee (PAMC)

2.17 The Performance Audit Management Committee, on behalf of the Executive Committee, manages the Performance Audit process. The Committee monitors the implementation of plans, makes recommendations on significant changes, receives regular reports and provides approvals of key milestones in the Performance Audit process. The Committee carries out its reviews to ensure that the Quality Management Framework has been applied as spelled out in the Performance Audit Manual. (October 2004)

2.18 The PAMC reviews the Final Approval Form for performance audits and studies prepared by the Principal and reviewed by the line AAG/CESD and approves the chapter for publication in the Report of the Auditor General.