International Peer Review of the Office of the Auditor General of Canada—2010

International Peer Review of the Office of the Auditor General of Canada—2010

Executive Summary

The Office of the Auditor General of Canada

The Quality Management System

The Peer Review

The Peer Review Opinion

Key observations on the design of the Quality Management System

Key observations on the operation of the Quality Management System

Performance audit and special examination

Leadership and Planning
Audit Management
Client Focus
People Management
Continuous Improvement

Annual audit

Audit Management
Assessing risks
Audit documentation
People Management and Continuous Improvement

Suggestions for the OAG to consider

Continuous improvement
Performance audit and special examination reports
Reporting to management
Using electronic audit working papers

Good practice identified

Managing the relationship with Parliament
Delivering a clear message
Collaborative audits
Criteria and sub-criteria for special examinations

Appendices:

1—The Quality Management System (QMS)

2—About the Peer Review

3—The OAG’s response to the Peer Review Report

Executive Summary

1. At the request of the Auditor General of Canada, an international peer review team reviewed the Quality Management System (QMS) that the Office of the Auditor General of Canada (OAG) uses to manage its audit and assurance practice. The objective of the peer review was to provide an independent opinion on whether the OAG’s QMS was suitably designed and operating effectively to provide reasonable assurance that the work of the OAG complied with relevant legislative authorities and professional standards.

2. The period under review covered audit and assurance engagements reported during September 2008 to October 2009. The review criteria were based on relevant legislative authorities and Canadian assurance and auditing standards. The peer review team was led by the Australian National Audit Office and also included representatives from the supreme audit institutions of the Netherlands, Denmark, Sweden and Norway.

3. The peer review team found that, for the period under review, the QMS for performance audit, special examination and annual audit was suitably designed to provide reasonable assurance that the work of the OAG complied with relevant legislative authorities and professional standards. Also, the QMS was operating effectively for the performance audit and special examination practices. For the annual audit practice, the QMS was generally operating effectively, but there were implementation issues in relation to the following areas that need attention:

4. This report sets out the peer review team’s observations about the design and operation of the QMS as it applied to the OAG’s performance audit, special examination and annual audit practice. The report includes two recommendations to improve the completeness of the risk assessment procedures and the sufficiency of audit documentation. The OAG agreed to these recommendations.

5. Suggestions for the OAG to consider as it continues to improve its audit practices have also been included. In addition, the report describes good practices adopted by the OAG that will be of interest to other audit offices. These practices include the OAG’s approach to managing its relationship with Parliament, its effort to disseminate a clear and consistent message in its performance audit reports and its support for collaborative audits.

Ian McPhee
Auditor-General for Australia, on behalf of the peer review members.
18 May 2010

The Office of the Auditor General of Canada

1. The position of Auditor General of Canada was established in 1878. As the Canadian federal government’s legislative auditor, the Auditor General audits federal government departments and agencies, most Canadian Crown corporations and other federal organisations. The Auditor General is an independent Officer of the Parliament of Canada and reports directly to Parliament on matters that she believes should be brought to its attention. She is also the auditor for the three Territorial Governments1, reporting to their legislative assemblies. The Office of the Auditor General (OAG) supports the Auditor General in executing her responsibilities.

2. The Auditor General’s powers and responsibilities are prescribed by legislation, including the Auditor General Act (1985) and the Financial Administration Act (1985). The Auditor General conducts:

3. The Commissioner of the Environment and Sustainable Development assists the Auditor General to perform her legislated responsibilities relating to the environment and sustainable development. On behalf of the Auditor General, the Commissioner:

4. The OAG has three main practice lines: performance audit; special examination; and annual audit.

5. Performance audits are conducted under sections 5 and 7 of the Auditor General Act. These audits examine the Government’s activities and programs to determine whether they are being carried out with due regard to economy, efficiency and environmental impact, and whether there are measures in place to determine how effective they are. The Auditor General’s performance audit reports include examples of good practice, areas requiring attention and recommendations for improvement.

6. The Financial Administration Act outlines the requirements and timing of a special examination. The purpose of a special examination is to determine if a corporation’s management practices and its financial, management and information systems were maintained in a manner that provided reasonable assurance that, for the period under examination:

A special examination report provides an opinion on whether there is reasonable assurance that there are no significant deficiencies2 in the corporation’s systems and practices examined. Where appropriate, the report also includes recommendations for improvements.

7. Section 6 of the Auditor General Act mandates the Auditor General to examine any financial statements to be included in the public accounts3 and any other statement that the President of the Treasury Board or Minister of Finance may present for audit. The several financial statements of the public accounts are required by section 64 of the Financial Administration Act. The Auditor General’s mandate includes annual audits of departments, agencies, Crown corporations and other federal organisations.

8. Performance audits, special examinations and annual audits are reported as follows:

Report Type Reported To
Performance audit reports

(including reports produced by the Commissioner of the Environment and Sustainable Development for the Auditor General)

Parliament and the legislative assemblies of Canada’s territories.
Special examination reports The corporation’s board of directors, Ministers and Parliament as appropriate.

A list of the examinations completed and a summary of the main points found is presented to Parliament annually.
Annual audit of the Government’s financial statements Parliament.
Annual audits of Crown corporations The responsible Minister for subsequent tabling in Parliament.
Annual audits of other federal organisations As prescribed in relevant enabling legislation.

9. In 2008–09, the OAG completed 32 performance audits, eight special examinations and 108 annual audits.4 It employed 628 full-time equivalent staff to carry out this work.5 The reported cost of the OAG’s auditing operations was as follows:

Audit Type Actual Cost (CAD)
2008–09
Performance Audit

(including $2 million for sustainable development monitoring activities and environmental petitions)

$46 million
Special Examination $8 million
Annual Audit $38 million
Total $92 million
Source: Office of the Auditor General of Canada, 2008–09 Estimates Performance Report, OAG, September 2009, pp.39–40.

10. Each year the OAG reviews a selection of its audits and special examinations as part of its quality assurance process. The objective of these practice reviews is to evaluate whether the audits have adhered to regulatory and legal requirements, professional standards, and office policies.

11. Professional auditing standards are issued by the Auditing and Assurance Standards Board of the Canadian Institute of Chartered Accountants (CICA). They outline the requirements that apply when performing an assurance engagement (Standards for Assurance Engagements) or when auditing financial statements (Generally Accepted Auditing Standards). The quality control standards current at the time of this peer review were CICA’s GSF-QC General Standards of Quality Control for Firms Performing Assurance Engagements and Section 5030 Quality Control Procedures for Assurance Engagements. GSF-QC establishes a framework and provides guidance on the quality control policies and procedures applicable to firms or audit offices performing assurance engagements, while Section 5030 applies to individual engagements. These standards are adopted by the OAG.

The Quality Management System

12. The OAG has developed a Quality Management System (QMS) to provide it with reasonable assurance that audit engagements are conducted in accordance with applicable legislative requirements, professional standards and OAG policies. The QMS is progressively updated as these requirements change. The system is made up of three subsidiary QMSs, one for each practice line: performance audit; special examination; and annual audit.

13. The performance audit QMS has five major elements and 22 sub-elements. The QMSs for special examinations and annual audit each have three major elements with 11 sub-elements. The major elements of each QMS are listed in the following table and all sub-elements are outlined in Appendix 1.

Performance Audit QMS
Elements
Special Examination QMS Elements Annual Audit QMS
Elements
Leadership and Planning Examination Management Audit Management
Audit Management People Management People Management
Client Focus Continuous Improvement Continuous Improvement
People Management    
Continuous Improvement    

14. The subsidiary QMSs capture all the components necessary to guide OAG teams when completing audits and special examinations. For each sub-element, a statement of criteria describes the matter about which the sub-element is providing reasonable assurance, and the key instruments that support the sub-element. These instruments include legislation, standards, practice manuals and other OAG guidance. Examples of criteria and key instruments are outlined in Appendix 1.

The Peer Review

15. In December 2008, the Auditor General of Canada invited the Auditor-General for Australia to lead an international peer review of the OAG. This is the second peer review of the OAG. The first was reported in 2004 and focused on the performance audit practice.6 An external audit of the OAG’s annual audit practice was undertaken in 2000.7

16. The objective of this peer review was to provide the Auditor General of Canada with an independent opinion on whether the OAG’s QMS was suitably designed and operating effectively to provide reasonable assurance that the work of the OAG complied with relevant legislative authorities and professional standards. A Memorandum of Understanding, between the OAG and the peer review members, set out the agreed terms and conditions of the peer review, including the objective.

17. The scope of the peer review included all audit and assurance practice lines, as well as key services that directly support the audit and assurance practice lines. These services included professional development and resourcing of individual audits. The peer review objective, scope and approach are described in more detail in Appendix 2.

18. The period under review covered audit and assurance engagements reported during September 2008 to October 2009. The sample of OAG audits reviewed was:

19. The peer review report includes:

20. The OAG provided the peer review team with full support and access to audit documentation to allow the review to be completed in an efficient and timely manner. The peer review members wish to record our appreciation for the high level of support and co-operation provided by the Auditor General and her staff.

The Peer Review opinion

21. In our opinion, for the period under review, the QMS for performance audit, special examination and annual audit was suitably designed to provide reasonable assurance that the work of the OAG complied with relevant legislative authorities and professional standards.

22. For the period under review, the QMS was operating effectively for the performance audit and special examination practices. For the annual audit practice, the QMS was generally operating effectively, but there were implementation issues in relation to the following areas that need attention:

23. At the time of our review, the OAG was undertaking a range of initiatives designed to improve its performance, including with respect to the issues raised in this report.

Key observations on the design of the Quality Management System

24. This section describes the peer review team’s key observations against the first facet of the peer review objective—the design of the QMS.

25. We analysed the design of the OAG’s QMS as it applied to performance audits, special examinations and annual audits. This analysis involved aligning the elements of each QMS with relevant legislation, assurance and auditing standards, policy and guidance to identify any gaps, overlaps or anomalies.

26. We consider that there is scope to improve the design of the QMS. For example, the performance audit QMS could be streamlined to eliminate some overlapping sub-elements. Additionally, the consistency of the scope of the criteria and the way they are expressed could be improved in the three subsidiary QMSs. We observed that some of the key instruments supporting the QMSs, such as policies relating to the Audit Management element of the annual audit QMS, could be revised to more closely align with the auditing standards.

27. During our review we also noted that, while the performance audit, special examination and annual audit manuals have not been kept up to date, the principles and essential elements remain the same and the OAG has addressed the gap by issuing interim guidance.

28. The OAG has commenced several initiatives that will impact on the design of the QMS. These include updating its audit manuals, with a scheduled completion date of December 2011, and clarifying audit roles and responsibilities.

Key observations on the operation of the Quality Management System

29. This section describes the peer review team’s key observations against the second facet of the peer review objective—the operation of the QMS. The observations address the major elements of the QMS.

30. To form an opinion on whether the QMS was operating effectively, we reviewed a sample of performance audits, special examinations, annual audits and practice reviews, as listed in paragraph 18. Our review methodology was developed with reference to relevant legislation and assurance and auditing standards.

Performance audit and special examination

Leadership and Planning

31. The majority of performance audit topics are determined through the OAG’s one-pass planning process. The purpose of this process is to ensure that OAG resources are focussed on the areas of highest significance and risk, and to provide assurance to Parliament that the office is fulfilling its responsibilities. It is a risk-based long-range planning exercise generally conducted at the entity level. The outcome is a One Pass Plan (OPP) that identifies proposed performance audits to be conducted over the next five years. The OPP is reviewed periodically and at the completion of each audit. Audit topics may also be selected in response to a request from the Government or a parliamentary committee, or identified as a priority area by the Auditor General or as a result of discussions by the OAG Executive at its annual planning meeting.

32. Generally, the audit team plans a performance audit based on the topic outlined in the OPP. This planning process is referred to as the survey phase and involves:

33. The planning process results in an audit plan, which includes an audit logic matrix. The audit logic matrix outlines the key elements of the audit, such as the audit objectives, lines of inquiry, criteria, questions, methodology and potential sources of data. Around 40 per cent of the audit’s budget is dedicated to the survey phase. This stage was satisfactorily completed for all the performance audits we reviewed.

34. Audit criteria are the benchmarks against which a subject matter is audited. The audit team should identify or develop criteria that are consistent with the objective of the audit and suitable for evaluating the subject matter.8 Sources of criteria include legislation, government policy, standards established by recognised bodies of experts, and generally accepted better practice. Criteria may also be adapted from other OAG engagements or sourced from organisations carrying out similar activities, such as other audit offices. We observed that the OAG generally uses criteria that are based on legislation or government policy.

35. The Financial Administration Act requires that each parent Crown corporation be subject to a special examination at least once every 10 years.9 The OAG has developed a schedule to ensure that special examinations are undertaken within the required timeframe.

36. For the three special examinations reviewed, we observed that examination scoping and planning was sound. As required by legislation, the audit teams had committed time and effort to understanding the Crown corporation, and analysing its expected results and the risks to achieving those results. The corporation’s key financial, management and information systems and practices (referred to as systems and practices) that would be assessed as part of the examination were also identified. In addition, the special examination plans clearly outlined the objective, key systems and practices, criteria, probable sources of evidence, required resources and examination timetable.

Audit Management

37. We reviewed the evidence to support the key issues, findings and conclusions for the six performance audits and three special examinations in our sample. We found that the audit teams had obtained sufficient and appropriate evidence to support the findings and conclusions expressed in the reports.

38. We observed that performance audit and special examination reports were balanced, coherent and written using plain language. We also found that the findings and conclusions in the performance audit reports addressed the audit objectives. While we recognise that the OAG has adopted a policy of publishing concise reports, we felt that, for some audits reviewed, providing additional context would have helped the reader to better understand the audit findings and their potential impact, and the merits of the recommendations.

39. Special examination reports have a standard structure that leads the reader through the report in a logical way. The reports did not always explain which systems and practices (or components within the systems and practices) were selected for detailed examination and the reasons why they were selected, although this information was communicated to the corporation’s board of directors in the examination plan. Also, in some reports there was not a clear explanation of the alignment between the findings and the reported conclusions for each system and practice examined, and the corporation’s systems and practices as a whole.

40. When reviewing the sample of audit and examination files, we observed that the audit/examination teams complied with OAG policies and procedures. This included documenting key sign-offs at significant points during the audit/examination and completing the necessary documentation. For example, as required by OAG procedures, the relevant practice management committee reviewed and approved the audit and examination plans and budgets for all performance audits and special examinations in our sample.

41. In addition, we compared the actual and budgeted hours for each of the audits reviewed. The OAG defines ‘on budget’ as completing an audit or examination in no more than 115 per cent of the approved budgeted hours. For example, an audit with a budget of 8 000 hours would be considered on budget if the actual hours were less than 9 200. We found that the audits and examinations were completed on time and within budgeted hours.10

42. Auditing standards and OAG policies require that audit team members are objective and have the necessary auditing proficiency. For all the performance audits and special examinations we reviewed, the OAG’s senior management considered the skills, competence, independence and collective knowledge of the team members during the planning phase and these were assessed as being sufficient for the requirements of each audit or examination.

43. The OAG uses an electronic audit management software program to record its audits and examinations. This software program was not always used by performance audit teams—some audits were completely paper-based, while others used a combination of electronic and paper-based files; only one of the six performance audit files we examined was recorded electronically. In addition, we found that when the software was used, its application was inconsistent, with considerable variation in the structure and organisation of electronic files. The audit management software program was used for special examinations, but, in two of the three examinations we reviewed, it was difficult to follow the audit trail (the evidence) in the electronic working papers without the assistance of the audit team, although appropriate supporting evidence was available.

Client Focus

44. The OAG devotes considerable effort to communicating a clear and consistent audit message. This includes briefing parliamentarians and the media when publishing performance audits, monitoring media coverage and providing testimony to parliamentary committees on the results of audits. The OAG also seeks feedback from its stakeholders about the audit process. It periodically surveys the key users of its reports (parliamentarians, audit committees and boards of directors of Crown corporations) about its services and products, and surveys entities following the completion of an audit or special examination.

45. We met with a range of OAG stakeholders, including parliamentarians, senior executives from the Canadian public sector and chief executive officers of Crown corporations. They were supportive of the work of the Auditor General and her office.

People Management

46. The OAG has various policies prescribing its approach to managing its people. These include policies on harassment and discrimination, performance management, employment equity and workplace accommodation. The OAG’s professional development policy covers the competency, skills and development of its staff. In April 2009, responsibility for general training and technical training was divided between Human Resources and the Professional Practices Group, respectively. The Professional Practices Group has developed an audit training strategy that, if successfully implemented, will result in an enhanced audit training curriculum by 2012.

Continuous Improvement

47. The lessons learnt from the OAG’s practice reviews of performance audits, special examinations and annual audits are shared with OAG staff via meetings and seminars and are incorporated into learning and development programs. We examined a small sample of practice reviews and found that they were conducted in a thorough and professional manner with appropriate conclusions based on a comprehensive review program. We consider that more detailed referencing of the practice review programs to the audit/examination working papers would facilitate more efficient assessment of the practice review work.

48. We found that the OAG seeks feedback about, and lessons learnt from, individual audits and examinations through post-audit surveys of the entities subject to audit, internal practice reviews and audit/examination team meetings. However, at present the OAG does not capture, disseminate and use this feedback in a systematic manner.

49. Recently, the OAG has committed time and resources to improving its performance audit, special examination and annual audit practices. Some of this work has been ongoing for several years. Other projects were initiated in anticipation of forthcoming changes to Canadian auditing standards or as a result of the OAG’s 2008–09 internal practice reviews that identified several areas of the QMS requiring improvement. The OAG’s improvement projects include:

Annual audit

Audit Management

50. We examined seven annual audit files comprising four financial audits of Crown corporations, one territorial audit, one other statutory financial audit and components (three files) of the audit of the summary public accounts of Canada. As part of this process we met with the OAG staff involved in each audit.

51. We found that the audits we examined were generally compliant with Canadian auditing standards and OAG policy, with improvement needed in the following areas:

52. In accordance with its quality assurance system, the OAG conducted an internal practice review of annual audit files sourced from the same period as the peer review. The results are consistent with our findings on risk assessments and the sufficiency of documentation.

Assessing risks

53. Risk assessments are an integral part of annual audit planning and underpin the overall audit strategy and development of the annual audit plan. The auditor conducts risk assessment procedures to identify and assess the risk of material misstatement to the financial statements so that appropriate and efficient audit tests can be designed. The risk assessment process requires the auditor to understand the entity and its environment, including the various elements of internal control. In understanding internal control the auditor considers whether the entity has responded adequately to the risks arising from the use of information technology (IT) or manual systems by establishing effective controls. For significant risks or where substantive testing is considered insufficient, the auditor needs to evaluate the design and confirm implementation of key controls.11

54. The audit planning stage of the risk assessment process needs to be conducted in consultation with senior members of the audit team, including experienced IT audit practitioners. In general, the involvement of IT specialists in the audit team facilitates a controls reliance approach on the audit.12 This has the advantage of being efficient and confirming the integrity of the IT system or, where this is not possible, providing the opportunity to bring important issues to management’s attention.

55. We found that IT audit considerations were not well integrated into audit team planning. This makes it difficult to determine the financial reporting risk associated with IT systems and whether a thorough and appropriate audit response to that risk has been applied. For example, we observed that reliance was placed on reports from IT systems without the necessary evidence of underlying IT controls being evaluated.

56. We also found that the audit approach to assessing fraud risk was variable in the audit files and, in some cases, not demonstrably in accordance with requirements of the relevant auditing standard.13 Our review found for some audits:

57. In addition, we noted issues in a number of the files around the completeness and consistency of conclusions related to the components of internal control.14 In some cases, we found the linkages from the risk assessment process to the planned approach, relevant assertions, execution and testing were difficult to follow. The practice review also found cases of incomplete risk assessment procedures, including over outsourced payroll functions.

58. These findings underline the importance of the OAG giving greater emphasis to risk assessment in its annual audits.

Recommendation 1

59. We recommend that greater emphasis be given to implementing the risk assessment phase of the annual audit planning process that informs the nature and extent of further audit procedures.

The OAG’s response

60. Agreed. This is an ongoing challenge for the Office. Our own internal practice reviews have also identified the need for improvements in this area. The Office’s Executive Committee will take the necessary steps to ensure our audit methodology is fully complied with in practice. Our plan is to address this challenge in two ways. Firstly, as an interim step, we have provided guidance to assist auditors in improving the linkages of our risk assessments with both the planned and actual audit approach. The guidance also included increased senior audit management involvement in the planning (including risk assessment) phase of our audits. Secondly, we will ensure greater emphasis in this area as part of our Renewal of Audit Methodology (RAM) Project, which includes revising and updating our audit methodology, related audit tools, checklists and training, and a change management component to ensure our methodology is put into practice. The target date for completion of the RAM Project is December 2011. Our progress in addressing this issue will be monitored by the Office’s Executive Committee and by our ongoing practice review program.

Audit documentation

61. Canadian auditing standards outline the auditor’s responsibilities for documenting a financial statement audit. Documentation must be sufficient to enable an experienced auditor with no previous connection to the audit to understand the audit work, the evidence gathered and the resulting conclusions.15 OAG staff document the audit procedures undertaken and the conclusions reached using an electronic audit management program.

62. The electronic system also captures supervisory feedback from senior members of the audit team and their sign-off when they are satisfied that the audit file meets the audit objectives and professional standards. OAG team structures are designed to support effective timely supervision and electronic review of the work performed by junior and less experienced staff by senior members of the audit team. This review can be conducted by either a detailed assessment of the work or via a discussion approach called ‘review by interview’. The key role of senior members of the audit team is to ensure that the:

63. In a number of the audits we examined, there were examples where the electronic working papers did not contain sufficient documentation to allow an experienced auditor to confirm that the extent of audit procedures was sufficient to support the judgments and conclusions in the audit file. This does not mean that the audit opinion is incorrect. However, it does mean that not all matters envisaged by the standards have been documented. These documentation gaps in the electronic working papers suggest issues with both the preparation and timely review of working papers. The most significant issue, as detailed below, related to the collation and analysis of most likely and possible errors in the final stage of the audit. The audit team’s file review processes did not identify these issues for attention.

64. Under OAG policy, a document called the Report Clearance Summary (RCS) is prepared as documentary support for the audit report. It provides summarised information and detail that the audit signatory needs to review prior to signing the audit report. On occasion, we observed that the RCS was subject to revision after the initial presentation to the signatory of the audit report and review was not evidenced prior to the issue of the audit report. The clearance process would be improved if the RCS was signed when it provides a sufficient basis to sign the audit report.

65. A key step in each annual audit is the collation of unadjusted errors found during the audit and identified uncertainties, including those generated by particular audit sampling techniques. The OAG methodology defines the outcome of this process as the Upper Error Limit (UEL) and, in accordance with auditing standards, requires this to be compared to materiality.16 The results of this step, especially where the UEL approaches or exceeds materiality, need to be carefully assessed by senior members of the audit team; and their recommendation, as well as the basis for that recommendation, needs to be formally advised to the Auditor General (or delegate) through the RCS.

66. We found that this step was not adequately collated or documented for one of the OAG’s significant annual audits. The Auditor General, as signatory of the audit, was aware of the results of the audit; however, the RCS did not contain any detail that demonstrated consideration of the aggregate analysis of the unadjusted errors and uncertainties.

Recommendation 2

67. To meet the requirement for sufficient and appropriate audit documentation in annual audits, we recommend the OAG reinforce to staff the need for documentation on the electronic working papers to demonstrate compliance with relevant standards and OAG policy.

The OAG’s response

68. Agreed. Our own internal practice reviews have also identified the need for improvements in our audit documentation. The Office’s Executive Committee will take the necessary steps to ensure our audit methodology is fully complied with in practice. Our plan is to address this challenge in two ways. Firstly, as an interim step, we have provided guidance to assist auditors in ensuring the sufficiency and appropriateness of our audit documentation within our electronic audit working papers. The guidance also included increased senior audit management involvement in ensuring appropriate audit file review. Secondly, we will ensure greater emphasis in ensuring sufficient and appropriate audit documentation as part of our Renewal of Audit Methodology (RAM) Project, which includes revising and updating our audit methodology, related audit tools, checklists and training, and a change management component to ensure our methodology is put into practice. The target date for completion of the RAM Project is December 2011. Our progress in addressing this issue will be monitored by the Office’s Executive Committee and by our ongoing practice review program.

People Management and Continuous Improvement

69. These elements were addressed in the relevant parts of the Performance Audit and Special Examination section of this report.

Suggestions for the OAG to consider

70. The OAG has shown, through its policies and practices, a commitment to improving its performance in delivering its statutory responsibilities. Against this background, the following suggestions are made for the OAG to consider as it continues to improve its audit practices. These suggestions relate to:

Continuous improvement

71. We suggest that the OAG develop and implement a more robust and systematic process for ensuring that the lessons learnt from audits, stakeholder surveys and practice reviews are identified, captured and disseminated. These lessons should desirably be incorporated into learning and development programs and audit/examination methodology and guidance.

72. The OAG is implementing a number of initiatives to improve its performance audit, special examination and annual audit practices. In particular:

The timely implementation of these measures will enhance the operation and quality of the OAG’s audit practices.

Performance audit and special examination reports

73. As previously mentioned, we observed that reports were balanced and coherent. We also recognised the OAG’s policy of publishing concise reports. Within this context, we suggest that some OAG performance audit and special examination reports may be improved by including more context about the issue, project or program subject to audit or examination and an explanation of the potential impact of the findings. In most instances, we believe that providing additional context will not substantially increase the length of reports. In addition, the 2004 peer review suggested that the presentation of reports could be improved through the use of graphics and tables to present complex numerical data and footnotes showing sources of evidence.17 We consider that this suggestion is still valid.

74. As special examinations are now reported publicly, it would assist readers to better understand the report, and how the examiner concluded against the examination’s objective, if reports included:

Reporting to management

75. The OAG has policy and guidance related to communication with the audit entity on important matters identified during the financial statement audit. We suggest that the OAG consider developing a classification system that ranks annual audit findings according to the risk they represent to the audit entity and use this as a basis for determining which issues need to be formally reported to stakeholders. For example, annual audit findings could, depending on their significance, be separated into different categories, such as: significant deficiencies; those representing a moderate business or financial risk; and minor procedural matters. This would support consistency of reporting to audit committees and management, and also draw management’s attention to issues considered to be of higher importance by the OAG.

Using electronic audit working papers

76. The auditing standards require that audit working papers are reviewed on a timely basis. The electronic working papers allow managers to record their review and supervision of the audit team’s work. We observed instances where relevant audit documentation and evidence of review was kept on a paper audit file and not cross-referenced to the electronic working papers. Similarly, the better use of hyperlinks and cross-references within the electronic papers would facilitate senior review of the audit file. We suggest that review of annual audit files should be evidenced directly in the electronic audit working papers.

Good practice identified

77. During our review, we drew on the broad and diverse experience of our international members to make observations about the OAG’s QMS and its audit practices. We identified a number of good practices adopted by the OAG that will be of interest to other national audit offices. These include:

Managing the relationship with Parliament

78. The Auditor General’s primary client is the Parliament of Canada. She actively engages with Parliament via a number of fora, such as briefing parliamentarians about audit reports and the operations of her office, testifying before parliamentary committees, and meeting formally and informally with parliamentarians. Also, Parliamentary Liaison, an area within the OAG, facilitates communications between the OAG and parliamentarians. This engagement helps to foster a very effective working relationship between the Auditor General, her office and the Parliament.

Delivering a clear message

79. Considerable effort and resources are devoted to disseminating a clear and consistent message in the OAG’s performance audit reports. This includes the use of plain language and a logical structure that leads the reader through the report. The OAG also has an effective communication strategy that includes briefing parliamentarians and their staff and the media about the content of performance audit reports on the day they are tabled. The strategy also incorporates monitoring media coverage to verify that reported messages are consistent with the audit reports.

Collaborative audits

80. In February 2009, the Auditor General published Managing Identity Information.18 At the same time the Privacy Commissioner published Privacy Management Frameworks of Selected Federal Institutions.19 The reports were the result of collaborative audits undertaken concurrently by each office, consistent with their respective mandates. The two offices audited the same four departments, focussing on complementary issues, collaborated on joint audit-related processes and regularly shared information. Collaborative audits with other organisations, such as federal bodies and provincial audit offices, allow more extensive coverage of issues and the increased credence and influence that flows from complementary findings. The offices also benefit from sharing ideas and approaches to auditing.

Criteria and sub-criteria for special examinations

81. The OAG has developed Recommended General Criteria and Sub-criteria for Special Examinations. The document identifies criteria and sub-criteria for examining systems and practices in seven areas: corporate governance; risk management; strategic planning; performance measurement and reporting; information technology; human resources management; and environment and sustainable development. The criteria and sub-criteria were developed with reference to several sources, including legislation, policy requirements and Canadian and international standards.

Appendix 1—The Quality Management System (QMS)

The elements and sub-elements of the subsidiary QMSs for the three practice lines.

Performance Audit QMS
Leadership and Planning Audit Management Client Focus People Management Continuous Improvement
Strategic Direction Conduct of the Audit Communicating the Audit Resourcing Practice Review
Selecting the Audit Project Management Feedback from Clients and Stakeholders Leadership and Supervision Lessons Learned
Operational Planning Planned Audit Effective Reporting Respectful Workplace  
Methodology Accessible, Sufficient and Appropriate Evidence   Performance Management  
  Reporting the Audit   Professional Development  
  Consultation      
  Independence, Objectivity and Integrity      
  Security, Access and File Retention      

 

Special Examination QMS
Examination Management People Management Continuous Improvement
Authority Resourcing Practice Review
Independence, Objectivity and Integrity Leadership and Supervision  
Conduct of the Examination Performance Management  
Consultation Professional Development  
Security, Access and File Retention Respectful Workplace  

 

Annual Audit QMS
Audit Management People Management Continuous Improvement
Authority Resourcing Practice Review
Independence, Objectivity and Integrity Leadership and Supervision  
Conduct of the Audit Performance Management  
Consultation Professional Development  
Security, Access and File Retention Respectful Workplace  

The following table provides examples, from each QMS, of statements of criteria and the key instruments employed.

QMS Sub-element Criteria Key Instruments Employed

Performance Audit1

Accessible, Sufficient and Appropriate Evidence

Sufficient appropriate evidence is obtained to provide a reasonable basis to support the conclusion expressed in the report.

Matters that in the auditor’s professional judgment are important to provide evidence to support the conclusion expressed in the report are documented.

Performance Audit Manual and guidance

Confirmation of findings resulting from field work with the entity audited

Review by Assistant Auditor General/Commissioner of the Environment and Sustainable Development, quality reviewer and advisory committee of significance and ordering of issues, report message and conclusions and their rationale

Special Examination2

Consultation

When dealing with complex, unusual or unfamiliar issues, examination teams refer to authoritative literature and seek the assistance of Office specialists and/or individuals from outside the Office with appropriate competence, judgment, and authority.

Special examination policies, guidance and practitioner’s (Audit Principal) responsibilities for quality control

Advisory committees

Product leader special examination and others

Approved templates for special examination plan and report

Quality reviewers (Principal or Assistant Auditor General)

Access to external specialists and experts, information technology and legal services

Annual Audit3

Authority

The Office only undertakes audits where it has the authority to do so and, in those limited circumstances in which the Office can exercise discretion in accepting an engagement, they pose no undue risk to the Office.

Auditor General Act

Financial Administration Act and entity specific legislation

Annual audit policies and guidance

Legal Services team advice and support

Approval by executive committee

Sources:

1. Office of the Auditor General of Canada, Performance Audit Manual, 2004, p.120.

2. Office of the Auditor General of Canada, Manual on Special Examinations of Crown Corporations, OAG, 2000, p.103; and Office of the Auditor General of Canada, Practitioner’s (Audit Principal) Responsibilities for Quality Control, unpublished, 2009, pp.3-4.

3. Office of the Auditor General of Canada, Annual Audit Manual, OAG, 2003, p.180.

Appendix 2—About the Peer Review

Objective and scope

The objective of the peer review was to provide the Auditor General of Canada with an independent opinion on whether the Office's quality management system was suitably designed and operating effectively to provide reasonable assurance that the work of the Office complied with relevant legislative authorities and professional standards.

The scope of the peer review included all audit and assurance practice lines, as well as key services that directly support the audit and assurance practice lines. The period under review covered audit and assurance engagements reported during September 2008 to October 2009.

Approach

The review criteria were based on relevant legislation and Canadian assurance and auditing standards.20 The review was conducted in two main phases: assessing the design of the QMS; and assessing the operation of the QMS.

Assessing the design of the QMS

The purpose of this part of the peer review was to assess whether the OAG’s QMS was suitably designed. We analysed the design of the QMS as it applied to performance audit, special examination and annual audit. The analysis involved aligning the elements of each practice line’s QMS with relevant legislation, auditing standards, policy and guidance to identify any gaps, overlaps or anomalies.

Assessing the operation of the QMS

The purpose of this part of the peer review was to determine whether the QMS was operating effectively during the period of our review. To assess the operation of the system, we reviewed a sample of audits using a file review methodology. The sample is outlined in the following table.

Peer Review Sample of OAG Audits
Six performance audits
  • three performance audits of federal agencies
  • one performance audit of a territorial government agency
  • one follow up audit
  • one performance audit conducted by the Commissioner of the Environment and Sustainable Development
Three special examinations of Crown corporations
Seven annual audits
  • audit of the summary public accounts including two departments where specified audit procedures were performed
  • four financial audits of Crown corporations
  • one territorial financial audit
  • one other statutory financial audit
Components of the audit of the summary public accounts of Canada
Four 2008-09 OAG practice reviews
  • one performance audit
  • one special examination
  • two financial audits

We reviewed the reports and working papers, for the audits selected in the sample, against the file review methodology. The file review methodology was developed with reference to relevant legislation and the assurance and auditing standards, and was structured around the QMS. We interviewed the Auditor General, Deputy Auditor General, Commissioner of the Environment and Sustainable Development, Assistant Auditors General and OAG staff to assess how the QMS operated in practice.

Review team members met with a range of stakeholders, including parliamentarians, senior executives from the Canadian public sector and chief executive officers of Crown corporations to obtain their views on the work of the OAG and their relationship with the office.

Team members

The peer review team included senior auditors from the supreme audit institutions of Australia (review leader), The Netherlands, Denmark, Sweden and Norway.

Performance Audit and Special Examination Peer Review Team

Australian National Audit Office, Australia

Algemene Rekenkamer, The Netherlands

Rigsrevisionen, Denmark

Annual Audit Peer Review Team

Australian National Audit Office, Australia

Riksrevisionen, Sweden

Riksrevisjonen, Norway

Appendix 3—The OAG’s response to the Peer Review Report

Letter to Ian McPhee, Auditor General of the Australian National Audit Office, dated 14 May 2010

[text version]


Footnotes:

1. The Territorial Governments of Nunavut, the Yukon and the Northwest Territories. (Return)

2. The Financial Administration Act does not define ‘significant deficiency’. The OAG interprets ‘significant deficiency’ as a deficiency in a system or practice that could have a significant effect on the achievement of key results and, subsequently, on the corporation’s statutory control objectives. (Return)

3. The public accounts are the Government of Canada’s financial statements. (Return)

4. Office of the Auditor General of Canada, 2008–09 Estimates Performance Report, September 2009, p.8. (Return)

5. ibid., p.38. (Return)

6. National Audit Office United Kingdom, Riksrevisjonen Norway, Cour des Comptes France, Algemene Rekenkamer The Netherlands, International Peer Review of the Value for Money Practice of the Office of the Auditor General of Canada—Peer Review Report, February 2004. (Return)

7. PriceWaterhouseCoopers, Office of the Auditor General of Canada, External Audit of the Quality Management System for Annual Audits, January 2000. (Return)

8. CICA General Assurance and Auditing Standard, Section 5025, paragraphs 0.32–0.39. (Return)

9. Prior to March 2009, each Crown corporation was subject to a special examination at least once every five years. (Return)

10. The budget and reporting date of one of the special examinations was revised during the examination. The examination was completed in accordance with the revisions. (Return)

11. CICA General Assurance and Auditing Standard, Section 5141. (Return)

12. A controls reliant approach is where the auditor seeks to rely on the operation of management’s policies and procedures that are designed to prevent or detect and correct material misstatement. (Return)

13. CICA General Assurance and Auditing Standard, Section 5135. (Return)

14. The components of internal control are set out in CICA Section General Assurance and Auditing Standard 5141, paragraph 0.043, and include the control environment, the entity’s risk assessment process, the information system and related business processes relevant to financial reporting, communication control activities, and monitoring of controls. (Return)

15. CICA General Assurance and Auditing Standard, Section 5145, paragraph 0.12. (Return)

16. CICA General Assurance and Auditing Standard, Section 5142. (Return)

17. National Audit Office United Kingdom, Riksrevisjonen Norway, Cour des Comptes France, Algemene Rekenkamer The Netherlands, International Peer Review of the Value for Money Practice of the Office of the Auditor General of Canada—Peer Review Report, February 2004, p.8. (Return)

18. Office of the Auditor General of Canada, Report of the Auditor General of Canada to the House of Commons—Managing Identity Information, 5 November 2008 (tabled February 2009). (Return)

19. Office of the Privacy Commissioner of Canada, Audit Report of the Privacy Commissioner of Canada—Policy Management Frameworks of Selected Federal Institutions, February 2009. (Return)

20. The key Canadian assurance and auditing standards current at the time of the peer review were GSF-QC General Standards of Quality Control for Firms Performing Assurance Engagements and Section 5030 Quality Control Procedures for Assurance Engagements. We also referred to other Canadian assurance and auditing standards as necessary. (Return)