Report on a Review of the Annual Audit Practice—Practice Reviews Conducted in the 2010–11 Fiscal Year

Introduction

1. The Office of the Auditor General conducts independent audits that provide objective information, advice, and assurance to Parliament, territorial legislatures, and Canadians. The Office has several product lines, including performance audits, annual audits, and special examinations.

2. Annual audits include audits of the summary financial statements of the Government of Canada and the territories, and of the financial statements of Crown corporations and other entities. They are performed in accordance with Canadian generally accepted auditing standards. The objective of annual audits is to provide an opinion on whether financial statements are presented fairly in accordance with Canadian generally accepted accounting principles. In certain cases, the auditor also provides an opinion on whether the transactions examined conform to the legislative authorities that govern the activities of the entity concerned.

3. The Practice Review and Internal Audit team conducted practice reviews of selected annual audits reported in 2010. This work was done in accordance with the monitoring section of The Canadian Institute of Chartered Accountants (CICA) Handbook—“Quality Control for Firms that Perform Audits and Reviews of Financial Statements, and Other Assurance Engagements (CSQC-1).” It was also done in accordance with the Office’s 2010–11 Practice Review and Internal Audit Plan, which was recommended by the Audit Committee and approved by the Auditor General. The plan is based on systematic monitoring of the work of all audit principals in the Office on a cyclical basis.

4. To meet CICA standards, the Office establishes policies and procedures for its work. These are outlined in an audit manual, various other audit tools, and a quality management system for each product line (which ensures that quality is built into the audit process). These guide auditors through a set of required steps to ensure that the audits are conducted according to professional standards and Office policies. There is a product leader at the assistant auditor general level for the annual audit product line, whose primary function is to provide leadership and oversight for the product line and to contribute to the quality of the individual audits.

5. This report summarizes the major observations related to the practice reviews of the selected annual audits.

Overview

Objective

6. The objective of practice reviews is to provide the Auditor General with assurance that

• annual audits comply with professional standards and applicable legislative and regulatory requirements,
• the quality management system (QMS) is effectively implemented, and
• issued audit reports are supported and appropriate.

Scope and methodology

7. We conducted practice reviews of nine annual audits and two components of the audit of the summary financial statements of the Government of Canada. The reviews were conducted on audit files for financial statements with year ends between September 2009 and August 2010. The nine annual audits include eight Crown corporation audits and territorial annual audits and one audit of an international organization.

8. Our reviews included an examination of electronic (TeamMate) and paper audit files. We examined audit files related to the planning, examination, and reporting of the audits. We also interviewed audit team members, quality reviewers, and other internal specialists, as appropriate. 

Quality management system elements and process controls reviewed

9. We focused our work on the selected elements and key process controls of the Quality Management System for Annual Audits (Appendix A) that we considered key or high risk. 

10. We also looked at how the quality reviewers carried out their responsibilities. Quality reviewers are management-level employees of the Office who are appointed to provide an objective evaluation, before the auditor’s report is issued, of the significant judgments that the audit team made and the conclusions that it reached in formulating the audit opinion. The quality reviewer is an important element of the Office’s quality control system and is involved in selected individual audits from initial planning decisions to the closing of the audit file. Appendix B describes the key process controls reviewed for each selected element of the Quality Management System for Annual Audits. 

Rating system 

11. We applied one of the following ratings to each selected QMS element of the individual annual audits under review:

• Compliance. Office policy requirements and applicable auditing standards or accounting principles were met; minor improvements might be possible.
• Compliance but needs improvement. Improvements are necessary in some area(s) to fully comply with Office policies and/or auditing or accounting professional standards.
• Non-compliance. Major deficiencies exist; there is non-compliance with Office policies and/or auditing or accounting professional standards.

12. After completing each practice review, we concluded on whether the audit opinion was supported and appropriate. 

13. This report highlights the procedures performed, the conclusions drawn, the deficiencies found, and the actions taken to resolve these deficiencies. 

Results of the Reviews

Compliance with the quality management system and process controls 

14. Overall, we found that, in all of the 11 files reviewed, the audit opinions were supported and appropriate. 

15. Three of the eleven audit files were fully compliant with the elements reviewed (none were fully compliant in the 2009–10 fiscal year). The remaining eight needed improvement in the application of certain elements of the quality management system (QMS). We noted improvements in the files that we reviewed this year—only 18 percent of the files needed improvement in three or more QMS areas (compared with 50 percent in the 2009–10 fiscal year). 

16. The 2008–09 practice review summary identified a number of instances of the QMS not being applied consistently and rigorously. As a result, the Office made it a priority to update and strengthen the design and implementation of the QMS. Given the nature and the extent of our findings in the 2008–09 fiscal year, and given that audits were under way at the time of our review, we did not expect all findings to be fully addressed in the 2009–10 fiscal year. 

17. Initiatives—such as major updates of the audit manual, a review and update of the professional development curriculum, and the development of an action plan to address the recommendations made in practice review and peer review reports from the prior year—were undertaken. 

18. We also noted that information sessions were held; checklists were issued to help auditors; the TeamMate audit tool was updated; and, during training, emphasis was placed on observations from prior reports. Individual audit groups also held group meetings to discuss the previous practice review recommendations and developed action plans to address these recommendations for their files. 

19. In our opinion, these actions contributed to the improvements we noted in this year’s practice reviews. As outlined later in this report, we identified two key areas for improvement: the application of the audit risk model and the extent of information technology audit work in annual audit engagements. 

Notable improvements

20. In the files we reviewed this year, two areas showed significant improvement over previous years: the role of the quality reviewer and consultation. 

21. Role of the quality reviewer. Five of the eleven files that we reviewed this year had a quality reviewer. We noted that reviews were timely, that they adhered to the Office’s guidance on quality reviews, and that they were more rigorous than those of the previous years. However, we observed that the quality reviewers did not always ensure that the audit strategy aligned with the work performed and risks identified. 

22. We noted that the Office guidance does not specify the key decisions and related audit work that the quality reviewer needs to review. Moreover, the guidance does not specify whether quality reviewers are expected to review the alignment between the strategic audit plan and the work that was actually carried out. In our view, guidance for quality reviewers should be more specific. 

23. Consultation. Although the 2009–10 practice reviews noted that progress has been made, more needed to be done to ensure that all required approvals are obtained before the Auditor’s Report is signed and the results are communicated to the audit entity. In each of 11 files that we reviewed in the 2010–11 fiscal year, we observed that the advice received from internal or external consultations, as well as the disposition of this advice, was well supported and documented. 

Opportunities for improvement in two key areas 

24. The observations throughout this section of the report and the recommendations at the end represent opportunities for improvement across the annual audit practice, based on the findings most commonly identified in the individual practice reviews. Two key areas for improvement are prevalent in the files reviewed:

• the alignment of risks, audit strategy, and audit work performed; and
• the integration of information technology work in the audit engagements.

Consistent with our 2009–10 report and with peer review results, the observations relate to how the audit was conducted.

Alignment of risks, audit strategy, and audit work performed

25. Understanding the entity’s business and its operating environment is important to assess the risks of material misstatements in the financial statements. It further allows the auditor to design an audit strategy that will address these risks and provides a basis for planning and determining the nature, timing, and extent of audit procedures. 

26. In 10 of the 11 files reviewed, we observed that senior management attended the planning meeting and was involved early in the planning process. In addition, for all of the 11 files, we noted that the identified audit risks were reflected in the preliminary audit approach. Overall, this is an improvement over last year. 

27. However, in 6 of the 11 audit files reviewed, the audit work performed did not match the level of work described in the preliminary audit approach or the planned level of assurance for the related financial statements assertions. We could not find justification for the change in audit approach or the required approvals. 

28. In three files, the audit programs had not been approved before the examination phase. Furthermore, in one file, the audit programs were too general to give auditors sufficient direction on the extent, timing, and nature of the audit procedures to be performed. 

29. Moreover, in our 2009–10 Report on a Review of the Annual Audit Practice, we noted a good practice: Senior management of an audit team (specifically, the Assistant Auditor General) challenged the strategic audit approach by performing a detailed review of the audit programs and planned procedures for significant audit cycles, which improved the efficiency of the audit. This year, we observed that, in two audit files, the Assistant Auditor General, challenged the audit approach. However, we further observed that in these two cases, not all of the recommendations were carried forward to the audit programs. 

30. In addition, as we had reported in previous years, audit teams have difficulty completing “summaries of comfort”—an audit tool used to link the risk of material misstatement with relevant audit assertions, audit work performed, and assurance gained. In at least seven of the files that we reviewed, several summaries of comfort were incomplete and did not align with the audit work performed. 

31. In all cases noted above, we observed that neither the file reviewers nor the quality reviewers identified the misalignment that we found between the planned audit approach and the actual work performed. 

32. As we noted in our previous reports, in our view, practitioners need to provide more direction on the strategic audit approach and to become more involved in ensuring the audit team is appropriately supervised and the work is adequately reviewed. 

33. Controls-reliant approach—identifying and testing controls. The Office is committed to taking a controls-reliant approach to an audit, whenever it is appropriate and practicable. We observed that audit teams still have difficulty applying the controls-testing methodology and guidance. In 7 of the 11 audit files that we reviewed, the audit teams performed tests of controls in certain significant cycles. 

34. While there is specific methodology for the sample size required for control testing, there is limited guidance on what to do when an exception or deviation is found. In conducting our reviews, we saw a variety of approaches: doing nothing, identifying compensating controls, or reverting to a substantive approach. 

35. We noted that in three files, out of the seven that tested controls, the sample size did not match the planned level of assurance. For example, two teams chose a sample size to achieve high assurance when moderate assurance was planned. Another team did not test a sufficient number of transactions to provide the audit assurance sought; however, we determined that the audit team had performed enough other procedures to obtain the level of assurance required. 

36.  In six of the seven audit files that we reviewed where exceptions were found in the control testing, the related conclusions were supported and justified for only one file. We noted that the Office guidance for control testing sampling and exceptions is limited. 

37. Finally, we noted that in the seven files that tested controls, the audit teams had difficulty identifying the key controls, which may have led them to perform more audit work than necessary to get the required assurance. 

38. Selective testing and audit sampling. Overall, four of the audits we reviewed used a substantive audit approach and did not rely on controls. The other seven audits used a combination of control testing and substantive testing. When using representative or statistical sampling, audit teams need to define the

• objective of the test,
• population from which the sample is selected,
• definition of an error,
• confidence level for the test; and
• tolerable error.

39. In five files, we noted that one or more of these elements were missing or incorrect. For example, in at least two of the audit files that we reviewed, we noted that the audit teams incorrectly identified the population subject to testing. In another, the audit team used dollar unit sampling but did not project the errors correctly. In another four files, the assurance level gained from testing did not match the assurance level expected from the audit procedure and the risk assessment.

40. We further noted that in two files, the audit teams chose targeted testing but left significant balances unaudited in one of the files and did not document how this was addressed through other audit work, as is required by office methodology. In the second file, the percentage of the population sampled by the audit team did not provide the assurance planned, as is required by office methodology. However, for this file, the reviewers determined that the audit team did perform enough of the other audit procedures to attain the required level of audit assurance. 

Integration of information technology work in attest audit engagements

41. Six out of the eleven audit entities that we reviewed use complex information technology (IT) systems. As we had in previous years, we found that IT considerations were not well integrated into audit planning, execution, and reporting. Only two files showed some IT work integrated into the audit. The other four did not include IT-related testing. They documented that either the systems were not complex or it was not efficient to perform IT-related control testing; or they remained silent on the risk of using IT in implementing and testing key controls.

42. There was no justification on file to support why it is inefficient to test IT-related controls. For example, in one case, the audit team chose to move away from IT reliance and to use a substantive approach without indicating why it was inefficient to test automated controls. In our discussions with audit teams, they told us that they are reluctant to do IT-related work because the hours required would put more pressure on the audit budget.

43. We again found that discussions between the annual audit teams and the IT audit team were not held at the appropriate level. In one audit file, the audit team identified a complex IT system as being “non-complex,” and the discussions between the audit team and the IT team were held at a junior level.

Other observations noted in our review

Authorities

44. Because the Office of the Auditor General is a legislative office, there is an expectation that the work related to compliance with applicable authorities is integrated into the overall audit strategy. We noted that not all teams identify the applicable authorities in the planning phase of the audit and in the strategic approach. While audit teams conduct some work on authorities, the nature of that work is not addressed in the strategic audit approach.

Documentation of reliance on the work of others

45. The audit teams have made a conscious effort to meet the standards for documenting their reliance on the work of others, such as specialists, internal audit, and service providers. In five audit files, we saw that the required documentation had improved. However, for three files, we found that the audit teams needed to improve how they documented the reputation and competencies of specialists and internal audit.

Independence

46. We noted overall improvement in this area. Independence forms were on file for all team members. However, we noted one area in which further improvement is required. We found that audit teams did not document the nature of any entity-specific threats to independence, nor did they document whether those threats had been discussed with the audit team.

File finalization

47. While file finalization has improved overall, there was one instance of a team not finalizing a file according to Office guidelines. In another case, the audit team finalized the electronic file, according to Office guidelines, but did not finalize the paper file.

Conclusion and Recommendation

48. For each of the 11 annual audits that we reviewed, the auditor’s report was supported and appropriate. However, we concluded that the implementation of certain elements of the quality management system and of the professional standards that we reviewed need to be improved.

49. Recommendation. The Office should develop

• interim training in the areas of the audit risk model (sources of assurance), including sampling, controls testing, and the alignment of planned work to actual work; and
• a strategy and a plan to increase the nature and extent of audit work on the entities’ IT systems.

Management has responded and agrees with the recommendation. Training is currently being offered or is planned for fall 2011 for each of the component parts of the audit risk/assurance equation.

Current offerings of newly developed audit training include

• audit risk assessment and preliminary audit approach by cycle,
• summary of comfort,
• summary of unadjusted errors,
• audit sampling, and
• best practices in working paper preparation.

Future offerings for fall 2011 will include:

• Our revised audit approach for deriving audit assurance from reliance on Controls (including controls testing) and
• Our revised audit approach for deriving audit assurance from substantive tests (including analytics and test of details).

An IT audit strategy and action plan is currently being revisited; changes to this strategy and plan are being made to fully implement the new IT audit policy approved by the Executive Committee in fall 2010. This updated strategy and related action plan will be completed this spring (spring 2011) and will address the need to increase the nature and extent of our IT audit work in large entities and in other entities with complex IT systems. The longer term objective of this strategy is to derive increased audit assurance from our reliance on these entities’ key internal controls.

Appendix A—Quality Management System for Annual Audits

[text version]

Appendix B—Quality Management System Elements and Process Controls Reviewed

Our review covers the following Quality Management System elements.

Conduct of the audit. We determined whether the audit was planned, executed, and reported in accordance with Canadian generally accepted auditing standards, applicable legislation, and Office policies and procedures. We considered whether the Office meets its reporting responsibilities by having in place appropriate audit methodology, recommended procedures and practice aids that support efficient audit approaches, producing sufficient audit evidence at the appropriate time.

As part of the conduct of the audit, we also reviewed the following process controls:

Audit file finalization. We determined whether audit files were closed within 45 days of the Auditor's Report being given final clearance by the signatory and the financial statements being approved by the Board of Directors of the entity, or its equivalent, as required by Office policy.

Consultation. We determined whether consultation was sought from authoritative sources and specialists with appropriate competence, judgment, and authority to ensure that due care was taken, particularly when dealing with complex, unusual, or unfamiliar issues. We also determined whether the consultations were adequately documented, and whether the audit team took appropriate and timely action in response to the advice received from the specialists and other parties consulted, including the quality reviewer.

We determined whether the quality reviewer carried out, in a timely manner, an objective evaluation of

• the significant judgments made by the team,
• the conclusions reached in supporting the auditor's report, and
• other significant matters that have come to the attention of the quality reviewer during his or her review.

We determined whether the work of the quality reviewer was adequately documented, and whether the audit team took appropriate and timely action in response to the advice received from the quality reviewer.

Resourcing. Based on interviews with staff and review of documents, we determined whether audit teams had collective knowledge of the subject matter and the auditing proficiency necessary to fulfill the audit requirements. As well, we determined whether the individuals carrying out the audit work had adequate technical training and proficiency. We also considered the number of staff and the timing of their availability.

Independence. We determined whether all individuals performing audit work, including specialists, had been independent in carrying out their responsibilities and in forming their conclusions.

Leadership and supervision. We determined whether individuals working on the audit received an appropriate level of leadership and direction and that

• adequate supervision of all individuals, including specialists, was provided to ensure that audits were properly carried out,
• all audit team members were encouraged to perform to their potential, and
• all received appropriate recognition.