COPYRIGHT NOTICE — This document is intended for internal use. It cannot be distributed to or reproduced by third parties without prior written permission from the Copyright Coordinator for the Office of the Auditor General of Canada. This includes email, fax, mail and hand delivery, or use of any other method of distribution or reproduction. CPA Canada Handbook sections and excerpts are reproduced herein for your non-commercial use with the permission of The Chartered Professional Accountants of Canada (“CPA Canada”). These may not be modified, copied or distributed in any form as this would infringe CPA Canada’s copyright. Reproduced, with permission, from the CPA Canada Handbook, The Chartered Professional Accountants of Canada, Toronto, Canada.
4044 Developing the Audit Strategy: Audit Logic Matrix
Early in the audit, it is important to plan carefully the work that will be done during the examination phase. The OAG has designed a planning tool for audits, called the audit logic matrix (ALM), which describes the logical relationship between the audit objectives, criteria, audit scope, and approach and the observations to emerge. The ALM documents the overall audit design. A well-designed ALM contributes to planning efficient and effective audits that will provide clear messages to Parliament.
CSAE 3001 Requirements
44. The practitioner shall plan the engagement so that it will be performed in an effective manner, including setting the objective, scope, timing and direction of the engagement, and determining the nature, timing and extent of planned procedures that are required to be carried out in order to achieve the objectives of the practitioner. (Ref: Para. A2-A3, A85-A89)
49. The practitioner shall consider significance when: (Ref: Para. A90-A98)
(a) Planning and performing the assurance engagement, including when determining the nature, timing and extent of procedures; and
51R. The practitioner shall obtain an understanding of the underlying subject matter and other engagement circumstances sufficient to:
(a) Enable the practitioner to identify and assess the risks of significant deviation; and
(b) Thereby, provide a basis for designing and performing procedures to respond to the assessed risks and to obtain reasonable assurance to support the practitioner’s conclusion. (Ref: Para. A99-A103, A105-A109)
CSAE 3001 Application Material
A2. The practitioner in a performance audit describes in the report the objective of the engagement and the underlying subject matter so that the reader can understand and properly interpret the results. The wording of the objective would be determined by the circumstances of the engagement. For example, the objective for a performance audit may be to conclude whether the entity being audited has adequately managed a program so that the entity’s key responsibilities under that program have been met. The practitioner’s conclusion relates to the objective and scope of the engagement and follows logically from the description of the criteria and findings. If the engagement has more than one objective, the assurance report provides a conclusion on each objective.
A85. Planning involves the engagement partner, other key members of the engagement team, and any key practitioner’s external experts developing an overall strategy for the scope, emphasis, timing and conduct of the engagement, and an engagement plan, consisting of a detailed approach for the nature, timing and extent of procedures to be performed, and the reasons for selecting them. Adequate planning helps to devote appropriate attention to important areas of the engagement, identify potential problems on a timely basis and properly organize and manage the engagement in order for it to be performed in an effective and efficient manner. Adequate planning also assists the practitioner to properly assign work to engagement team members, and facilitates the direction, supervision, and the review of their work. Further, it assists, where applicable, the coordination of work done by other practitioners and experts. The nature and extent of planning activities will vary with the engagement circumstances, for example the complexity of the underlying subject matter and criteria. Examples of the main matters that may be considered include:
- The characteristics of the engagement that define its scope, including the terms of the engagement and the characteristics of the underlying subject matter and the criteria.
- The expected timing and the nature of the communications required.
- The results of engagement acceptance activities and, where applicable, whether knowledge gained on other engagements performed by the engagement partner for the appropriate party(ies) is relevant.
- The engagement process.
- The practitioner’s understanding of the appropriate party(ies) and its environment, including the risks of significant deviation.
- Identification of intended users and their information needs, and consideration of significance and the components of engagement risk.
- The extent to which the risk of fraud is relevant to the engagement.
- The nature, timing and extent of resources necessary to perform the engagement, such as personnel and expertise requirements, including the nature and extent of experts’ involvement.
- The impact of the internal audit function on the engagement.
A86. The practitioner may decide to discuss elements of planning with the appropriate party(ies) to facilitate the conduct and management of the engagement (for example, to coordinate some of the planned procedures with the work of the appropriate party(ies)’s personnel). Although these discussions often occur, the overall engagement strategy and the engagement plan remain the practitioner’s responsibility. When discussing matters included in the overall engagement strategy or engagement plan, care is required in order not to compromise the effectiveness of the engagement. For example, discussing the nature and timing of detailed procedures with the appropriate party(ies) may compromise the effectiveness of the engagement by making the procedures too predictable.
A87. Planning is not a discrete phase, but rather a continual and iterative process throughout the engagement. As a result of unexpected events, changes in conditions, or evidence obtained, the practitioner may need to revise the overall strategy and engagement plan, and thereby the resulting planned nature, timing and extent of procedures.
A88. In smaller or less complex engagements, the entire engagement may be conducted by a very small engagement team, possibly involving the engagement partner [...] working without any other engagement team members. With a smaller team, co-ordination of, and communication between, team members is easier. Establishing the overall engagement strategy in such cases need not be a complex or time-consuming exercise; it varies according to the size of the entity, the complexity of the engagement, including the underlying subject matter and criteria, and the size of the engagement team. [...]
A99. Discussions between the engagement partner and other key members of the engagement team, and any key practitioner’s external experts, about the susceptibility of the underlying subject matter to significant deviation, and the application of the applicable criteria to the facts and circumstances of the engagement, may assist the engagement team in planning and performing the engagement. It is also useful to communicate relevant matters to members of the engagement team, and to any practitioner’s external experts not involved in the discussion.
A101. Obtaining an understanding of the underlying subject matter and other engagement circumstances provides the practitioner with a frame of reference for exercising professional judgment throughout the engagement, for example, when:
- Considering the characteristics of the underlying subject matter;
- Assessing the suitability of criteria;
- Considering the factors that, in the practitioner’s professional judgment, are important in directing the engagement team’s efforts, including where special consideration may be necessary (for example, the need for specialized skills or the work of an expert);
- Establishing and evaluating the continued appropriateness of quantitative and qualitative factors that are significant;
- Developing expectations for use when performing analytical procedures;
- Designing and performing procedures; and
- Evaluating evidence, including the reasonableness of the oral and written representations received by the practitioner.
The audit team shall develop an audit logic matrix that sets out the strategy for the audit and states the audit objective and risks, context, scope and approach, criteria, the audit questions to be answered based on the criteria, the evidence-gathering and analysis methods, any data limitations, and its potential messages for users. For performance audits, the audit logic matrix shall also include the planned value added of the audit. [Nov-2015]
The scope of all special examinations of Crown corporations shall, at a minimum, cover “core” systems and practices which are assessed using the Office’s standard criteria. Based on a risk and control assessment performed in the planning phase, the engagement leader can justify expanding the scope of the special examination beyond the core systems and practices. [Nov-2017]
What CSAE 3001 Means for Developing the Audit Strategy
CSAE 3001 refers to the need to plan the work so that it will be performed in an effective manner and to ensure that those performing the audit are properly supervised. Planning includes developing an objective and determining scope and approach, criteria, and possible sources of evidence. These are captured in the audit logic matrix (ALM), which sets out a plan to obtain sufficient, appropriate evidence to conclude against the audit objective. The ALM includes a high-level plan outlining the nature, timing and extent of audit procedures. Planning also includes developing a detailed approach to carrying out the audit, which is addressed in OAG Audit 4070 Audit programs.If audit procedures are incorporated in the ALM and not in separate audit programs, requirements under OAG Audit 4070 Audit programs still apply.
Significance considerations in the context of planning an audit are addressed in OAG Audit 2020 Significance, whereas requirements for understanding the underlying subject matter to identify and assess risks are addressed in OAG Audit 4010 Understanding the subject matter in planning an audit and in OAG Audit 4020 Risk assessment respectively.
The standards require that the engagement leader obtain sufficient appropriate evidence to support the conclusion. Since the work reported in audit reports is performed at a reasonable assurance—the highest level of assurance that can be provided concerning the subject matter—observations, conclusions, and recommendations must be able to withstand critical examination. In determining whether they have gathered evidence of sufficient quantity and appropriate quality, auditors need to be certain that, in their judgment, there is minimal risk of making erroneous observations, faulty conclusions, or inappropriate recommendations. In other words, auditors need to minimize engagement risk (see OAG Audit 4020 Risk assessment).
Adequate planning also assists in properly assigning the work to the team members, and supervising and reviewing their work. These requirements are addressed in sections OAG Audit 3062 Engagement leader responsibilities for audit quality (OAG Audit 3061 Engagement team: assigning and managing tasks, and OAG Audit 3071 Review of audit work and documentation respectively).
The Audit Logic Matrix
The audit logic matrix (ALM) is a planning tool designed to help the team set out the audit strategy in a logical way by showing the alignment of the various elements and by identifying, at an early stage, any constraints to conducting the audit. It is used to communicate the key elements of the audit plan. (For details on key aspects of the audit strategy, please also refer to sections OAG Audit 4041 Audit objectives, OAG Audit 4042 Audit scope and approach, OAG Audit 4043 Audit criteria, and OAG Audit 4045 Evidence gathering methods).
The team develops the ALM based on information gathered in the planning phase and updates the ALM as it acquires more in-depth knowledge of the audit subject matter. The ALM is the culmination of planning decisions including the team’s risk assessment, consideration of internal controls (OAG Audit 4025 Internal controls), as well as a variety of other considerations around the scope and approach of the audit. This planning process is iterative. A well-designed ALM contributes to planning efficient and effective audits that will provide clear messages to Parliament or the board of directors of Crown corporations.
In designing the audit and drafting the ALM, the audit team should consider the implications of likely findings and potential key messages. The audit team can consider what would constitute a “pass” or “fail” of the criteria, and how big a gap would need to be to constitute a failure to meet the criteria. This assessment should guide the team in designing appropriate audit questions and evidence gathering techniques. The team can also consider what the impact of a “failed” criterion would be—the “so-what” of the finding, either in terms of the observed impact, or as a logical deduction. The audit team should also look for underlying causal factors (the “why so’s”), which may explain issues. Identifying and explaining the “why so’s” will help the team make more meaningful recommendations in the audit report. This in turn will help the entity, Crown corporations, or parliamentarians to follow up on the issues in a more enlightened manner.
Although special examinations generally follow the same planning procedures as other direct engagements, the OAG has developed a set of “core” systems and practices and related standard criteria that must be examined in every special examination. These are based on a portfolio-wide risk assessment and are to be included in the ALM without needing further justification by the audit team’s planning process.
The ALM will also include any additional systems and practices and criteria that the Engagement Leader considers necessary, based on a risk and control assessment, to ensure that the audit scope and approach responds to the risk for the specific Crown corporation under audit. Refer to OAG Audit 4020 Risk assessment, OAG Audit 4025 Internal controls, OAG Audit 4042 Audit scope and approach, and OAG Audit 4043 Audit criteria for further guidance.
The ALM Review Process
The ALM provides the team with an opportunity for having its audit plan and approach thoroughly reviewed and challenged. The team should start preparing the ALM as early as possible in the planning phase and circulate it, as appropriate (for example, to the quality reviewer and relevant internal specialists or other advisors).
As described in OAG Audit 3081 Consultations and OAG Audit 3082 Resolution of differences of opinion, all consultations about the ALM and the team’s responses to the advice given must be documented when dealing with difficult or contentious matters or other matters requiring specialized knowledge or experience.
The engagement leader is responsible for the final review and approval of the audit scope and approach as documented in the audit logic matrix. The audit logic matrix forms the basis of planning communication with the entity (see OAG Audit 4100 Special examination plan and OAG Audit 4090 Audit plan summary for performance audits).
The team should document any significant changes to the ALM subsequent to approval. Any changes to the objectives or criteria should be approved by the engagement leader, discussed with the quality reviewer, and communicated to the entity if made after the Audit Plan Summary (or Special examination plan) is sent to the entity. Significant changes in the direction of the audit should also be discussed with the assistant auditor general and the Auditor General. Other changes, such as to the information sources and evidence-gathering methods, do not need to be approved. The team makes these types of changes as the audit progresses.
Tips for Preparing the Audit Logic Matrix
Given the varying complexity of audits, the different matters being audited, and entity differences, no one ALM example fits all. The following are a few general tips for completing the sections of the ALM. The examples are taken from a number of different performance audits.
Audit Objective: This topic including guidance on wording the objective is addressed in OAG Audit 4041 Audit objective.
Subject Matter and Context:
This section provides an overall description of the subject matter and explains the risk-based rationale for the audit.
Summarize the program or activity to be audited and its results, outputs, or outcomes.
Describe the main objectives related to the subject matter (i.e. priorities, commitments, outcomes, mandate)
Explain the materiality and potential for impact of the subject matter: for example, program costs, number of employees, or number of clients served.
Provide any relevant history: for example, findings from previous related audits, and recent and current government initiatives.
Provide relevant, recent developments affecting the entity. For example, “Recent reorganization within the department has resulted in unclear roles and responsibilities.”
For performance audits, explain the importance of the subject matter to the OAG mandate and to Parliament or to Canadians: for example, “Chemical substances enter our air, water, land, and food from many sources. Because Canadians cannot always tell which chemical substances they may come in contact with, they rely on government to ensure that chemicals in the Canadian market present no unacceptable risks to their health and the environment.” For special examinations, the audit is mandatory under the FAA.
Audit Scope and Approach
A high-level description of the audit scope and approach. Refer to discussion of scope in OAG Audit 4042 Audit scope and approach. It should also describe any key areas excluded from the audit scope, including a rationale.
Risks. A summary of key risks from the team’s risk assessment process conducted during planning and how the audit team plans to respond to these risks (OAG Audit 4020 Risk assessment). This section describes
the risk-based rationale for the audit (e.g. subject matter risks identified from the team’s risk assessment process and how the audit approach addresses them); and
any significant engagement risks and/or auditability issues and how the team plans to manage them.
Entity Management Responsibility. This section describes the entity(ies) responsibility for the subject matter as it related to the audit objective. It is a description of the key accountabilities upon which the audit is based and refers to the relevant legislation and/or regulation for the entity(ies) involved. For example,
“Social Insurance Numbers are issued and administered under the Employment Insurance Act by the Canada Employment Insurance Commission. The Commission has delegated the responsibility for the issuance of SINs and the administration of the Register to Human Resources and Social Development Canada. Service Canada, within HRSDC, is largely responsible for the operational policy, the delivery, and the administration of the SIN, including the Register. The Privacy Act governs the protection of personal information, including the SIN. The President of the Treasury Board (TB) is responsible for the administration of the Privacy Act within the federal government, including the preparation and distribution of related directives and guidelines . . .”
This information is used as part of confirming management responsibility when communicating the terms of the audit with entity management (OAG Audit 4090 Audit plan summary for performance audits or OAG Audit 2030 Communication with the audit entity initial and ongoing for special examinations).
Period(s) covered by the audit. Period covered by the audit is a scope consideration addressed in OAG Audit 4042 Audit scope and approach. Differences in the period covered by the audit by each line of enquiry or by individual evidence gathering methods should be identified.
Planned Value Added for Performance Audits
Planned value added is a scope consideration addressed in OAG Audit 4042 Audit scope and approach. The audit team should critically evaluate how each component of the audit (i.e. each Line of Enquiry) contributes to the planned value added of the performance audit.
Potential Overall Key Messages
Insert the overall message that could be reported to Parliament or the board of directors in the audit report, based on possible audit findings and conclusions, and what impact (“so what?”) statements could be made.
Be neutral and give alternative outcomes so the audit will not be biased in one direction or another. For example:
“CRA, CIC and HRSDC had adequate (do not have adequate; are missing important elements of) practices to manage the quality of service delivered to individuals. Depending on our findings we will report either by line of enquiry or by organization . . . The audit does not intend to compare entities, although it may point to good practices relevant to service delivery in general.”
“Until the government concludes whether the outstanding chemical substances are toxic, no measures under CEPA, 1999 can be put in place to control the risks they may represent to human health and the environment.”
“To monitor its performance, the organization considers (does not consider) a complete set of information: input on how well or poorly it is doing from its clients and from its own staff. Therefore the organization can identify (risks missing) important service issues and areas for improving service quality and client satisfaction”.
Lines of Enquiry
Lines of enquiry are areas to be audited within the scope. Additional information about lines of enquiry is discussed in OAG Audit 4042 Audit scope and approach. The ALM sets out the audit approach for each line of enquiry by including the following:
How this piece of work fits into and supports the overall audit objective as well as any additional information on the topic or context that is specific to the line of enquiry, if relevant. For example, describe the subject matter risks that the LOE addresses (see topic and context above).
Any additional information on scope and approach (including the period covered by the audit) that is specific to the line of enquiry, if relevant (see above scope and approach and period covered by the audit). For special examination, this would include listing the selected systems and practices to be looked at within the LOE.
Criteria and their sources: this topic is discussed in OAG Audit 4043 Audit criteria.
Audit questions: see below for more information.
Information required and sources: see below for more information.
Evidence gathering methods and limitations: see below for more information.
Potential key messages that are specific to the line of enquiry (see overall potential key messages above).
The specific planned value-added statements supported by the line of enquiry (see above concerning planned value added).
Audit questions are the set of questions for each criterion that should yield sufficient appropriate evidence to assess and ultimately conclude on the criterion. Audit questions flow directly from the audit criteria and form the basis for identifying required documents or data necessary to answer these questions.
Ask the questions that will yield sufficient appropriate evidence to assess and ultimately conclude on the audit criteria.
In most cases, ask questions that produce a “met” or “not met” answer; for example, “Has the organization determined how good its service delivery needs to be?” Exceptions might be questions that look to explore the cause and impact of the situation.
Then add the subsidiary question that would elicit by how much the entity had failed or exceeded the expectation. For example,
- “Has the organization defined what it means by quality service?
- Does it have service commitments?
- Has it set standards associated with its service commitments (measurable levels of performance that clients can expect)?
- Has it set measurable internal or operational performance targets for these standards?”
Make sure the questions
- fully cover the criterion,
- do not go beyond that criterion,
- are not too detailed or numerous, and
- address the why so (cause) and so what (impact) of the situation.
Information Required and Sources
Identify the type of information required and sources of the information, giving examples of documents and data.
Provide examples of the positions and levels of individuals who will be interviewed and their department and region.
List all groups and stakeholders from whom evidence will be sought.
“Through interviews and document review:
Map the timeline of events.
Identify what risk assessments have been done.
Determine how risk assessments are used to prioritize investigations.
Review the study done on the SIN Application Review Program as a possible tool to guide investigations and determine if a priority setting process was implemented.”
Evidence Gathering Methods and Any Limitations
This section sets out the high-level nature, extent and timing of the audit procedures that the team plans to use to obtain sufficient appropriate evidence to assess each criterion. Evidence gathering techniques are discussed in OAG Audit 4045 Evidence gathering methods.
Each data collecting and analysis test should answer an audit question or set of questions.
Do not forget to add work to examine the why so (cause) and so what (impact) of a situation.
Provide a summary description of the audit test and the evidence-gathering method(s) but leave the details for the audit programs. For example,
“Test whether risk assessments of SIN program have been completed and integrated into the investigation function (including, using trend analysis and lessons learned of current investigations to identify risks and modify priorities and investigative responses).”
“Test whether investigators in the field have and use guidance on how to prioritize investigations and that it is based on risk.”
This section also includes a discussion of any potential limitations that would limit audit evidence and the ability to conclude on the audit question or expectation. Consideration of the limitations is done to help the audit team ensure it gathers sufficient appropriate evidence to minimize the risk of forming an incorrect conclusion. For example,
“Regions and local sites in each organization may have different processes and may collect and use different types of information. Conclusions may be limited to verifications done at selected sites and may not be representative of the entire organization.”
“Surveys and other analysis undertaken by organizations are secondary sources of evidence and use of this information will be limited to determining what the organization does with it. If survey or analysis results are quoted in the audit report, this will be for context only with the appropriate source statement.”
See section OAG Audit 4020 Risk assessment for more information on managing engagement risk as well as in section OAG Audit 4042 Audit scope and approach for additional information on determining the nature and extent of procedures including limitations.
Examples of performance audits
The following examples demonstrate the alignment between the audit topic, underlying subject matter, objective, criteria, findings and conclusion(s) in a performance audit. These examples have been simplified to show this alignment.
|Example 1: Real estate|
|Performance audit topic||Management of government infrastructure|
|Underlying subject matter||Lifecycle management of government real estate|
|Objective||To determine whether Department X has managed effectively government buildings over their useful life through the buildings program.|
|Conclusion||Department X has managed effectively government buildings over their useful life through the buildings program.|
|Example 2: Climate change|
|Performance audit topic||Climate Change|
|Underlying subject matter||Government strategy to reduce greenhouse gas emissions and implement actions to adapt to the effects of climate change|
To determine whether the Department of Environment has detailed action plans and targets to reduce greenhouse gas emissions, is on track to meet those targets, and is monitoring and reporting on its progress.
Greenhouse Gas Reduction
Greenhouse Gas Reduction
|Conclusion||The objective is not met. The Department lacks detailed action plans and targets to reduce greenhouse gas emissions. As a result, it cannot assess whether it is on track to meet its targets. The Department has not reported publicly on its actions and progress.|
The ALM forms the basis for the engagement leader’s examination approval (see OAG Audit 4080). Once the ALM is finalized, the team moves on to writing the audit plan summary (or the special examination plan) for entity approval and developing detailed audit programs (OAG Audit 4070 Audit programs). It is a common practice to use the ALM as a basis for developing audit programs. It is possible to incorporate the audit procedures in the ALM instead of developing separate audit programs, but requirements under OAG Audit 4070 Audit programs would still apply.