4044 Developing the Audit Strategy: Audit Logic Matrix

Jul-2020

What CSAE 3001 Means for Developing the Audit Strategy

CSAE 3001 refers to the need to plan the work so that it will be performed in an effective manner and to ensure that those performing the audit are properly supervised. Planning includes developing an objective and determining scope and approach, criteria, and possible sources of evidence. These are captured in the audit logic matrix (ALM), which sets out a plan to obtain sufficient, appropriate evidence to conclude against the audit objective. The ALM includes a high-level plan outlining the nature, timing and extent of audit procedures. Planning also includes developing a detailed approach to carrying out the audit, which is addressed in OAG Audit 4070 Audit programs.If audit procedures are incorporated in the ALM and not in separate audit programs, requirements under OAG Audit 4070 Audit programs still apply.

Performance audits and special examinations are always conducted at a reasonable assurance level (OAG Audit 101 Overview of performance audits and OAG Audit 102 Overview of special examinations).

Significance considerations in the context of planning an audit are addressed in OAG Audit 2020 Significance, whereas requirements for understanding the underlying subject matter to identify and assess risks are addressed in OAG Audit 4010 Understanding the subject matter in planning an audit and in OAG Audit 4020 Risk assessment respectively.

The standards require that the engagement leader obtain sufficient appropriate evidence to support the conclusion. Since the work reported in audit reports is performed at a reasonable assurance—the highest level of assurance that can be provided concerning the subject matter—observations, conclusions, and recommendations must be able to withstand critical examination. In determining whether they have gathered evidence of sufficient quantity and appropriate quality, auditors need to be certain that, in their judgment, there is minimal risk of making erroneous observations, faulty conclusions, or inappropriate recommendations. In other words, auditors need to minimize engagement risk (see OAG Audit 4020 Risk assessment).

Adequate planning also assists in properly assigning the work to the team members, and supervising and reviewing their work. These requirements are addressed in sections OAG Audit 3062 Engagement leader responsibilities for audit quality (OAG Audit 3061 Engagement team: assigning and managing tasks, and OAG Audit 3071 Review of audit work and documentation respectively).

The Audit Logic Matrix

The audit logic matrix (ALM) is a planning tool designed to help the team set out the audit strategy in a logical way by showing the alignment of the various elements and by identifying, at an early stage, any constraints to conducting the audit. It is used to communicate the key elements of the audit plan. (For details on key aspects of the audit strategy, please also refer to sections OAG Audit 4041 Audit objectives, OAG Audit 4042 Audit scope and approach, OAG Audit 4043 Audit criteria, and OAG Audit 4045 Evidence gathering methods).

The team develops the ALM based on information gathered in the planning phase and updates the ALM as it acquires more in-depth knowledge of the audit subject matter. The ALM is the culmination of planning decisions including the team’s risk assessment, consideration of internal controls (OAG Audit 4025 Internal controls), as well as a variety of other considerations around the scope and approach of the audit. This planning process is iterative. A well-designed ALM contributes to planning efficient and effective audits that will provide clear messages to Parliament or the board of directors of Crown corporations.

In designing the audit and drafting the ALM, the audit team should consider the implications of likely findings and potential key messages. The audit team can consider what would constitute a “pass” or “fail” of the criteria, and how big a gap would need to be to constitute a failure to meet the criteria. This assessment should guide the team in designing appropriate audit questions and evidence gathering techniques. The team can also consider what the impact of a “failed” criterion would be—the “so-what” of the finding, either in terms of the observed impact, or as a logical deduction. The audit team should also look for underlying causal factors (the “why so’s”), which may explain issues. Identifying and explaining the “why so’s” will help the team make more meaningful recommendations in the audit report. This in turn will help the entity, Crown corporations, or parliamentarians to follow up on the issues in a more enlightened manner.

The ALM Review Process

The ALM provides the team with an opportunity for having its audit plan and approach thoroughly reviewed and challenged. The team should start preparing the ALM as early as possible in the planning phase and circulate it, as appropriate (for example, to the quality reviewer and relevant internal specialists or other advisors).

As described in OAG Audit 3081 Consultations and OAG Audit 3082 Resolution of differences of opinion, all consultations about the ALM and the team’s responses to the advice given must be documented when dealing with difficult or contentious matters or other matters requiring specialized knowledge or experience.

The engagement leader is responsible for the final review and approval of the audit scope and approach as documented in the audit logic matrix. The audit logic matrix forms the basis of planning communication with the entity (see OAG Audit 4100 Special examination plan and OAG Audit 4090 Audit plan summary for performance audits).

The team should document any significant changes to the ALM subsequent to approval. Any changes to the objectives or criteria should be approved by the engagement leader, discussed with the quality reviewer, and communicated to the entity if made after the Audit Plan Summary (or Special examination plan) is sent to the entity. Significant changes in the direction of the audit should also be discussed with the assistant auditor general and the Auditor General. Other changes, such as to the information sources and evidence-gathering methods, do not need to be approved. The team makes these types of changes as the audit progresses.

Tips for Preparing the Audit Logic Matrix

Given the varying complexity of audits, the different matters being audited, and entity differences, no one ALM example fits all. The following are a few general tips for completing the sections of the ALM. The examples are taken from a number of different performance audits.

Audit Objective: This topic including guidance on wording the objective is addressed in OAG Audit 4041 Audit objective.

Subject Matter and Context:

This section provides an overall description of the subject matter and explains the risk-based rationale for the audit.

• Summarize the program or activity to be audited and its results, outputs, or outcomes.

• Describe the main objectives related to the subject matter (i.e. priorities, commitments, outcomes, mandate)

• Explain the materiality and potential for impact of the subject matter: for example, program costs, number of employees, or number of clients served.

• Provide any relevant history: for example, findings from previous related audits, and recent and current government initiatives.

• Provide relevant, recent developments affecting the entity. For example, “Recent reorganization within the department has resulted in unclear roles and responsibilities.”

• For performance audits, explain the importance of the subject matter to the OAG mandate and to Parliament or to Canadians: for example, “Chemical substances enter our air, water, land, and food from many sources. Because Canadians cannot always tell which chemical substances they may come in contact with, they rely on government to ensure that chemicals in the Canadian market present no unacceptable risks to their health and the environment.” For special examinations, the audit is mandatory under the FAA.

Audit Scope and Approach

A high-level description of the audit scope and approach. Refer to discussion of scope in OAG Audit 4042 Audit scope and approach. It should also describe any key areas excluded from the audit scope, including a rationale.

Risks. A summary of key risks from the team’s risk assessment process conducted during planning and how the audit team plans to respond to these risks (OAG Audit 4020 Risk assessment). This section describes

• the risk-based rationale for the audit (e.g. subject matter risks identified from the team’s risk assessment process and how the audit approach addresses them); and

• any significant engagement risks and/or auditability issues and how the team plans to manage them.

Entity Management Responsibility. This section describes the entity(ies) responsibility for the subject matter as it related to the audit objective. It is a description of the key accountabilities upon which the audit is based and refers to the relevant legislation and/or regulation for the entity(ies) involved. For example,

This information is used as part of confirming management responsibility when communicating the terms of the audit with entity management (OAG Audit 4090 Audit plan summary for performance audits or OAG Audit 2030 Communication with the audit entity initial and ongoing for special examinations).

Period(s) covered by the audit. Period covered by the audit is a scope consideration addressed in OAG Audit 4042 Audit scope and approach. Differences in the period covered by the audit by each line of enquiry or by individual evidence gathering methods should be identified.

Planned Value Added for Performance Audits

Planned value added is a scope consideration addressed in OAG Audit 4042 Audit scope and approach.  The audit team should critically evaluate how each component of the audit (i.e. each Line of Enquiry) contributes to the planned value added of the performance audit.

Potential Overall Key Messages

Insert the overall message that could be reported to Parliament or the board of directors in the audit report, based on possible audit findings and conclusions, and what impact (“so what?”) statements could be made.
Be neutral and give alternative outcomes so the audit will not be biased in one direction or another. For example:

Lines of Enquiry 

Lines of enquiry are areas to be audited within the scope. Additional information about lines of enquiry is discussed in OAG Audit 4042 Audit scope and approach. The ALM sets out the audit approach for each line of enquiry by including the following:

• How this piece of work fits into and supports the overall audit objective as well as any additional information on the topic or context that is specific to the line of enquiry, if relevant. For example, describe the subject matter risks that the LOE addresses (see topic and context above).

• Any additional information on scope and approach (including the period covered by the audit) that is specific to the line of enquiry, if relevant (see above scope and approach and period covered by the audit). For special examination, this would include listing the selected systems and practices to be looked at within the LOE.

• Criteria and their sources: this topic is discussed in OAG Audit 4043 Audit criteria.

• Audit questions: see below for more information.

• Information required and sources: see below for more information.

• Evidence gathering methods and limitations: see below for more information.

• Potential key messages that are specific to the line of enquiry (see overall potential key messages above).

• The specific planned value-added statements supported by the line of enquiry (see above concerning planned value added).

Audit Questions

Audit questions are the set of questions for each criterion that should yield sufficient appropriate evidence to assess and ultimately conclude on the criterion. Audit questions flow directly from the audit criteria and form the basis for identifying required documents or data necessary to answer these questions.

• Ask the questions that will yield sufficient appropriate evidence to assess and ultimately conclude on the audit criteria.

• In most cases, ask questions that produce a “met” or “not met” answer; for example, “Has the organization determined how good its service delivery needs to be?” Exceptions might be questions that look to explore the cause and impact of the situation.

• Then add the subsidiary question that would elicit by how much the entity had failed or exceeded the expectation. For example,

  • “Has the organization defined what it means by quality service?
  • Does it have service commitments?
  • Has it set standards associated with its service commitments (measurable levels of performance that clients can expect)?
  • Has it set measurable internal or operational performance targets for these standards?”

• Make sure the questions

  • fully cover the criterion,
  • do not go beyond that criterion,
  • are not too detailed or numerous, and
  • address the why so (cause) and so what (impact) of the situation.

Information Required and Sources

• Identify the type of information required and sources of the information, giving examples of documents and data.

• Provide examples of the positions and levels of individuals who will be interviewed and their department and region.

• List all groups and stakeholders from whom evidence will be sought.

• For example,

Evidence Gathering Methods and Any Limitations

This section sets out the high-level nature, extent and timing of the audit procedures that the team plans to use to obtain sufficient appropriate evidence to assess each criterion. Evidence gathering techniques are discussed in OAG Audit 4045 Evidence gathering methods.

• Each data collecting and analysis test should answer an audit question or set of questions.

• Do not forget to add work to examine the why so (cause) and so what (impact) of a situation.

• Provide a summary description of the audit test and the evidence-gathering method(s) but leave the details for the audit programs. For example,

  • “Test whether risk assessments of SIN program have been completed and integrated into the investigation function (including, using trend analysis and lessons learned of current investigations to identify risks and modify priorities and investigative responses).”

  • “Test whether investigators in the field have and use guidance on how to prioritize investigations and that it is based on risk.”

  This section also includes a discussion of any potential limitations that would limit audit evidence and the ability to conclude on the audit question or expectation. Consideration of the limitations is done to help the audit team ensure it gathers sufficient appropriate evidence to minimize the risk of forming an incorrect conclusion. For example,

  • “Regions and local sites in each organization may have different processes and may collect and use different types of information. Conclusions may be limited to verifications done at selected sites and may not be representative of the entire organization.”

  • “Surveys and other analysis undertaken by organizations are secondary sources of evidence and use of this information will be limited to determining what the organization does with it. If survey or analysis results are quoted in the audit report, this will be for context only with the appropriate source statement.”

See section OAG Audit 4020 Risk assessment for more information on managing engagement risk as well as in section OAG Audit 4042 Audit scope and approach for additional information on determining the nature and extent of procedures including limitations.

Examples of performance audits

The following examples demonstrate the alignment between the audit topic, underlying subject matter, objective, criteria, findings and conclusion(s) in a performance audit. These examples have been simplified to show this alignment.

Next Steps

The ALM forms the basis for the engagement leader’s examination approval (see OAG Audit 4080). Once the ALM is finalized, the team moves on to writing the audit plan summary (or the special examination plan) for entity approval and developing detailed audit programs (OAG Audit 4070 Audit programs). It is a common practice to use the ALM as a basis for developing audit programs. It is possible to incorporate the audit procedures in the ALM instead of developing separate audit programs, but requirements under OAG Audit 4070 Audit programs would still apply.