This Web page has been archived on the Web.

Peer Review Report's Suggestions for Improvement and Our Action Plan

1. Looking at the scope of some audits for example by giving more emphasis to setting out the issues, criteria, methodology, sources of evidence and likely areas for recommendations at the design and planning stage. This could also help identify the need for external expertise, and internal technical and subject matter expert input.

Action

At the design and planning stage of each value-for-money (VFM) audit, we produce a survey report that sets out significant audit issues, audit objectives and criteria, audit approaches and methodologies, sources of evidence, and potential areas for recommendations. After consultation with internal and external experts, we develop an audit proposal.

By December 2004 we will

  • update our guidance and methodology on survey reports and audit proposals to specify coverage of the issues suggested by the peer review;
  • communicate our expectations concerning these planning reports to our value-for-money practitioners;
  • systematically monitor our practices against our expectations; and
  • include the names of potential external advisors and their areas of expertise in the audit proposal.

(Our action on suggestion 4 deals with audit recommendations.)

2. Reviewing the breadth of the audit methodologies routinely used on VFM audits to identify the opportunities for expanding the range of techniques used including use of comparative analysis and more quantitative and qualitative techniques such as focus groups. This could add depth to the audits and may lead to some savings in audit resources which could be used to tackle more complex areas of the audits.

Action

By December 2004 we will

  • identify and assess methodologies used by other national audit offices;
  • revise and promote our Auditor's Tool Kit, which contains guidance on alternative methodologies; and
  • devote a significant portion of the 2004 annual symposium for value-for-money practitioners to how alternative methodologies can be applied.

(Also see our action on suggestion 3.)

3. Expanding the training and guidance available to VFM staff on methodologies and presenting report data. This could include stronger recommendations and improving the presentation of audit reports by greater use of figures, graphics and case examples to support the main findings. There is also scope for learning lessons from audits and ensuring the Office has the appropriate mix of staff competences to deliver quality value-for-money products.

Action

Methodologies and presenting report data

By June 2005 we will

  • develop and deliver training courses for value-for-money practitioners on evidence-gathering methodologies and techniques for presenting data in reports.

Stronger recommendations

By December 2004 we will

  • develop a course on how to make stronger recommendations and include it in our value-for-money professional development program.

Improving presentation of reports

By December 2004 we will

  • review the Canadian Institute of Chartered Accountants' study on using graphics and tables to present complex numerical and financial data, tailor it to our needs, and reference it in our Value-for-Money Audit Manual.

Scope for learning lessons

By December 2004 we will

  • develop a database containing lessons learned from value-for-money audits and suggestions for improvement to our quality management framework; and
  • establish a systematic approach to ensuring that lessons learned and suggestions received from practitioners are considered, and, where appropriate, incorporated into our quality management framework and value-for-money audit procedures.

Ensuring appropriate mix of competencies

By December 2004 we will

  • complete a review of all value-for-money audit teams to ensure that the appropriate mix of human resources is available to manage audit workload over the long term;
  • redesign our professional development curriculum to focus on the competencies necessary to deliver value-for-money products; and
  • determine whether core elements of the curriculum should be mandatory for all value-for-money practitioners.

4. Recommendations should be specific and action-orientated. Earlier consideration of the areas for recommendation in the audit cycle could enable audit teams to focus on beneficial change and discuss with departments at an earlier stage.

Action

By December 2004 we will

  • consult our stakeholders—including our value-for-money practitioners, federal departments, and the Treasury Board Secretariat—to define the key issues associated with our report recommendations;
  • finalize guidance on how to write specific and action-oriented recommendations and get approval from our Practice Development Committee;
  • reference the guidance from our Value-for-Money Audit Manual and incorporate it in standard practice;
  • adjust our practice to ensure that potential recommendations for each line of enquiry are considered in the planning stage of the audit; and
  • develop a course on how to make stronger recommendations and include it in our value-for-money professional development program.

In 2004, we issued with the Treasury Board Secretariat a protocol for deputies and senior managers in departments on working with the Office on VFM audits. The protocol encourages a transparent process and open dialogue with departments at each stage of an audit.

5. Improving the presentation of reports both in terms of use of graphics and tables to present complex numerical and financial data to enliven the chapters. Footnotes might be used to show sources of evidence. Skeleton or outline reports could also be used to develop key graphics at an earlier stage. Finally, more prominence could be given in audits (perhaps in the About the Audit section of each chapter) to where experts and consultants have been used and thereby add weight to the audit findings.

Action

By December 2004 we will

  • review the Canadian Institute of Chartered Accountants' study on using graphics and tables to present complex data, tailor it to our needs, and reference guidance in our Value-for-Money Audit Manual (same as action for suggestion 3).

By June 2005 we will

  • develop courses for value-for-money practitioners on techniques for presenting complex data in reports (see action for suggestions 3 and 4).

By September 2004 we will

  • determine whether or not reference notes would add value to our reports and if so, how they might best be incorporated.

By June 2004 we will

  • identify options for highlighting, in our reports and communiqués, the methods we use to ensure quality in our value-for-money audits, including the use of experts; and prepare a proposal for approval by the Executive Committee.

6. Doing more to show that the VFM audits make a difference, for example in following-up on not only where recommendations have been implemented but what they have achieved in terms of improved performance.

Action

By February 2005 we will

coordinate the processes we use to generate information for our annual status report, which addresses progress on audit recommendations, and for our performance report, which highlights action and changes from our audit work, to ensure that they adequately describe instances where our audit work can be linked to improved government performance.

We have recently revised our methodology for following up on audit recommendations. Our revised methodology will address suggestion 6.

By June 2005 we will

  • survey a random sample of parliamentarians and members of Parliament from key committees, such as the standing committees on Public Accounts and Environment and Sustainable Development. The survey will help us understand how familiar parliamentarians are with the Office and our products, and how we can improve our value-for-money products to better serve their needs.

By November 2005 we will

  • survey a sample of parliamentary and legislative assistants who read our reports and brief parliamentarians, to see how we can improve our value-for-money products to better serve parliamentarians.

By June 2005 we will

  • survey parliamentary committee clerks and research staff of the parliamentary library who read our reports, to see how we can improve our value-for-money products to better serve parliamentarians.

We try to survey parliamentarians within one year after an election.

We will

  • continue to survey senior managers responsible for areas we have audited, after each report is tabled, to obtain their feedback on the value of our process and products.

7. Looking at the cost of individual audits with more analysis of the comparative costs of audits and possibly the cost elements of individual audits to determine whether the resources allocated to them reflect the needs of the audit—this is closely linked to the approval of audits and the methods used and the perceived 'cost' of the QMF (quality management framework).

The work done in response to this suggestion is fully integrated with the Office's Modern Comptrollership Capacity Assessment and Action Plan.

Action

By March 2005 we will

  • refine our existing guidelines used to plan and monitor value-for-money audits to address the level of effort normally associated with each stage of the audits;
  • communicate these refinements to our value-for-money practitioners and reference them in our Value-for-Money Audit Manual; and
  • refine our approval and monitoring processes to reflect these guidelines, including the introduction of our new executive reporting package.