2005 November Report of the Auditor General of Canada

Reporting the quality of surveys

2.19 The Treasury Board Secretariat's annual Preparation Guide—Departmental Performance Reports should ask departments and agencies to ensure that

• references to survey data are accompanied by a basic description of how the survey was conducted, along with key indicators of data quality and any data limitations;
• this information is readily accessible through footnotes or endnotes; and
• more detailed methodological information is publicly accessible, preferably through an Internet link to the final survey report.
  (2.14–2.18)

Through its guidelines on the preparation of departmental performance reports, the Treasury Board Secretariat encourages departments to explain in their reports why the reader can have confidence in the methodology and data used to substantiate performance. Among other things, good-quality public performance reports should be balanced, easy to understand, and unburdened by large amounts of technical detail. The Secretariat recognizes that reasonable documentation of survey methodology, presented in an unobtrusive manner, can enhance the quality of a public performance report. The Secretariat will address the reporting of survey methodology in upcoming versions of its guidelines and will continue to work with departments to improve the quality of reporting to Parliament.

Leadership for the quality of public opinion surveys

2.49 Public Works and Government Services Canada should ensure that the Public Opinion Research Directorate, as a mandatory common service organization, place sufficient emphasis on providing expert advice to departments and agencies on the quality of all surveys.
(2.44–2.48)

While departments are accountable under the policy framework for ensuring survey quality, the Public Opinion Research Directorate (PORD) recognizes the importance of this recommendation and will continue to provide expert advice to departments on survey quality. In May 2004, new contracting tools for national public opinion research were implemented, which included more stringent contractual requirements for the industry to improve research quality. Further quality elements will be added to contracting tools for public opinion research, to be renewed by 2007 and communicated to departments. PORD will also update its existing research guides to better inform departments of survey quality matters. As a further measure, the Directorate will implement a method for systematizing and documenting expert advice given to departments and will continue its practice of double-reviewing project files prior to sending projects for contracting.

2.53 To assist departments in distinguishing between good-quality and poor-quality surveys, the Public Opinion Research Directorate should adapt the relevant standards of the market research industry and develop suitable benchmarks applicable to surveys commissioned by departments.
(2.50–2.52)

The Public Opinion Research Directorate (PORD) recognizes the importance of this recommendation and will establish an expert technical advisory panel, which will include participation from Statistics Canada. The panel will assist in the development of suitable benchmarks for inclusion in the contracting tools that will be developed and made available to departments for the procurement of public opinion research. PORD will also continue to apply and adapt the relevant standards of the marketing research industry when reviewing projects and providing advice.

2.62 When the Public Opinion Research Directorate advises departments and agencies about surveys, Public Works and Government Services Canada should ensure that

• the advice is based on the systematic application of relevant standards and benchmarks to each survey on a case-by-case basis;
• all advice is documented; and
• departments and agencies are encouraged to reflect this advice in the contract so that concrete expectations of quality will be established.
  (2.54–2.61)

The Public Opinion Research Directorate (PORD) accepts this recommendation. The Directorate will develop a framework and working tools to aid advisory staff in systematically applying standards and benchmarks to the formulation of advice on a case-by-case basis. New practices will be put in place to better document advice given to departments. PORD will continue to provide advice to departments to be incorporated in the contract so that concrete expectations of quality will be established. In addition, further quality elements will be added to our next contracting tools for public opinion research, to be renewed by 2007 to assist departments in obtaining quality elements in the contracting process.

2.68 Public Works and Government Services Canada should ensure that the Public Opinion Research Directorate review the final survey reports for completeness and reliability.
(2.63–2.67)

The Public Opinion Research Directorate (PORD) recognizes the importance of final survey reports being complete and reliable. Prior to submitting final reports to the Library of Parliament and Library and Archives Canada, the Directorate will continue to review final reports for completeness and advise departments of required elements under Government of Canada policies. In keeping with institutional accountability for the quality of research under the Communications Policy of the Government of Canada and the management of expenditures under the Financial Administration Act, PORD will continue to advise and encourage departments to include necessary elements for describing quality and take recommended steps in any instance where standards and benchmarks have not been met. Further quality elements to promote completeness and reliability will be added to the next contracting tools for public opinion research, to be renewed by 2007.