Conducting Surveys

Note: This guide is intended to ensure that surveys conducted in the Office meet reasonable requirements and expectations of survey professionals as well as the value-for-money (VFM) audit standards of the Office of the Auditor General (OAG). The use of the terms "must" and "should" in this guidance document do not necessarily have the status of OAG standards and policies. However, they reflect methodological requirements and expectations in the conduct of surveys.

What is a survey? The administration of standardized procedures, such as questionnaires or structured interviews, to obtain information on 25 or more individual cases with the intention of making aggregate statements about the matters surveyed.

The aggregate statements based on survey information may be quantitative, e.g. "Only 10 of 20 departments identify priorities systematically and formally," or they may be qualitative, e.g. "The time taken to finalize departmental structures has made planning difficult."

Surveys are used to ask individuals about factual situations, their views and perceptions, and actual behavior. Great caution should be exercised in considering the use of a survey to obtain reactions to hypothetical situations.

Survey techniques, such as questionnaires, can be used when there is no intention to make an aggregate statement. At times, some survey techniques, such as probability sampling, can be used in obtaining factual information from documents and direct observations rather than from individuals, with the intention of generalizing to a population. Although not surveys per se, the use of these techniques in other contexts raises many of the same considerations that apply to their use in surveys.

Requirements

Methodological advice is required because of the critical importance of technical issues. The conduct of surveys is both an art and a highly technical speciality. Advice should be obtained from the FRL Surveys. Advice can also be obtained from expert external advisers and internal methodological advisory committees.

Internal and external advice should be reviewed with the FRL Surveys. Consideration may be given to consulting the FRL surveys when survey techniques are used in other contexts than surveys as defined above.

Consideration should be given to consulting the FRL Quantitative Measurement when quantitative statements are or will be made based on survey information.

The principal (PX) is responsible for ensuring that the audit team has available sufficient technical expertise (see VFM manual, section on Competence of the Audit Team) and experience when surveys are conducted. Expertise and experience are required in the following areas:

• Sample design and statistics;
• Data collection instrument design;
• Data management and statistical analyses;
• Survey administration, including interviewer training, logistics, and managing non-responses; and
• Reporting of survey results.

The FRL Surveys should be consulted when external expertise is sought.

Evidentiary requirements and expectations

Quality Assurance. The PX is responsible for ensuring that appropriate procedures are followed to ensure that survey information is valid and reliable (see VFM manual, section on Sufficient Evidence). The evidence provided by surveys is no stronger than the information sources surveyed. Survey procedures should ensure that conclusions are drawn from survey information that is sufficiently precise, accurate and consistent and are not affected by limitations in:

• Data collection instruments;
• Sampling;
• Survey administration;
• Non-responses; and
• Data processing and analyses.

In particular:

All data collection instruments are required to be pre-tested. Pre-testing is the administration of the instrument to a small set of respondents from the population for the full-scale survey in order to identify potential problems with the instrument.

Procedures for administering large surveys are expected be pilot-tested. Pilot-testing is the implementation of all aspects of the survey, from identifying the sample, through preparing mail out packages, to analyzing data on a small sample from the population. The purpose is to identify potential problems in administering the survey.

Probability sampling is required to be used whenever the auditor intends to make a quantitative statement about a larger population based on the results of a sample (see VFM manual, section on Auditing the Control Systems).

The quality assurance procedures applied and their results should be part of the working papers for the audit.

Bias. The PX is responsible for assuring that data collection instruments, sampling and procedures are not subject to and are not perceived as subject to undue bias. Error is introduced into survey information as a result of limitations in data collection, sampling and other survey procedures. Error can result in unintentional biases and incorrect audit observations and conclusions. Unintended or intended biases can have negative consequences for the validity of survey information and for the perceived objectivity of the Office.

Corroboration. When surveying individuals, consideration is required to be given to the need for an appropriate form of corroboration of written and verbal responses to surveys (see VFM manual, guide section on Sufficient Evidence). The written and verbal responses given by individuals represent their views on the matter at hand. Therefore, individual responses constitute the same kind of evidence as provided by audit interviews. Managers' written or verbal responses to questionnaires on management practices, program results and financial matters constitute their views on these matters. Therefore, the responses of individual managers must be treated as management representations. As a result, the auditor must give the same consideration to the need for corroboration as would be the case with audit interviews and management representations.

Reporting Requirements and Expectations:

The PX is responsible for ensuring that professional practice is relied on in reporting survey findings (see VFM manual, section on Due Care). Professional practice in conducting surveys has evolved a set of accepted practices for reporting survey information. These practices are intended to provide the reader with sufficient information to judge the weight that can be placed on survey information. In an audit context, this information allows an assessment of the extent to which survey information is sound, consistent, objective and contributes to the provision of necessary evidence (see VFM manual, section on Audit Evidence).

Professional practice requires the reporting of the target population, sample size, sampling procedures, survey response rates, estimates of sampling error, frequencies of specific responses to individual questions and the specific wording of questions whose results are reported. It also requires the reporting of potentially significant limitations in survey information or procedures.

"About the Audit" should describe the target population, the method by which the sample was selected, the means by which data was collected, the overall response rate and an overall confidence interval (where appropriate). The description of findings should include actual question wordings for selected questions supporting key findings. The description of findings should also discuss the implications for audit observations of any significant limitations to survey procedures or data. More detailed information on sampling procedures, survey administration, specific question wordings, analyses, response frequencies, error estimates for individual questions, and approaches to dealing with significant limitations in survey procedures and data must be included in substantiation binders.

Professional requirements and expectations relating to respondent burden and informed consent

A set of professional practices have also evolved that are intended to protect survey practice and survey respondents through minimizing respondent burden and ensuring that respondents have sufficient information to understand fully the implications of agreeing to participate in the survey.

The PX is responsible for ensuring that requirements for minimizing respondent burden and for informed consent are met. These include ensuring that:

• a survey is the best available means, considering cost (see VFM manual, section on Sufficient Evidence), for collecting the information required;
• surveys request only information needed to address audit objectives;
• information that identifies or permits the identification of individuals is collected only when absolutely necessary and that measures are taken to ensure confidentiality to the extent possible; and
• questions are non-sexist, respectful of minorities and protect language rights.

Respondents should be adequately advised of the following:

• the purposes of the study;
• the uses to which the information will be put;
• the fact that the Office of the Auditor General is conducting or has commissioned the survey;
• the intent to publish the findings of the survey in the Report of the Auditor General of Canada to Parliament;
• the extent to which the identity of respondents and their responses can and will be kept confidential;
• the name and telephone number of a person to contact if they have questions or complaints; and
• the survey findings.

The auditor cannot guarantee respondent confidentiality if the survey gathers identifying information of any sort. This is due to a number of factors, including questioning of the Auditor General by parliamentary committees or the discovery of fraud, abuse or other illegal behaviors. Therefore, whenever the Office (or any person exposed to the material) can identify individuals from their responses, great caution should be exercised in making any commitment regarding confidentiality or anonymity. Any such commitment must be approved by the responsible Assistant Auditor General.

Questions regarding requirements for the confidentiality of survey information are expected to be brought to the attention of the FRL for Access to Information. The specific wording of any pledge of confidentiality or anonymity and any issues regarding the potential of survey questions to elicit legally sensitive information should be brought to the attention of Legal Services. Procedures for maintaining the confidentiality of individual responses are expected to be discussed with the FRL Surveys.

Publication of survey findings in the Report of the Auditor General of Canada must meet the requirements for informing respondents of survey findings. If findings are not published, consideration should be given to some other means of informing respondents (e.g. management letters, briefing sessions, etc.).

Key Survey Decisions

The audit decisions outlined in the following survey - decision maps - will help the responsible auditor to decide on the appropriateness of the use of a survey, and ensure that the requirements and expectations discussed in this Guide are met.

Survey Decision Map 1: Audit Planning Phase

Survey Decision Map 2: Audit Examination Phase

Survey Decision Map 3: Audit Reporting Phase