Conducting Surveys

Note: This guide is intended to ensure that surveys conducted in the OAG meet reasonable requirements and expectations of survey professionals as well as the VFM audit standards of the Office of the Auditor General. The use of the terms "must" and "should" in this guidance document do not necessarily have the status of OAG standards and policies. However, they reflect methodological requirements and expectations in the conduct of surveys.

Pre-testing is the administration of the data collection instrument with a small set of respondents from the population for the full scale survey. If problems occur in the pre-test, it is likely that similar problems will arise in full-scale administration. The purpose of pre-testing is to identify problems with the data collection instrument and find possible solutions.

It is not possible to anticipate all of the problems that will be encountered during data collection. Terminology used in questionnaires or interviews may not be understood by respondents and information to be retrieved from documents may not be readily available. Reducing error to acceptable levels requires the pre-testing of data collection instruments.

Because standardized procedures are essential for ensuring that general statements can be made, it is advisable to make as few adjustments as possible to data collection instruments once data collection has actually started. In the case of mailed questionnaires, adjustments are impossible once the data collection instruments have been distributed. Pre-testing mail questionnaires or other data collection instruments allows adjustments to be made before full scale administration of the instrument, helping to ensure that standardized procedures are applied during data collection.

If pretesting indicates that there is a low likelihood of obtaining sufficiently sound, consistent and relevant data for addressing audit objectives, troublesome items should be dropped or other techniques for data collection should be pursued.

Principles for pre-testing

Pre-testing should be conducted in circumstances that are as similar as possible to actual data collection and on population members as similar as possible to those that will be sampled.

Careful notes should be taken on the problems encountered and possible solutions should be identified.

Pre-testing questionnaires

One important objective of pre-testing questionnaires is to get at the thinking behind the answers so that the auditor can accurately assess whether the questionnaire is being filled out properly, whether the questions are actually understood by respondents, and whether the questions ask what the auditor thinks they are asking. Pre-testing also helps assess whether respondents are able and willing to provide the needed information.

In pre-testing, the respondents should actually fill out the questionnaire, giving their views along the way or afterward. One approach is to give the questionnaire as an interview, asking for clarification of answers and clarifying questions along the way. The respondents' views can also be obtained during a post-questionnaire interview or in a focus group. Another common approach is to have respondents think out loud as they answer.

Pre-testing allows the responsible auditor to test solutions to problems with the questionnaire. For example, if considering different wordings for a question, one wording can be used with half the pre-test sample and a second wording with the remaining sample to see which works best.