Conducting Surveys

Note: This guide is intended to ensure that surveys conducted in the OAG meet reasonable requirements and expectations of survey professionals as well as the VFM audit standards of the Office of the Auditor General. The use of the terms "must" and "should" in this guidance document do not necessarily have the status of OAG standards and policies. However, they reflect methodological requirements and expectations in the conduct of surveys.

Well designed data collection instruments and samples will not yield information with acceptable levels of significant error if survey procedures are not properly implemented.

Consistency is important. In order for data collection instruments to yield consistent and valid information, they required to be consistently administered in the fashion intended. In order for a sample to continue to be representative of its population, information must be obtained from as many of those selected as possible.

Training and supervision. Maintaining the integrity of the survey requires training those responsible for carrying out survey procedures and supervision and monitoring of the execution of survey procedures. Two problems that are important to address through training, supervision and monitoring are non-response and interviewer bias.

In large or complex surveys, it is extremely advisable to pilot-test survey procedures to avoid the risk of major problems being undetected until all the responses are in.

Pilot-testing. All aspects of survey procedures are expected to be tested on a small sample from the population of interest. Pilot-tests of mail surveys should include all phases, from sample identification, through preparation of mail-out packages to data analyses. Problems could occur at any point that could threaten the integrity of the survey. Questionnaires could be missing pages due to inadequate controls, addresses in a sampling frame may be out of date and inaccurate, procedures for tracking returned questionnaires may be inadequate for identifying how much of the sample has been returned or for protecting against lost questionnaires, instructions for data entry may be inadequate etc.

Pilot-tests of interview surveys are expected to include all aspects of the process - from interviewer selection and training through to data analysis. For documents or data-based surveys, pilot tests should start with procedures for obtaining documents or data-bases.

In addition to pilot-testing survey procedures, data collection instruments required to be pre-tested, i.e. administered to a small set of respondents from the full scale survey, in order to identify and find solutions to problems with the instruments.