Conducting Surveys

Section 11: Managing, Processing, and Analyzing Survey Data

Note: This guide is intended to ensure that surveys conducted in the OAG meet reasonable requirements and expectations of survey professionals as well as the VFM audit standards of the Office of the Auditor General. The use of the terms "must" and "should" in this guidance document do not necessarily have the status of OAG standards and policies. However, they reflect methodological requirements and expectations in the conduct of surveys.

Error can occur at various points during data collection and analysis. Questionnaire respondents and interviewers can add error while recording information. Error can also be introduced while transferring data from data collection instruments to computerized databases, writing program routines for statistical programs to read and analyze data from computerized databases and transferring data from printouts or readouts into project or audit reports.

Errors in data collection

Major risks to the accuracy of data occur in the initial recording of data. Frequently occurring problems during the completion of data collection instruments include the following:

These kinds of errors may result from the following:

Survey planning, and data collection instrument design are important. Error from some of these sources can be reduced through careful survey planning and data collection instrument design. Careful consideration of the information needed to support audit objectives, restriction of data collection instruments to essential information items, avoidance of sensitive questions and explanation of the purposes and uses of the survey will help ensure an adequate level of respondent motivation. Complete and clear instructions, definitions of key terms used in questions, and careful attention to the structure and layout of the instrument will help reduce missed items and inappropriate responses. Errors in entering information and illegible answers can be minimized by improving motivation and by use of instruments that require simple responses, such as ticking off "yes" or "no" boxes.

These considerations also apply in surveys that require office staff or consultants to review and categorize the information contained in agency documents.

Potential errors can be detected and timely adjustments made by pre-testing data collection instruments.

Interviewer and coder training are vital. In survey interviews, the interviewer can add error through misunderstanding the respondent or incorrectly transcribing the information onto interview forms. In particular, interviewers can intentionally or unintentionally bias the results of interviews. Research has clearly shown that the interviewer can influence a responses through a variety of subtle verbal and non-verbal cues, even when there is no intention to do so. For example, respondents may use an interviewer's smiles or confirmatory "uh huh"s an as cues about what the interviewer wants to hear or believe.

The risk of bias is greater with open-ended questions. Interviewers may have difficulty completely and accurately recording lengthy answers. They will also have to judge when a question has been adequately answered, perhaps asking additional questions for clarification or to ensure a complete response. These circumstances all create the opportunity for bias.

Minimizing these problems requires careful selection and training of interviewers.

Quality control practices. Following up with respondents to verify responses in a sub-set of interviews, having a second interviewer present, or having the respondent verify a transcript of the interview will all help assess the effectiveness of training, detect bias and correct for errors. Conducting a pilot-test will help ensure that interviewer selection and training have been effective.

Error can also be introduced when information obtained from interviews or documents has to be categorized to facilitate analyses. Taking information in a respondent's own words or from an existing document and putting them into categories created by the auditor can involve considerable judgement. For example, in an audit of evaluation in the federal government, auditors reviewed over 600 evaluation reports and made judgements as to whether each evaluation addressed the success, relevance or cost-effectiveness of the program evaluated. Because the reports did not often use this terminology in describing their focus, the auditors were required to judge which categories the evaluations best fit.

In these circumstances, error can be reduced by having each document or interview coded by more than one person and by training coders to make consistent judgements. In the audit of evaluation, each report was reviewed by two auditors. On an initial set of evaluation reports, evaluators discussed their judgements on each report until they reached agreement. This process was repeated until they were agreeing on at least 90 percent of their decisions without discussion.

Data must be checked. It is essential to check and edit each data collection form to identify inconsistent, incomplete or illegible data. Where possible, respondents, interviewers or original data sources should be contacted to resolve inconsistent or incomplete data. When it is not possible to correct these problems prior to analyses, statistical procedures can sometimes be used to estimate and correct for these problems. These statistical procedures are not substitutes for complete and accurate data and require considerable expertise in their application and interpretation.

Transferring information to databases

Error can also be introduced in transferring information from data collection instruments to computerized databases. Errors can result from misinterpretation of difficult to read information, missed items and incorrect keying of data.

Often, questionnaire or interview data on a written form are entered into machine-readable form. Clear instructions to those keying in the data are important for minimizing error. Once entered, data should be verified by comparing a printout of the data with that on an original form. Verification can be conducted on each individual form or, in the case of very large databases, through statistical samples.

Human error in transferring information to databases can be reduced by using computer assisted techniques. These techniques involve entering data directly into machine-readable form, or directly onto a computer by using a computerized form that allows responses to be transferred to a database by software. Although using machine-readable forms is efficient and helps minimize error, the Office does not have the required equipment for scanning these forms, and contracting for the required services can be expensive.

Computerized data collection instruments can be designed with built-in checks. For example, software can prevent a respondent from proceeding to the next question until the previous one has been completed, or can direct the respondent to the next appropriate question based on the response to a previous question. This approach is feasible with respondents who have access to compatible computer systems, with face-to-face or telephone interviews and email or internet surveys.

Analysis

Many errors and deficiencies in data collection can be identified during data analysis. Examining frequency distributions of answers to individual questions and comparing answers to different questions may reveal contradictions or unusual patterns that are due to errors in the original data set or weaknesses in the transformation of data into analyzable form. Those responsible for performing the analyses should remain alert to these possibilities. The FRLs for Quantitative Measurement and Surveys can provide advice.