Conducting Surveys

Note: This guide is intended to ensure that surveys conducted in the OAG meet reasonable requirements and expectations of survey professionals as well as the VFM audit standards of the Office of the Auditor General. The use of the terms "must" and "should" in this guidance document do not necessarily have the status of OAG standards and policies. However, they reflect methodological requirements and expectations in the conduct of surveys.

General considerations

Auditors are often familiar with challenges in the collection of information from documents, direct observations, computer data bases and individuals. However, special technical considerations apply when collecting information from individuals as part of a survey.

There are a variety of approaches to surveying individuals, each with their particular strengths and weaknesses. The most common approaches include the following:

• structured or semi-structured questionnaires;
• structured face to face interviews; and
• structured telephone interviews

As well, there is growing experience with the use of newer approaches to contacting individuals, such as the distribution of questionnaires via fax, email and the internet.

Questionnaires

Advantages. The central advantage of questionnaires over interviews is that they allow for the collection of information from a large number of individuals relatively inexpensively. The savings result from the reduced need for staff and, possibly, travel expenses. The savings are most important where a large sample is needed.

Another advantage of questionnaires in comparison to interviews is that they contribute to reliability by promoting greater consistency. This is achieved through eliminating the variation in questioning that can occur when a number of different interviewers are used. They also reduce the introduction of bias by eliminating the ability of interviewers to influence answers either intentionally or inadvertently.

Disadvantages. Unlike survey interviews, questionnaires do not provide an opportunity for the auditor to clarify questions, verify that answers are understood, seek clarification or elaboration of answers or ensure that the respondent answers all questions on the form. There may not be an opportunity to go back to respondents if all the information needed to support a conclusion was not asked for or provided, or if it becomes evident that questions were not clear. Nor is it possible in most instances to seek corroboration of answers. In general, the same depth of information is not available from a questionnaire as through interviews. In addition, the respondents being surveyed must have the requisite literacy skills.

Questionnaire development. Because of the difficulty of obtaining or giving additional clarification and information, careful questionnaire development is essential to ensure that questions will elicit all the required information, and that the questions are clear and unambiguous. In particular, it is essential that the auditor examine closely the audit objectives to clarify what specific items of information are required that can be reasonably obtained through a questionnaire approach.

Furthermore, development of a quality questionnaire requires knowledge of the area being questioned and of the capability of respondents to provide the information required. It also requires that the auditor have enough understanding of the respondents to word questions so that they will be understood.

The amount of advance knowledge is greater to the extent that close-ended questions (yes-no, check off the option, or rating scale) will be used as compared to open-ended questions (fill in the blank, short answer or paragraph answers). The effective use of close-ended questions also requires that the range of possible answers can be correctly anticipated. If respondents do not have the requisite knowledge and if terminology is not clearly understood or defined, there is a heightened risk of incorrect answers.

On the other hand, close-ended questions are more readily tallied and analyzed than open-ended ones. Sorting through a large number of answers to open-ended questions in which respondents will have used widely divergent terminology and may have nearly illegible writing is both technically challenging and time consuming. Many questionnaires will try to strive for a balance between the two types of questions.

Questionnaire delivery mechanisms. Questionnaires can be distributed by mail, in person, or through electronic means, including via fax, email, via the internet or through distribution of diskettes.

The most common means of distributing questionnaires is by mail. When delivered by mail, a questionnaire survey may not be as quick to administer as a series of structured interviews. In addition to a longer development time for questionnaires than for interviews, time must be allowed for respondents to receive questionnaires in the mail, and to answer and return them.

Five to six months is a reasonable amount of time to allow for designing questionnaires and obtaining information using mailed questionnaires for the kinds of surveys conducted by the Office. Properly constructed surveys using questionnaires several pages long sent to samples designed to be representative of several sub-groups in a large population can take 9 months or more to develop and complete. Under ideal conditions, it may be possible to complete a simple survey in seven or eight weeks.

Shorter time frames require the following:

• a short questionnaire (a single page or less) asking straightforward factual questions;
• a census of a population of under 100 (representative samples will generally be larger than this);
• a readily identifiable population (i.e., mailing or courier addresses are readily available);
• the likelihood of an initial response rate of 85 percent to 95 percent (minimizing the amount of follow-up required); and
• respondents who are motivated to respond quickly.

The amount of time needed to distribute and receive answers to questionnaires can be reduced through the use of electronic means. Electronic delivery can also be designed to eliminate the need for manual transfer of responses to analysable form.

The development of electronic formats can be time consuming however. In the case of email and internet distribution, adequate means for protecting anonymity and confidentiality are still being developed. Typically, in an internet survey, individuals can be chosen for a sample only after they have decided to access the site through which the questionnaire is being distributed, making it very difficult to apply probability sampling procedures to a population likely to be of interest in an audit.

In choosing a means of delivery, consideration must also be given to the population that can be reached in that way. Mailed questionnaires restrict the survey to individuals who can be reached by mail. Electronic distribution is practical only for respondents who have access to computers, fax, email, the internet etc. Moreover, incompatibilities in software or hardware may hinder or prevent a response.

The FRL Surveys can assist in estimating the amount of time required to conduct a survey that meets audit needs. Because of the amount of time needed to design and complete surveys, it is important to begin planning and contact the FRL at the earliest possible moment.

Structured Face-to-Face Interviews

Advantages. Face-to-face interviews may be quicker to conduct than questionnaire surveys because it is not necessary to add time to account for mail delivery and for the respondents to turn their attention to the questionnaire. A major advantage is that they allow more opportunity to assess the respondent's understanding and interpretation of the questions and to clarify any confusion that arises about the meaning of the question or the response. They also allow the opportunity to present material to respondents and obtain their reactions. For example, face-to-face interviews have been used to assess the meaning that non-literate subjects attach to symbols. For these reasons, face-to-face interviews are useful for pilot-testing mail-out questionnaires

Face-to-face interviews can be useful in dealing with certain situations that pose challenges for mail-out questionnaires. They are generally better suited than mail or electronic questionnaires with respondents whose reading and writing skills may not be adequate for the questions being asked. They may also be helpful when sensitive information is being sought. Interviewers may be able to establish a relationship of trust with the respondent and be better able to solicit answers to questions which respondents may otherwise be reluctant to answer or to answer truthfully.

Where less is known about the way in which respondents think about an issue or about the range of possible answers to a question, structured interviews create the opportunity for interviewers to ask supplementary questions, when needed to obtain adequate answers.

Disadvantages. However, interviews also create the potential for an interviewer to intentionally or unintentionally influence results and violate consistency in measurement. Survey respondents will be sensitive to cues given by the interviewer's verbal and non-verbal behavior. As well, an interviewer will have to ask additional questions or provide clarifications and may unduly influence responses.

Consequently, adequate interviewer training is essential. Training is needed to ensure that interviewers understand the ways in which they could inadvertently influence responses, the importance of not doing so, and the proper techniques that can be used to assist the respondent or elicit needed information without affecting the integrity of the interview.

Although they may be quicker to conduct than mail questionnaire surveys, face-to-face interviews are costly due to the amount of staff time required to conduct interviews and to the cost of travel.

Structured Telephone Interviews

Advantages. Telephone interviews share some of the advantages and limitations of face-to-face interviews, while reducing travel costs. As with face-to-face interviews, it is possible for interviewers to provide clarification, probe for additional information or more complete answers, and to encourage answers to sensitive questions. Telephone interviews are also faster to conduct than questionnaire and face-to-face surveys.

Disadvantages. Careful interviewer training is still important, as with face to face interviews. Telephone interviews have to be short however, no more than 20 to 30 minutes in length, so that longer and more complex interviews are not feasible. Unlike in a face to face interview, it is not possible for the interviewer to present material and obtain the respondents' reactions. Nor is it possible for the interviewer to look for non-verbal indications of confusion or uneasiness.

Other considerations

Although interviews provide more opportunity for clarification and elaboration than do mail-out questionnaires, careful data collection instrument design is still critical for ensuring a standard approach that will yield consistent information from respondent to respondent. Interview guides must be pre-tested.

In choosing an approach, it is important to consider the distinct characteristics of the population that you are trying to reach. Although interviewers may be successful in generating trust with some types of respondents, other respondents may feel more comfortable with the greater anonymity of a mail-out questionnaire. Some respondents may be impatient with paper questionnaires and prefer the ease of electronic approaches, while others may be intimidated by technology. Pre-testing helps ensure that you have chosen the most appropriate approach.