Conducting Surveys

Non-response refers to the unavailability of sampled units. In a survey, it is likely that it will not be possible to reach all members of the sample. For example, individuals may be unavailable because they have moved with no forwarding address. In probability sampling, non-response reduces sample size, affecting the calculation of sampling error and confidence intervals.

More importantly, in both probability and non-probability sampling, non-response is a source of potential bias. For example, in sampling small businesses that received financial aid from the government, a number of them may be unreachable for a variety of reasons, including the fact that they may have gone out of business. The unavailability of unsuccessful businesses will bias efforts to estimate the success of aid to small businesses and to identify characteristics that could be used to develop more successful funding criteria.

The existence of bias resulting from non-response can be difficult, if not impossible, to determine. For example, sampled agency staff may be unavailable for a variety of reasons, including illness, travel status, or having recently changed jobs. The impact of excluding these individuals may be difficult to assess.

Therefore, it is important that every effort possible be made to capture all members of the intended sample. It is difficult to determine what response rate is acceptable. Although a 85 percent to 95 percent response rate is generally considered adequate, there is little scientific basis for this "rule of thumb", due to the potential for bias. In some areas of enquiry, or with certain populations, response rates are historically lower. In these areas, the auditor will have to balance the importance of the information, the availability of other approaches and the centrality of the information to audit objectives in order to determine whether a survey is appropriate.

Non-response is an important weakness in evidence. Therefore the response rate to a survey must be reported in "About the Audit" or in the text of the report (see VFM Manual, Chapter 5).

Encouraging a high response rate

The auditor can take several steps to try and ensure an adequate response rate.

Motivating respondents. Respondents may be more or less likely to respond to a survey depending upon how important the survey appears to be, how easy it is to complete and the amount of time required. The auditor can maximize respondent motivation by: clearly indicating that the survey is being conducted by or for the Auditor General of Canada; clearly explaining the purposes of the survey and the use to which the information will be put; pre-notifying respondents that they will be surveyed; providing clear instructions; and preparing a short, easy to follow, data collection instrument. Interview surveys are generally considered better than mailed questionnaires for reaching and motivating respondents.

Using an accurate sampling frame. If the sampling frame is out of date or inaccurate, many respondents could be unavailable.

Monitoring and following up on non-responses. Progress in capturing the sample should be monitored. Interviewers should log their efforts to reach respondents, to ensure that potential respondents are not unnecessarily let slip. Multiple follow-ups should be undertaken with non-respondents. In the case of mail surveys, repeated waves of the questionnaire should be sent if the response rate is not adequate and individual non-respondents cannot be identified. If non-respondents can be identified, telephone follow-ups may be helpful. Interviewers should make a specified number of attempts to contact persons for face-to-face or telephone interviews.

Many people feel inundated by requests to participate in surveys. In following up with non-respondents, auditors should be sensitive to their reactions.

Pilot-testing survey procedures. Pilot-tests are important for assessing the adequacy of the sampling frame and for ensuring that sample members or their responses are not "lost" during the administration of the survey.

Adjusting for non-responses

Sampling with replacement. At times, a larger sample than is needed may be drawn, or substitutes for non-respondents may be randomly selected from the remaining population. These procedures are useful in maintaining sample size, especially important where respondents with important characteristics are not frequent in the population. However, cases added in this way are still from that portion of the population which is cooperative and reachable. Therefore, these procedures do not necessarily correct for the bias that may result from the loss of non-respondents. The appropriateness of these procedures should be reviewed by the FRL Surveys.

Care should be taken not to use systematic procedures for replacing non-respondents, such as the next document in order from a file, or the house next door to the one with no one home. Population members sampled in this way may not share important attributes with those unavailable.

Statistical Estimation of the effect of non-responses. Statistical procedures can be employed to estimate the effect of non-respondents or to provide estimates of their likely response. These procedures can be employed if the auditor can determine characteristics likely to influence response, e.g. relative wealth, age, education and if these are known for the non-respondents.