Conducting Surveys

Note: This guide is intended to ensure that surveys conducted in the OAG meet reasonable requirements and expectations of survey professionals as well as the VFM audit standards of the Office of the Auditor General. The use of the terms "must" and "should" in this guidance document do not necessarily have the status of OAG standards and policies. However, they reflect methodological requirements and expectations in the conduct of surveys.

The purpose of a survey is to gather the information needed to make aggregate or general statements about the topic being audited. Doing this requires gathering information from a sufficient number of individuals with the appropriate characteristics to have a reasonable degree of confidence (assurance) that the general statements are warranted. Choosing the right number of the right population members using the right methods is known as sampling. The decision about the number of individuals to select and how to select them is critical to the survey process. The same is true when the intention is to make aggregate or general statements based on a sample of documents, direct observations, or computer based case records.

Census or sample?

The extent to which a sample represents in kind and proportion the general population from which it is drawn is particularly vital when the intention is to make general statements about the population. The greatest assurance as to representativeness is to conduct a census; that is, select all individuals or cases that make up the population of interest. In many audits, lower, but reasonable, assurance can be obtained by selecting a subset of the population.

A census is appropriate when

• the individual instances that make up a population are important in and of themselves or the information is of a critical nature and the population is small enough to make a census feasible
• when statistical requirements dictate a sample almost as large as the population (when the population is very small or the matter of interest occurs too rarely in a small population for sampling to yield a sufficient number of cases for study; e.g. the performance reports of all FAA Schedule I departments)

Sampling can be more efficient and less costly than a census when the population of interest is very large. However, when the auditor intends to generalize to an entire population, rather than simply describe the sample, the sampling process adds an additional source of error to audit observations. Error occurs because a sample, regardless of how well constructed and implemented, is only an approximation of the population. The resulting error, known as sampling error, requires that the auditor estimate the degree of confidence that can be had as to whether the sample results are a reasonable estimate of the results that would have been obtained from a census of the entire population.

Types of sampling

Sampling approaches are generally divided into two types: probability sampling (also known as random or statistical sampling) and non-probability sampling (also known as non-random, judgmental or purposive sampling).

Probability Sampling. Probability sampling involves selecting individuals using accepted non-biased procedures, thereby helping to ensure that the sample is typical of the larger group (population) from which it is drawn. This is accomplished by using one of a variety of established procedures for drawing sample members at random. When using procedures that ensure randomness, the auditor can estimate statistically the amount of error and the likelihood that error is greater than acceptable limits. This in turn assists judgements as to the appropriateness of the survey evidence. Probability sampling must be used whenever the auditor intends to make a quantitative statement about a larger population based on the results of a sample.

In order to use probability sampling, the auditor must be able to do the following:

• clearly define the population able to provide the information required to address audit objectives;
• be able to identify members of the population for the purposes of sampling (establish a sampling frame);
• have access to individual members of the sample (i.e. they must be reachable by telephone, have mailing addresses, etc. depending upon the approach to be used); and
• ascertain the appropriate sample size.

In some cases, the auditor may not be able to identify all population members, e.g., income tax evaders. Instead it may be necessary to rely on a sampling frame that includes only a portion of the population, e.g. lists of tax filers being pursued for non-payment of taxes. The auditor must decide whether the population represented by an available sample frame is sufficient for meeting audit objectives.

Probability sampling can be a highly technical area requiring the use of staff or external assistance with appropriate training and expertise. Sampling techniques vary considerably in the expertise required to design the sample and to estimate sampling error. Sampling designs become more complex, requiring greater expertise, when

• there is a need to examine differences among sub-groups in a population
• key characteristics of sample members are not uniformly allocated across the population, e.g. there are a very few transactions with high materiality and a large number with a very low materiality
• individual sample members can only be identified based on their membership within different groups, or their characteristics differ according to group membership ( e.g. contracts administered by different departments)

The FRL Surveys can provide advice on options available when complex samples (e.g., stratified random samples) are required, and on the selection of the required expertise.

The results of a well-executed probability sample are objective and defensible and can be replicated. Poor execution in the sampling process or in survey administration can reduce the defensibility of the sampling approach taken and even invalidate the results of the survey. One critical factor is the extent to which information is collected from all members of the sample. Failure to obtain a full response can affect the estimation of error and introduce significant bias. More detail can be found in the sections 10, 11, and 12.

The use of even a well-executed probability sample does not protect against errors in information collection (measurement error). Care must be taken in the development of information collection procedures (e.g. questionnaires or structured interviews) to ensure that the data collected are valid and reliable.

Non-probability sampling. Non-probability sampling may be appropriate when a sample of interest for audit purposes can be identified and the auditor does not intend to reach conclusions about other cases than those in the sample; for example, when the audit will examine management practices for all large transactions, without any intention of applying the audit findings to management practices for smaller transactions. It may also be appropriate when general statements are qualitative rather than quantitative in nature.

For instance, the intention may be to identify the types of views possible without commenting on the frequency with which they are held, e.g. listing the types of problems managers encounter when trying to put in place performance measurement systems without enumerating how often each problem was identified by managers, or how many managers encountered problems.

This approach to sampling may be appropriate

• the auditor can identify a number of individual cases that are material in and of themselves
• the identification of any individual departure from criteria is significant, e.g. in determining vulnerability to fraud
• the auditor is interested in the range of practices or deficiencies possible, and is not interested in enumerating them

The amount of error and the likelihood that error exceeds acceptable limits cannot be estimated as readily for non-probability as for probability sampling.

Non-probability sampling is not an abandonment of systematic approaches to choosing individual instances. It requires careful assessment of the information required and sufficient knowledge of the population for identifying those instances capable of providing the breadth and quality of information required to address audit objectives.

Unacceptable sampling procedures

Haphazard sampling procedures, such as using volunteers, whoever seems typical or whoever happens to be available, constitute neither random sampling nor adequate non-probability sampling procedures. Haphazard approaches are not likely to result in appropriate evidence because a high degree of bias is likely. In addition, the auditor will not know what population these samples represent, and cannot be assured that those who will be sampled represent a population pertinent to meeting audit objectives.