2003 November Report of the Auditor General of Canada
Chapter 5—Management of Public Opinion Research
5.1 We found that public opinion research activity was managed with a certain degree of transparency. For the most part, roles, responsibilities, and procedures were clear.
5.2 However, in some cases departments did not establish a clear statement of the need for undertaking public opinion research. In a small number of troubling cases, we noted that the government had failed to follow its own guidelines in effect at the time and had paid for syndicated research that monitored, among other things, voting behaviour and political party image.
5.3 Many departments subscribed to the same studies, and Communication Canada, the mandatory service provider, did not take advantage of savings opportunities by co-ordinating bulk purchases of syndicated surveys.
5.4 The common service agencies and departments need to improve their management practices to ensure that value for money is obtained in public opinion research activities.
The Privy Council Office, on behalf of the government, has responded. The entities we audited agree with the findings contained in chapters 3, 4 and 5. Our recommendations and the detailed responses appear in the Overall Main Points at the beginning of this booklet.
Government's use of public opinion research
5.5 The guidelines to the Treasury Board's Policy on the Management of Government Information Holdings, in effect during the period covered by our audit, stated that the government had a responsibility to find out the concerns and views of Canadians in order to establish priorities, develop policies, and implement and monitor programs that serve Canadians. Public opinion research is a tool the government frequently uses to capture public awareness and concerns.
5.6 The federal government's public opinion research is the planned gathering of opinions, attitudes, perceptions, judgments, feelings, ideas, reactions, or views—information to be used for any government purposes. The information is collected from persons (including government employees), businesses, institutions, or other entities, using both quantitative and qualitative methods.
5.7 The Treasury Board's policies on contracting, common services, communications, and management of government information holdings apply to public opinion research activities. Public opinion research is a mandatory common service. This means that before entering into any contracts, departments must consult Communication Canada to determine whether the collection of the information they want qualifies as public opinion research.
5.8 The government's policy on communications requires departments to submit their public opinion research plans, strategies, and initiatives to Communication Canada as the technical and co-ordinating authority. Communication Canada provides advice to departments and agencies based on its reviews of proposed research projects. It assigns a project registration number that authorizes Public Works and Government Services Canada to contract for each public opinion research project on behalf of a department or agency. It does so through a competitive process, sole-sourcing, or call-ups from standing offers it has already established—the contracting method used in most cases.
5.9 Within three months of completing their projects, departments are required to send the research reports to Communication Canada for release to the public through the National Library and the Library of Parliament.
5.10 Federal spending on public opinion research. Communication Canada co-ordinated 576 public opinion research projects in 2002-03, at a total cost of $23.7 million. The year before, it co-ordinated 686 projects totalling $26.2 million. The departments or agencies that spent the most were Health Canada, Human Resources Development Canada, Natural Resources Canada, Industry Canada, and Communication Canada—which, like any other department, conducts its own research as well as co-ordinating public opinion research across departments and agencies.
5.11 Custom public opinion research is research commissioned from a private firm and tailored to fit specific research needs of the client department or agency. The government retains ownership of the intellectual property rights on all research and other materials generated by the research project. Thus, the results can be shared within and among departments or made public.
5.12 In contrast, research firms own the intellectual property rights on syndicated studies they conduct: multi-client, cost-shared, public opinion research studies. Departments that purchase subscriptions to these studies cannot share the results with other departments or make them available to the public. At an added cost, they can ask the supplier to add specific questions to a survey. Of the $23.7 million the government spent on public opinion research in 2002-03, $3.1 million was on syndicated studies.
Focus of the audit
5.13 Our objective was to determine whether the government had adequate controls over its public opinion research activities between 1999-2000 and 2002-03. We wanted to know whether, in contracting for public opinion research services, it obtained the best value for the Crown and followed a transparent process that gave equitable access to suppliers of public opinion research services.
5.14 We also wanted to determine whether departments designed their public opinion research activities to achieve specific results. Finally, we wanted to assess whether the systems and procedures in place allowed for a corporate approach to public opinion research and the co-ordination of research activities, as required by Treasury Board policies that applied in the period covered by our audit. Further details are found at the end of the Chapter in About the Audit.
Control over public opinion research
Activities were managed adequately
5.15 Based on our review of a sample of transactions and management practices, we found that the government managed its public opinion research activities adequately. The activities were centrally co-ordinated, as required by policies. Roles and responsibilities in the majority of activities were sufficiently clear; Communication Canada had issued an orientation guide outlining procedures for public opinion research and had developed a research guide to assist departments. In 2001-02, it published an annual report showing the number and value of government contracts awarded to each supplier.
Not all public opinion research reports were released to the public
5.16 For the most part, Communication Canada ensured that results of custom public opinion research were made public, as the government's policy on communications requires. However, where departments delayed forwarding their final project reports to Communication Canada, results were not released within the required three months. Some results were not released at all. Communication Canada regularly tracked the reports that had not been released to the public. Despite its follow-up efforts, each year about 14 percent of custom surveys were not made public.
5.17 Communication Canada explained to us that it had been unable to release the results of a few research projects for the Department of Finance Canada because, according to the Department, it had received only verbal reports and had no written reports on these projects.
5.18 We also noted that in one case, the Canada Information Office had received and paid for a final research report but did not make it public. The research consisted of focus group discussions to investigate the results of the February 2000 survey, Listening to Canadians.
The government could not prepare a strategic plan for public opinion research
5.19 The public opinion research co-ordinator in each department has direct control over a portion of the department's budget for public opinion research. However, program managers can use their program funds and initiate research projects themselves. We found that while the departmental co-ordinators could provide advice, they did not have full information on the public opinion research carried out by their departments; therefore, the plans they provided to Communication Canada did not accurately reflect all research their departments would undertake. As a consequence, Communication Canada was not in a position to estimate the planned government-wide expenditures on public opinion research. The government never prepared the government-wide strategic plan for public opinion research called for in the guidelines to the Policy on the Management of Government Information Holdings.
Some project files did not clearly explain project objectives or rationale
5.20 The usefulness of public opinion research depends on how well the research results meet specific needs. We expected to find clear objectives for public opinion research projects and clear evidence demonstrating that the results were ultimately used. The policy guidelines in effect during the period covered by our audit stated that departments were to define the scope of their research requirements and determine that the need could not be met with information from existing studies. Under the present common service policy, Communication Canada is responsible for reviewing the research projects proposed by departments to confirm that they are consistent with government priorities.
5.21 According to Communication Canada's orientation guide, departments are to prepare a summary of a proposed project that clearly outlines the research objectives, collection methods, target audience, budget, deliverables, and a timetable. Most of the research project files we reviewed showed reasons for the research and suggested how the results might be used. For about 20 percent of the projects in our sample, however, departments did not demonstrate why they needed the research or how they would use the results. Moreover, Communication Canada officials told us they do not believe they have a mandate to challenge the rationale for a proposed project or the clarity of its objectives.
5.22 A survey commissioned by the Canada Information Office and later by Communication Canada itself serves to illustrate. Listening to Canadians is a survey conducted three times a year to gauge Canadians' views about the government's performance on such public policy issues as health and the economy. Communication Canada spent about $3 million on this survey between April 1998 and March 2003. It presented the survey results to departments and posted them on its Web site.
5.23 We expected Communication Canada's files on the survey to contain a clear statement of the need for and purpose of the survey as well as evidence showing how the results were used. We saw some indication that the Privy Council Office may have used the results to develop a government-wide advertising plan. However, the file documentation was largely incomplete, with little to show how Communication Canada had planned for the survey or determined its objectives. In contrast, public opinion research files in Human Resources Development Canada contained project summaries with detailed analyses of the research objectives and rationale.
5.24 We found no analysis in Communication Canada's files to support the need for carrying out the Listening to Canadians survey three times a year, and no evaluation of the survey's effectiveness over the last five years. Communication Canada's files did not demonstrate that the survey provided good value for the cost, as called for in policy guidelines.
Public funds were used to monitor voting behaviour and party image
5.25 The 1995 guidelines to the Policy on the Management of Government Information Holdings stated that "public funds should not be expended on Public Opinion Research concerned with monitoring voting behaviour or party image."
5.26 In May 2001, the Canada Information Office and the Intergovernmental Affairs Secretariat of the Privy Council Office paid a total of about $150,000 to obtain a new syndicated survey that included questions about the voting intentions and images of provincial and federal party leaders, among other topics. The survey also asked about voting intentions in two provincial by-elections.
5.27 In 2002-03, Communication Canada, the Privy Council Office, and Canada Economic Development for Quebec Regions subscribed to the same syndicated study dealing, among other topics, with voting intentions and approval ratings of the provincial and federal party leaders. The three organizations shared the total cost of about $158,000.
5.28 In 2002-03, the government subscribed to a syndicated survey whose questions covered many topics, including the approval ratings of the federal and provincial party leaders. Communication Canada's records showed that eight departments purchased the survey at a cost of about $21,000 each.
5.29 Under the common services policy, Communication Canada has a mandate to not only co-ordinate public opinion research activities but also advise and inform departments about effective practices and policies. We saw no evidence that Communication Canada took any steps to prevent or advise against the purchase of syndicated surveys that monitored, among other things, voting behaviour and party image.
5.30 A change in May 2003 to the Treasury Board Policy on the Management of Government Information Holdings deleted the guideline that prohibited political polling. However, Communication Canada's orientation guide still says, "Monitoring of electoral voting intention(s) or party image is not an activity that can be contracted using public funds." Communication Canada told us that in the absence of Treasury Board guidance, it expects departments to follow its orientation guide and not use public funds in this way. The Treasury Board Secretariat informed us that it will reissue this guidance as part of the guidelines to the Communications Policy.
Many departments purchased the same syndicated surveys
5.31 Communication Canada maximizes the cost effectiveness of public opinion research by co-ordinating the sharing of information and facilitating the pooling of resources for bulk rate reductions—for example, by purchasing core subscriptions to syndicated surveys, allowing for the information to be shared. We found that it had co-ordinated the negotiation of a reduced bulk rate for a small number of syndicated surveys, but not for most. A core subscription to Rethinking Government, for example, costs about $27,000. We noted that 10 copies were purchased by departments in 2002-03 at a total cost of $270,000.
5.32 As a result of the Treasury Board Secretariat's review of public opinion research in 2002, Communication Canada undertook a study of syndicated research (see the section Treasury Board Secretariat initiatives in Chapter 3 of our Report).
Contracting for public opinion research
Selecting suppliers of public opinion research services followed the rules
5.33 In 1997, when PWGSC created its Communications Coordination Services Branch (CCSB), both program co-ordination and contracting for public opinion research were part of its operations. Unlike advertising and sponsorship activities, where there was no segregation of duties, the program co-ordination and contracting activities for public opinion research were managed as two separate functions. In June 2000, program co-ordination was transferred to the Canada Information Office, but contracting for public opinion research remained with CCSB. In September 2001, Communication Canada took over program co-ordination from the Canada Information Office, and a procurement branch of the PWGSC took over contracting for public opinion research from CCSB.
5.34 We expected that CCSB and PWGSC would have contracted for public opinion research in accordance with the Treasury Board's contracting policy. The policy states that government contracting shall be conducted in a manner that will stand the test of public scrutiny in matters of prudence and probity; facilitate access; encourage competition; and reflect fairness in the spending of public funds. We also expected they would have followed the rules for standing offers and properly justified any use of sole-source contracts.
5.35 Our review of a sample of standing offers found that the competitive process had been followed in pre-qualifying suppliers. Requests for proposals were issued, and the selection process resulted in standing offer agreements with the successful bidders. We note that the vast majority of these suppliers were not the same as those providing sponsorship or advertising services. CCSB managed the selection process until August 2001, after which it was managed by a procurement branch of PWGSC together with Communication Canada.
Some departure from rules in managing standing offers and call-ups
5.36 Suppliers were chosen for a term of usually three to four years. At the time of our audit, 12 standing offers and one supply arrangement were in place. Each standing offer could have several suppliers who were qualified to provide the services required (a supplier could pre-qualify for more than one standing offer). Standing offers were established either for use by a specific department or for PWGSC, which made them available for use by all departments.
5.37 Although the period covered by a standing offer was specified in the statement of requirements, we observed that 4 of the 12 standing offers had been extended beyond the specified period. We found no rationale for extending the standing offers. We are concerned that the practice prevented other potential suppliers from competing for new work.
5.38 A call-up limit or threshold on the value of a job is a key element in managing the use of standing offers. PWGSC's guidelines state that each standing offer must have a specified call-up limit; we observed that 8 of 12 standing offers did not. Depending on the size of the project, the call-up limit provides an opportunity for competition because projects exceeding the threshold must be opened for bids. Without a call-up limit, the opportunity for competition is reduced.
5.39 Similarly, Appendix Q of the contracting policy required that on contracts over $25,000, CCSB and PWGSC call for bids from the list of qualified suppliers. The policy stated, "If the contracting authority creates a pre-qualified suppliers' list for a particular type of service, the contracting authority must invite all qualified suppliers on the particular list to submit a proposal in connection with every procurement of that type of service." Instead, many large contracts were awarded to a single supplier based on one bid.
5.40 Some standing offers we examined had several pre-qualified suppliers listed for the same services. We saw no document showing the rationale for awarding a contract to any one supplier out of the several on the list. In October 2002, PWGSC introduced a policy that a system be in place to ensure that call-ups are awarded according to a predetermined criterion.
5.41 For the most part, we found that the federal government was managing public opinion research in a transparent manner and with adequate controls. The activities were centrally co-ordinated, as required by policies. Selection of suppliers for standing offers followed the competitive process. However, PWGSC needs to improve its management of standing offers and call-ups for public opinion research in order to be consistent with the rules in place.
5.42 We found that Communication Canada and departments did not always establish clear objectives for the results of public opinion research projects. We are also concerned about the use of public funds in some cases to acquire syndicated studies on voting intentions and party image. This was clearly in violation of the guidelines then in place.
5.43 Communication Canada needs to continue working with departments to make public more reports on custom public opinion research within the required timeframe.
5.44 Some changes are being made to public opinion research activities. In June 2003, the Minister of Public Works and Government Services announced changes to the Department's contracting process for public opinion research. PWGSC and Communication Canada are working to put in place new standing offers and supply arrangements. The new instruments are expected to have call-up limits and to promote competition among suppliers. We have not audited the new contracting process.
About the Audit
Our audit objectives were to determine
- whether the government has exercised adequate control over its public opinion research activities,
- whether the results of these activities have been measured and reported to Parliament, and
- to what extent the government has taken corrective action as a result of previous audits or reviews.
Scope and approach
We examined the systems and practices used in managing public opinion research activities. We examined a risk-based sample of 30 public opinion research projects from 1999-2000 to 2002-03. We examined files in Communication Canada and in the Canada Information Office, Canadian Heritage, Health Canada, Human Resources Development Canada, National Defence, the Canada Customs and Revenue Agency, Department of Finance Canada, Public Works and Government Services Canada, and the Canadian Tourism Commission when it was part of Industry Canada. We conducted interviews with officials of the departments and of Communication Canada. We also examined a risk-based sample of standing offers and a supply arrangement.
We expect that the government would do the following:
- comply with authorities;
- ensure that public opinion research activities were designed to achieve expected results;
- exercise due diligence in approving individual projects;
- ensure due diligence in spending and account for public funds spent;
- have reasonable assurance that funding was used for the intended purposes;
- appropriately manage the risks inherent in third-party delivery of services, where applicable;
- have a clearly communicated accountability framework in place, including performance management and reporting; and
- conduct periodic review and appropriate follow-up.
Assistant Auditor General: Shahid Minto
Principal: Ronnie Campbell
Directors: Sue Morgan, Louise Bertrand, Johanne McDuff
For information, please contact Communications at (613) 995-3708 or