This Web page has been archived on the Web.

1999 November Report of the Auditor General of Canada

Main Points

• They focus on people , recruiting, developing and retaining the right mix of talent in a positive and supportive environment.
• They show leadership , aligning themselves with the needs of those who depend on them for results, achieving buy-in of the vision, values and goals, and undertaking the right research at the right time and at the right investment.
• They manage research to ensure excellence and results, the leveraging of resources, and the capture of organizational learning.
• They strive for a high level of organizational performance, being widely known and respected, and meeting the needs of those who depend on them for results.

22.3 While the attributes describe ideal outcomes, and are therefore not attainable, they are stated so that progress toward the ideal is observable and measurable. In our view, the extent to which an attribute is demonstrated by an organization is a measure of the quality of its management.

Background and other observations

22.5 In Science and Technology for the New Century: A Federal Strategy (1996), the federal government recognized the need for better management of S&T activities and provided new governance mechanisms and a set of general operating principles to improve S&T management across and within departments and agencies. Our work complements and reinforces the Strategy by describing what good management should look like in a research organization.

Introduction

22.7 Of the $5.5 billion, the government spent $3.2 billion or 59 percent on intramural activities - that is, activities carried out primarily by departments, agencies and Crown corporations of the federal government. Of that amount, $1.59 billion (29 percent of the total activities) was spent on research and development - that is, creative work undertaken in a systematic way to increase the stock of scientific and technical knowledge and to use that knowledge in new applications; and $1.64 billion (30 percent of the total activities) was spent on research-related activities - that is, activities that complement or extend research and development by generating, disseminating and applying new scientific and technological knowledge. Good management of these S&T activities is critical for the federal government to achieve expected results.

Our past work on science and technology

22.9 At the government-wide level, we identified a need for clear priorities and direction, clear performance expectations and action plans, effective co-ordination and oversight, leadership that transcends departmental mandates, and better information for Parliament and the public on S&T activities and performance. At the departmental level, we identified the need to set goals that focus more on results, to establish priorities based on an assessment of client needs and opportunities in the sectors, to strengthen human resources planning and the development of research management capability, to establish clear accountability for results and to promote their use, and to improve project selection and review processes and project management practices.

22.10 In 1996, the federal government released a federal strategy for science and technology entitled Science and Technology for the New Century: A Federal Strategy (S&T Strategy), partly in response to our audits. The Strategy outlined three goals for the federal investment in S&T: sustainable job creation and economic growth, improved quality of life, and advancement of knowledge. It also set out government-wide operating principles to guide departments and agencies in the management of their science and technology activities. At the same time, to support the implementation of these principles, the government also released the Framework for the Human Resources Management of the Federal Science and Technology Community (HR Framework). The Framework was designed to help government develop and implement policies and tools that science managers could use to help align their organizations and S&T staff with the science direction and business of departments.

22.11 In 1998, we reviewed the government's progress in implementing the commitments of the S&T Strategy. The 1998 chapter reported that progress in establishing the elements required to improve the management of the federal S&T effort had been slow. The management regime set up to oversee the federal S&T effort was not yet working as intended. Departments we reviewed were making varying degrees of progress in acting on the operating principles enunciated in the Strategy. We identified three areas requiring special attention: planning, priority-setting and performance reporting for mission-driven, results-based research; use of external peer review to ensure scientific excellence; and the development of strategies for the use of partnerships, both external to government and across departments.

22.12 We reported in April 1999 on a follow-up of the management of federal S&T personnel. We concluded that the science and technology community is showing leadership and perseverance in dealing with human resource management issues that we identified in 1994. But considerable challenges remained, particularly in coping with the impact of the current demographic profile, in attracting and recruiting new talent, and in retaining experience.

22.13 Our work to date pointed to the need for a description of what a well-managed research organization looks like. This study uses the guidance provided in the S&T Strategy, the HR Framework and other sources to create a set of ideal outcomes of research management. We call these ideal outcomes attributes. The extent to which an attribute is demonstrated by an organization is an indication of the quality of management.

Management challenges in the research environment

22.15 Generic challenges. Research activities, by their very nature, pose challenges to managers. Whether in the public or private sector, setting priorities and articulating target impacts or end results are difficult. There are several reasons for this:

• There is uncertainty about where the most valuable discoveries lie.
• There is risk associated with scientific uncertainty.
• The nature of research is serendipitous and results are unpredictable, given the pursuit of the unknown (it is not uncommon to have findings that are useful in themselves but are not aligned with expected results).
• The time frames are often long before the outcomes or impacts of research become evident.

22.17 Mission-oriented research organizations conduct research to support their mandates. Their work is a blend of requirements-driven and related exploratory research. The majority of effort is focussed on achieving target outcomes or end-results. However, some effort is invested in exploratory research to identify emerging needs and opportunities and new and better approaches to addressing identified needs. Some of the challenges that managers in mission-oriented research organizations face include: identifying their constituencies and their expectations and developing effective research programs to respond to these expectations; balancing the need to provide researchers with some freedom to explore (to nurture creativity and innovation) with the requirement to deliver expected results; and using appropriate methods to monitor quality and relevance.

22.18 An emerging challenge is the growing use of collaborative arrangements and partnerships involving researchers from different organizations, different sectors and even different countries. Among the factors driving this trend are the need to bring together expertise from different disciplines to tackle complex research questions, and the need to share research costs. As a result, managers need to develop new approaches to managing research, to protecting intellectual property, and to communicating results in a useful form.

22.19 Challenges in the management of government research. The federal S&T Strategy identified the following challenges for managers of government research:

• using foresight (the ability to anticipate future challenges and to make adjustments while staying focussed on long-term goals), risk assessment and prevention practices to address issues before they become problems;
• putting in place effective mechanisms to co-ordinate research activity horizontally across departments and with other sectors (climate change, polar science and biotechnology are examples of issues requiring horizontal management); and
• striving for greater transparency and openness through the regular use of independent peer review for validation of quality, and expert client/stakeholder review for validation of relevance.

22.21 Measures to strengthen the management of government research activities bring managers and researchers face to face with another challenge - namely, preserving the bicultural (scientific and administrative) character of the organization. The research culture is based on strongly held scientific values, many of which are derived from the broader scientific community. Among these is the deep desire to learn and acquire knowledge for its own sake. The challenge for research managers in the public sector is to gain researcher acceptance that government research is first and foremost mission-oriented, not just knowledge-driven, and that to achieve expected results, research must be actively managed.

22.22 Unfortunately, over-emphasis of administrative imperatives and processes can overwhelm scientific values, stunt scientific curiosity, or drive out the very creativity that is needed to address complex research questions. As in most things, balance is needed: managers must recognize and work with the deeply held scientific values of the research culture; and researchers must appreciate and accommodate the administrative imperatives under which managers operate.

22.23 All sectors are facing recruitment, retention and rejuvenation challenges, but these are particularly acute in the public sector at present. Expenditure and work force reductions and expanding opportunities in other sectors have led to demographic profiles that could compromise the government's S&T capacity. Public sector managers are exploring more flexible recruitment measures, appropriate compensation packages, and ways to improve work environments.

22.24 The government identified a new role for itself in the S&T Strategy, namely that of information analyst, knowledge disseminator and network builder. Fulfilling this role effectively requires the strategic management of intellectual capital across the federal system. Intellectual capital is the interplay of the knowledge assets of an organization - the structures, processes and policies that nurture the creation, sharing and exploitation of knowledge. It includes the talent of employees, the management systems, and the relationships with constituencies. Because know-how resides in people, and is reinforced in processes and interactions, an organization needs to have a strategy to manage its intellectual capital and that strategy should align with its mission and overall strategy.

Focus of the study

• to inform Parliament about attributes of well-managed research organizations that can be used to assess the management of research activities in federal departments and agencies; and
• to provide guidance to federal research managers on ideal outcomes of good management, and examples of practices to achieve the performance ideals described by the attributes.

Observations

Definition of Attributes

22.28 Having said that, the attributes are not a recipe for action. They are meant to capture performance ideals that organizations strive to achieve, but will never fully reach. As such, they are offered as statements of the direction that management action should take, rather than as a plan for getting there.

22.29 The attributes are stated so that progress toward the ideal is observable and even measurable. By design, the attributes can be used by research managers in developing performance improvement strategies, and they can be used by auditors and evaluation experts in assessing the management of a research organization. The extent to which an attribute (appropriately modified to suit a specific organization's circumstances) is demonstrated, is a measure of the quality of management.

22.30 The attributes are presented in Exhibit 22.2. They are grouped around four key perspectives on organizational success: people focus, leadership, research management and organizational performance. These perspectives tie together the attributes around themes we identified in the course of our work. Because the first three perspectives produce the fourth - organizational performance - they are critical in their own right and demand an equal amount of management attention. Conversely, organizational performance directly impacts the other three perspectives. For example, organizational success nurtures a focus on people, strengthens leadership and facilitates research management.

22.31 Similarly, the individual attributes are interdependent and work together to provide an overall picture of good research management. While there is overlap among the attributes, each one sets out an ideal outcome for a different dimension of the management of research.

22.32 We have used the word "right" several times in the attributes. The idea is that when management is faced with choices, some are better than others and one, the right or ideal one, is best of all. Management is responsible for deciding what the right option is, and does that by reference to relevant standards - in particular, its vision, goals and values.

Approach to the Development of Attributes

22.34 We based our preliminary set of attributes on the operating principles of the government's S&T Strategy, the ingredients of the HR Framework, the quality management criteria developed jointly by the National Quality Institute and the federal government, and the attributes of effectiveness developed by CCAF-FCVI Inc. (a Canadian research and educational foundation dedicated to building knowledge for meaningful accountability and effective governance, management and audit.)

22.35 We refined the attributes through fact finding and analysis and by subjecting the evolving attributes to internal and external challenges. We conducted a review of the research management literature and analyzed several documents and reports published by the federal government. The following federal resources significantly influenced our thinking:

• the report of the Independent Review Panel, Modernizing Comptrollership in the Government of Canada, which emphasizes open, accessible, value-driven and results-oriented government;
• a document under development by federal science-based departments entitled Best Practices for the Conduct, Management and Use of Science in the Government of Canada;
• Guide to Good Management 1998 - An Evolving Approach, recently published by Natural Resources Canada;
• the recently released Council of Science and Technology Advisors report entitled Science Advice for Government Effectiveness; and
• draft S&T management core competency profiles from the federal working group on the management of S&T, and the National Water Research Institute's Development Resource Guide for R&D Management.

22.37 Our evolving set of attributes was subjected to various tests and challenges. We sought input from our own advisory committee, from research managers at different levels in federal departments, and from other knowledgeable sources such as the American Association for the Advancement of Science, the National Science Foundation, and the U.S. General Accounting Office.

22.38 We visited a selection of known and respected R&D organizations in Canada and in the U.S. to test the attributes and to obtain examples of supporting practices. The organizations we visited were the following:

• Alberta Research Council
• Argonne National Laboratory, U.S. Department of Energy
• Army Research Laboratory, U.S. Department of the Army
• Office of Research and Development, Environmental Protection Agency
• Goddard Space Flight Center, NASA
• Merck Frosst Canada and Co.
• U.S. National Institute of Standards and Technology
• Nortel Networks

22.39 We assumed that research managers in the federal government were generally aware of each other's practices but might find it helpful to learn about practices in other organizations in Canada and the U.S. Consequently, we used practices from outside the Canadian federal system to illustrate ways and means of moving in the directions laid out by the attributes. These practices were described to us as what each organization expected its managers and staff to do, not what always happened.

22.40 Our work suggests that the attributes are applicable to both private and public sector organizations. The emphasis that is placed on any one attribute, and the practices that support progress toward an attribute, vary with the mandate and mission of the organization. This is particularly true for public sector organizations where mandates can cover a very broad array of responsibilities under the heading of public good.

22.41 In the remainder of the chapter, we describe each attribute, and provide examples of practices from organizations we visited.

People Focus

Management knows what research and other talent it needs to accomplish the mission, and recruits, develops and retains the right mix of people

22.43 Management must decide on the core competencies that are needed to address current and evolving research priorities. The required core competencies are not only those needed to conduct high-quality research but also those needed to access and synthesize knowledge produced by others and use it effectively, to manage research performed internally and externally, and to integrate science into policy advice and standards/regulations development.

22.44 Merck Frosst views recruitment as a long-term investment in the future of the company. Consequently, its approach includes science literacy outreach activities in the elementary and high schools, a well-developed co-op university student program, and tracking of high potential students through graduate school. Merck Frosst treats recruiting as among the most important functions of management. If this is done well, everything else falls into place.

22.45 At Nortel Networks, we heard that hiring and retaining the best people are essential to maintaining its leadership position in a highly competitive environment. Nortel looks for a good balance of "mavericks" and team players.

22.46 At the Goddard Space Flight Center, the importance of building a complement of strong researchers was emphasized, particularly in view of the high degree of mobility of talented research staff today. Building a strong complement involves both recruiting "superstars" to spark creativity and innovation, and growing excellence from within by building on the enthusiasm and motivation of existing talent and their willingness to learn.

22.47 In 1997, the Alberta Research Council put in place a strategic hiring program as an investment in the future. It set aside $1 million (3.5 percent of the salary and benefits budget) to increase its core competencies and develop new capabilities, as well as expand its business development and entrepreneurial expertise. It continues to use this program to recruit and maintain its core competencies.

22.48 Nortel Networks invests in the development of its research managers. Management recognizes that the skill set of a research manager differs from that of a researcher. Good research managers must not only understand the culture of research and have technical competence but also have well-developed leadership and management skills, people skills and communication skills.

22.49 The Army Research Laboratory (ARL), along with a number of other U.S. Defense Department laboratories, recently embarked on a new alternative personnel demonstration system designed to provide the ARL Director with increased flexibility to manage the work force. Changes had been called for in many studies of the Defense labs by external review panels over the years. Studies unanimously pointed to the inflexibilities of the U.S. civil service personnel system as limiting the labs from attaining their full potential as world-class research institutions. The new approach encompasses the following benefits: a compensation system consisting of broad bands that are more competitive with the private sector at the entry level, and within which employees can be moved with much greater flexibility; a pay-for-performance system enabling the appropriate rewarding of outstanding performance; a streamlined job classification system; enhanced training opportunities; and simplified discipline and separation procedures.

22.50 To maintain technical excellence and move employees through career levels as their skills and abilities permit, Argonne National Laboratory has adopted a career management system based on competencies for its scientific and engineering staff. The system allows individuals to measure their growth and plan for their professional development. The system is also used as the basis for the compensation program. Salaries are based on the requirements of the position and the outside job market, which determines the competitive salary range for similar positions in the research and development field. The United States Army Research Laboratory has the following perspective on the need to maintain in-house competencies. "If the Army is to be able to intelligently acquire the complex technological developments for tomorrow's battlefield, it must have a cadre of people that understand both the technology and what the private sector is offering to deliver, and can then evaluate what is delivered to assure that it can do the job."

22.51 We found that research organizations use alternative approaches to recruitment (for example, secondments and use of contractors), build bridges to future talent pools (for example, outreach activities with schools, colleges and universities) and mechanisms to respond to shifts in core competency requirements (for example, transition assignments, retraining and outplacement).

22.52 Other examples of constructive initiatives that we found from our research and visits to organizations include:

• use of parallel career paths (research and management) with opportunities to move back and forth to develop new leaders with appropriate skills in both areas;
• continuous learning opportunities (exchanges, deployments, attendance at professional conferences, and opportunities to participate in international working groups);
• peer recognition and awards (publication in scientific journals, patents, membership on prestigious external committees and internal award program);
• incentives for being creative and innovative (special fund for new initiatives or special projects to encourage the best to pursue promising new areas);
• training programs that are aimed at developing researchers' non-technical skills such as communicating and collaborating with others; and
• strategies for identifying, attracting and developing the required talent, such as:
  • sponsoring university research to solidify linkages with universities;
  • engaging in collaborative research projects with universities to stay at the cutting edge;
  • coaching and mentoring of young researchers by more experienced researchers;
  • using career management planning to ensure appropriate progression; and
  • putting in place succession plans and developmental programs to equip potential managers with the necessary skills to move into key positions.

Employees are passionate about their work, have confidence in management, and are proud of their organization

• trust management;
• are treated with respect and feel valued and relevant;
• have opportunities to learn, upgrade their knowledge and skills, and reach their full potential;
• appreciate and can deal with change, including the termination of a research project due to changing priorities or more attractive opportunities;
• are encouraged to contribute ideas and feel free to speak out on issues of concern without fear of retribution (trust and openness);
• are provided reasonably timely feedback on suggestions and requests;
• receive reasonable empathy concerning challenges that they face on and away from the job; and
• believe that their overall compensation and job-related benefits are fair and reasonable.

22.55 Employees tend to be passionate about their work and proud of their organization if they feel that they are making contributions to the success of their organization, are recognized for their contributions and are empowered to achieve agreed results. This requires a work environment where the roles are reasonably clear, and teamwork and participative management are results-oriented. Ensuring conditions such as these requires proactive management. Use of organizational climate surveys can assist management in gauging the health of the work environment. For example, in 1996, the Environmental Protection Agency's (EPA) Office of Research and Development (ORD) launched a new Strategic Plan that introduced a number of significant changes to how ORD conducted its business. Through a series of workshops on managing change, staff identified issues in five areas for improvement: reduce red tape; communications; career advancement and development; resources and infrastructure; and integrate science with the EPA's mission (put science first). Exhibit 22.5 provides an overview of the lines of inquiry used in the organizational climate survey.

22.56 Prompt response by management to survey results is important to maintaining employee confidence. The Alberta Research Council (ARC) measures employee satisfaction and opinions with annual surveys. Survey results, including all employee comments, are reviewed by senior management for action and follow-up discussion with managers. Overall survey results are also shared with all employees at the annual all-employee meeting and are posted on an internal ARC Web site. ARC has set a goal to increase employee satisfaction from 67 percent in 1997.

22.57 Employee confidence in management and pride in the organization are nurtured through transparency and openness. For example, Nortel Networks aims to achieve this by:

• establishing a climate of security - for instance, because the failure or termination of a project does not mean the end of a researcher's career, researchers feel secure in suggesting that a project be stopped without fear of losing their job;
• encouraging openness by sharing a reasonable level of information concerning company plans and strategies with employees; and
• delivering on promises or explaining why not.

22.59 Other examples of constructive initiatives that we identified include:

• Managers are open and frank about the challenges facing the organization and strategies for dealing with them.
• Management seeks and acts on employee feedback in a timely manner.
• Senior management is visible and interacts with staff (for example, participates in celebrations, awards ceremonies, retirement parties).
• Management recognizes employee participation in activities that enhance the organization's reputation, fostering pride in the organization (outreach, science literacy, expert committees, and conferences).

Leadership

The current and anticipated needs of dependent constituencies drive the organization and its research programs

22.61 Because the very existence of an organization depends upon its ability to respond effectively to constituency needs, the clear definition of dependent constituency groups and their needs is important in order to channel efforts and resources into achieving what is relevant and important to these groups. For public sector research organizations, the government itself is a dependent constituency. The results of government research are increasingly inputs to the development of policy, regulations and standards.

22.62 Both government and private sector research organizations focus most of their effort on research aimed at achieving target outcomes or end-results; they direct some effort at ``exploratory" research to identify yet unknown needs and opportunities (from their constituencies' perspectives) as well as to identify new and better approaches to addressing known needs.

22.63 Research planning involves assessing the scientific challenges associated with achieving target outcomes, and determining the research thrusts and programs to address those challenges. It also involves assessing the requirement for exploratory research to better define issues, identifying the approaches to addressing the scientific issues, and aligning current research with agreed priorities. Some of the exploratory research should lead to the initiation of significant research programs.

22.64 As part of anticipating needs and opportunities, research organizations have mechanisms for identifying:

• end-use issues and trends as well as relevant and emerging scientific developments; and
• new constituencies who would benefit from the research organization's expertise and research findings.

22.66 The participation of research staff in planning activities strengthens their understanding of constituency needs and promotes alignment of research activity with those needs. Their participation also alerts management to organizational policies and procedures that may need adjustment to support the achievement of research goals. At Merck Frosst, the research staff participate actively in determining the exploratory and requirement-driven research that responds best to the objectives established by corporate management.

22.67 The NASA Strategic Roadmap and Performance Plan are good examples of research plans that focus on outcomes, contain precise goals and performance expectations, and use plain language so that employees at all levels understand how the research, overall and at the project level, addresses constituency needs. Exhibit 22.7 provides an extract from these documents that illustrates goals specific to the Goddard Space Flight Center.

22.68 Other examples of constructive initiatives that we found include the following:

• Research organizations engage dependent constituencies in reviewing past successes and failures, and identifying opportunities for improving the timeliness, usefulness and impact of the research organizations' input.
• Research organizations engage dependent constituencies in planning research programs and projects as well as reviewing progress and making the necessary adjustments to plans.

Employees and dependent constituencies share management's vision, values and goals

22.70 It is easier to point to people who were or are leaders than to describe leadership. Nevertheless, by describing leadership from a number of angles, a clearer picture emerges.

• Leadership functions include creating and promoting values and expectations, setting directions (the vision), projecting a strong constituency (client/stakeholder) focus, aligning the systems, policies and resources with the vision and mission, and empowering employees to be productive.
• Leadership in a research environment recognizes that researchers are highly trained professionals who are guided in what they do by the standards, expectations and structures provided by the research disciplines in which they work. Leadership is less about directing and controlling, and more about establishing and promoting a shared vision and shared values, establishing a pathway and harnessing the organization's talent and resources to achieve the vision, supporting research professionals by creating and maintaining a positive research environment, and building relationships with dependent constituencies (those individuals, groups or organizations that depend upon the organization's work).
• Leadership provides the conditions under which researchers can exercise creativity. Leaders model the values they espouse, such as respect for individuals, trust, integrity, honesty, openness, transparency and a reasonable work/life balance. They remove roadblocks that hinder performance and adopt organizational policies that support the research environment (for example, policies on intellectual property management, scientific publication, and attendance at conferences).
• Leaders serve as examples and sources of inspiration. A leader's actions say more about what he or she values than do slogans and pep talks. Integrity and trust are established when a leader's actions are in harmony with personal and organizational values. Senior executives' behaviour and actions shape the environment within the organization and its relations with collaborators and dependent constituencies. Their actions, in particular, influence the behaviour of managers, teamwork among scientists and staff, and the sharing of knowledge and learning - conditions that are essential for creativity and innovation in a research organization.

• Merck Frosst believes that skilled leaders are needed at all levels to enable the organization to deal with complexity, to have a productive work environment and to recognize the importance of a work/life balance. Consequently, leadership skills training is being provided to all employees, not just senior executives. The implementation of leadership principles is reinforced by including leadership in the performance management system.
• The Environmental Protection Agency's Office of Research and Development is conducting an exercise with input from staff at all levels to describe its purposes, values and envisioned future. This is the first step in the development of its Strategic Plan 2000. The objective of seeking employees' input is a step to achieving their buy-in to the Strategic Plan.
• The U.S. Army Research Lab (ARL) uses a "planning thread" concept to help ensure that the research activities of its scientists are aligned with the organization's mission and vision. ARL's Strategic Plan describes the ARL mission and vision and lays out the Army requirements and needs in the short, medium and long terms. For each of ARL's primary mission areas, a statement of strategic intent is followed by several long-term goals, each associated with a desired outcome. From these, a series of short-term technical goals are expressed in the annual performance plans. By following the planning thread, a researcher can see where his/her work connects to the overall mission of ARL and to the Army.
• At NASA's Goddard Space Flight Center (GSFC), managers meet with their employees to discuss how their individual performance contributes to the achievement of NASA's Performance Plan and GSFC's Performance Plan. Exhibit 22.8 provides an example of the tool used to summarize this discussion.

The portfolio of programs represents the right research, at the right time and at the right investment

22.74 The following are examples of criteria that can be used to manage research portfolios:

• importance to the dependent constituencies;
• fit with the organization's mission, goals and overall priorities;
• need for involvement (for example, no one else is doing it, cannot rely on other research performers);
• benefit-cost considerations (for example, significance of the opportunity/problem, urgency, potential impact if successful, required level of investment);
• likelihood of success (based on technical considerations, timeliness, required versus available expertise, affordability); and
• overall balance within the portfolio (for example, mission-oriented versus relevant exploratory research, short-versus longer-term target impacts, level of risk, support among various dependent constituencies).

• independent evaluations of programs;
• maintaining a lookout for attractive new opportunities; and
• management-driven reviews of portfolios that assess the performance of current programs and compare existing with potential new programs, and that result in identifying and making the necessary changes.

22.77 The National Institute of Standards and Technology (NIST), for example, contracted the U.S. National Research Council Board of Assessment to assess the technical quality of the Measurement and Standards Laboratories. The focus of the assessment was on:

• the technical merit of the laboratory programs relative to the current state-of-the-art worldwide;
• the effectiveness with which the laboratory programs are carried out and the results disseminated;
• the degree to which the laboratory programs are meeting the needs for which they are intended; and
• the adequacy of the laboratories' facilities, equipment and human resources insofar as they affect the quality of the technical programs.

22.78 Some U.S. government laboratories are operated by a contractor. The agreement with the contractor requires the use of an independent review process. For example, the University of Chicago manages the Department of Energy's Argonne National Laboratory. A board of governors provides guidance, oversight, direction and advice to the Laboratory management. Program oversight is carried out through a Science and Technology Advisory Committee that oversees the independent review process. The review committee assesses the quality of the staff and its performance during the year, the quality and timeliness of the programs and, to the extent that members feel appropriate, the relevance of the work to the long-range goals of the Laboratory and the missions of sponsoring agencies.

22.79 The Environmental Protection Agency (EPA) uses annual research program reviews, jointly organized by the Office of Research and Development (ORD) Research Co-ordination Teams and EPA's Program and Regional Offices, to present to EPA senior managers the entire ORD research portfolio in a given area. These joint reviews focus on the status and accomplishments of the ORD research program to ensure that ORD's research continues to meet its own and client objectives. The reviews also present selected ongoing research conducted by the program offices and regions so that the complementary research can be viewed. These reviews complement external peer reviews of ORD laboratories.

22.80 Other examples of constructive initiatives that we found include the following:

• Dependent constituencies are collectively asked to provide comments on existing or proposed portfolios to develop acceptance of the level of effort devoted to the various constituencies.
• Results from traditional scientific peer reviews are used as input to portfolio reviews (for example, to assess progress and likelihood of technical success).
• Research organizations undertake structured reviews of their portfolios, whereby programs are compared based on criteria that reflect their vision, goals and overall strategies and priorities.

Research Management

Research projects embody excellent science, involve the right people, are on track and within budget

22.82 This attribute includes:

• ensuring that the project produces and is based upon excellent science and technology and that it stands up to the scrutiny of world-class experts;
• involving the appropriate and best available persons in the planning, conduct, technology transfer and review of projects (recognizing that different persons are likely to be involved at different times in the life of a project);
• periodically reviewing projects using increasingly demanding assessment criteria over time (referred to as stage-gating). These reviews normally include assessments from both a technical perspective and the dependent constituent's perspective;
• ensuring that research equipment and facilities are appropriate to the nature and requirements of the projects;
• ensuring that the research results are communicated and/or transferred to all relevant constituencies. This usually requires the involvement of team members beyond the completion of the research; and
• managing the project, including the timetable and the budget, on a continuing and active basis to reflect the dynamics of the research environment.

• Nortel Networks uses a project plan framework to monitor both fundamental and applied research projects and development projects. The framework includes the following elements: project scope, description of tasks, key assumptions, criteria for success, resources, milestones (stop/go gates), risks and contingencies. Information on these elements ensures that resources (human, equipment, financial) remain focussed on priorities and are well co-ordinated, and that a proper balance of reporting and activity is maintained.
• The type of expert review used to assess research projects varies with the nature of the research. Projects that are more short-term and developmental, with more predictable results, are best suited to reviews with quantifiable metrics (results-oriented milestones and performance expectations derived from the business plan). Projects of a more long-term and fundamental nature, with more unpredictable results, are best reviewed through peer review of quality and leadership. Even projects involving proprietary information are subjected to expert review. In these cases, procedures are modified to maintain confidentiality. For example, Merck Frosst calls upon scientists from other parts of the global Merck family of companies to undertake peer review while protecting commercial confidentiality.
• The Alberta Research Council recently adopted a stage-gating process with well-defined decision-making criteria, based on technological and market considerations, to better manage research projects from the idea stage to the commercialization stage. ARC has developed a formal decision-making grid, using a proprietary tool, to ensure due diligence in its investment decisions. The factors comprising the grid include technical advance, technical capability of the team, project management, strategic fit, market size, market accessibility, partner capabilities, probability of success, intellectual property position, return on investment to ARC, and impact on Alberta. For proposals in the early stage, the review is usually done within the business unit; however, as the project moves toward development and commercialization, external reviewers are involved.
• Researchers at Merck Frosst meet weekly to review the progress of major projects. When a project is not meeting expectations, managers take appropriate action and move staff resources to higher-priority projects. They encourage researchers themselves to recognize when a project should be abandoned, rather than imposing a top-down decision. In other words, it is recognized that results that do not meet expectations are a normal output of a risky activity such as research, and do not constitute a personal failure. Such findings may in fact lead to very useful insights even if they do not move the project toward the expected results. In this respect, some low-level activity may continue even if the project is no longer of priority, which serves to keep options open.

• Proposals are solicited from within the organization to identify attractive projects that should be funded, and that might otherwise not be.
• Plans for deployment of the research findings are included in project plans. Furthermore, the knowledge transfer process begins at the project planning stage by involving dependent constituents.

Research projects leverage external resources

22.86 Research problems are becoming increasingly complex. Expertise from several disciplines and organizations is frequently required to effectively address the issues. As a result, private and public sector research organizations are increasingly turning to collaboration and partnerships to leverage their own expertise and resources. In government, issues are increasingly cutting across departmental boundaries (for example, climate change). Leveraging increases the likelihood of achieving objectives by optimizing the impact of available human and financial resources and bringing to bear the best expertise from a variety of sources. It also helps focus the research organization on the target end-result and expedites the technology transfer process.

22.87 The U.S. Army Research Laboratory's Federated Laboratory (FedLab) initiative (see Exhibit 22.9) demonstrates a number of the dimensions of this attribute. These include a novel delivery arrangement that focusses on ARL's strengths and brings in complementary expertise from other partners to achieve the Army's goals; use of a competitive approach to finalize a suite of highly innovative projects that address an identified need (project proposals are subjected to a competitive selection process involving external peer review); and the involvement of external partners in project monitoring and reporting processes.

22.88 Scientists and engineers of the Environmental Protection Agency's Office of Research and Development contribute to every stage of EPA's risk assessment and management process. This develops and maintains effective relationships between researchers and program and regional staff. The research staff not only identify and characterize environmental problems (risk assessment) but also help other parts of EPA to find and implement efficient, cost-effective solutions to the problems. They are involved in identifying risk management options, evaluating their performance, cost and effectiveness, and monitoring improvements.

22.89 Other examples of constructive initiatives include the following:

• Leverage is used as one of the criteria in selecting and comparing both projects and programs.
• Graduate students at universities are provided funding to undertake exploratory research in areas of interest to the research organization and its dependent constituencies.

Organizational knowledge is systematically captured and turned into needed work tools

22.91 Knowledge is the key asset of a research organization. Organizational knowledge encompasses personal and collective knowledge (know-how, expertise, experience and wisdom). Management of organizational knowledge, sometimes also referred to as intellectual capital, pulls together the knowledge that is dispersed throughout the organization, making it accessible and usable by others inside the organization and by its dependent constituencies. It involves the continuous recycling and creative use of shared knowledge and experience. It promotes continuous learning on an organization-wide level by ensuring effective use of existing knowledge, and sharing of new knowledge and lessons learned from past experience. Management of organizational knowledge ensures that competence remains with the organization even when individuals leave.

22.92 There are two key facets to the development and effective management of organizational knowledge. The first is processes to capture and tend the knowledge, to make it available to others, and to keep track of who is contributing to it and who is using it. The second facet is an organizational culture that values the sharing of knowledge.

22.93 There are natural barriers to the sharing of knowledge that result from rigid organizational structures, dispersed locations, turnover of staff, specialization within disciplines, and the "not-invented-here" syndrome. Leadership and the alignment of policy, incentives and performance measurement with the right organizational values can help overcome barriers.

22.94 The nurturing of "communities of practice" - informal networks of individuals with common interests who come together spontaneously and choose to communicate across the rigid organizational structures - promotes organizational learning. They do this by improving understanding of concepts; increasing awareness of techniques available inside the organization or elsewhere; facilitating learning and sharing of experience; providing a forum for the peer review of ideas, theories and interpretation of data; and enhancing technology transfer. Management can support communities of practice through the provision of resources to facilitate activities.

22.95 Nortel Networks recognizes the value of networking for organizational learning. Among the methods used to promote the sharing of knowledge and experience are social events, co-location of project staff, informal coffee groups, formal events to discuss issues, and widely disseminated formal reports on activities. Nortel managers identified networking on a global basis as a more difficult challenge. Quarterly get-togethers and videoconferencing are techniques being used to address this challenge.

22.96 To better capture and share knowledge among scientific and technical staff on an organization-wide basis, the Goddard Space Flight Center is implementing the ISO 9001 standard. Processes, procedures and the know-how developed by individuals are being documented in accordance with the requirements of the standard. While ISO 9000 standards and guidelines are generally associated with organizations in the manufacturing sector, NASA is leading the way in the application of the standards to an R&D organization.

22.97 Other examples of constructive initiatives include the following:

• Policies, incentives and performance metrics are aligned to reinforce collaboration and knowledge sharing.
• All levels of management emphasize and practice developing trust and collaboration among organizational units, focussing on win-win outcomes.

Organizational Performance

The organization is widely known and respected

22.99 An organization commands respect when knowledgeable observers (individuals in the same or complementary areas who are not necessarily among the organization's constituencies, such as world-class researchers in other countries) judge that its science output is of high quality and at the leading-edge. Respect results from an organization's ability to maintain its reputation for excellence over a long period of time and to have its research staff sought after as participants in collaborative activities and partnerships and as members of prestigious committees. The participation of a significant fraction of the organization's research staff on international committees charged with the development of international standards and regulatory regimes provides an indication of the organization's credibility and reputation. For example, the National Institute of Standards and Technology (NIST) tracks staff participation in and chairmanships of standards committees and international committees.

22.100 An organization is also respected when dependent constituencies believe that it is performing an essential service, is accessible and responsive to their needs, is reliable, demonstrates flexibility, and is service-oriented. Government policy makers and regulatory bodies depend upon public sector research organizations for high-quality scientific advice in the development of policies and regulations intended to protect the public good. To assure decision makers that advice is based on current and sound science, public sector research organizations are adopting a number of principles: early identification of issues requiring science advice; broad consultation on issues with experts in many disciplines and sectors; use of due diligence procedures to assure quality and reliability of the science; recognition of the limitations and risks resulting from scientific uncertainty; transparency and openness through the broad dissemination and publication of scientific evidence and analysis underlying policy/regulation; and periodic review of decisions in light of new advances in science.

22.101 We found that research organizations take steps to track their reputation:

• The Alberta Research Council monitors customer satisfaction annually by surveying how well its services meet customer requirements, and how important the services are to the customer. The survey helps management set improvement targets.
• The U.S. Army Research Laboratory also uses customer feedback to ensure quality, relevance and timeliness. ARL is exploring going beyond customer satisfaction to "customer value" - that is, "capturing the hearts of customers to the point that they will not only be satisfied when they receive your product, but they will go out of their way, and even sacrifice time, money or convenience, to come back to you the next time." Through a customer value survey, ARL identified "communication with the customers" as the attribute most valued by customers. Communicating with the customers includes talking and listening to them, visiting them, keeping them informed, and involving them in planning processes.
• Argonne National Laboratory values the confidence and support of its customers, stakeholders, neighbours and employees. Argonne uses many communication and outreach activities to enhance its reputation, visibility and image (science and technology publications, participation in technical conferences and workshops, public affairs, education and industrial technology programs). To enhance community relations, Argonne established a Community Leaders Roundtable that involves over 30 community leaders from towns, homeowner associations, governments, businesses and public-interest groups. The Roundtable provides a forum for ongoing dialogues about the Laboratory's activities, the expected impacts on the surrounding area and any community concerns.

• Feedback on the organization's credibility and reputation is obtained from experts, such as internationally renowned scientists, industry analysts, environmentalists and consumer groups.
• Feedback from surveys and expert reviews is shared with employees and advisory boards, with a view to identifying ways to improve organizational performance.
• Research staff is encouraged to accept invitations for membership of expert committees, to participate in external peer reviews, and to undertake collaborative research.

The organization meets the needs of its dependent constituencies

22.104 To ensure and demonstrate the achievement of this attribute, research organizations take measures to promote understanding of research findings by constituencies, and assess the relevance, timeliness and significance of their research for the purpose of identifying opportunities for improvement.

22.105 Networking and personnel exchanges with constituencies are among the best ways to promote effective communication and understanding of results. Providing opportunities for people to exchange ideas is more effective than simply publishing results in various outlets. Such opportunities include researchers talking to other researchers, users interacting with researchers, policy and regulatory staff interacting with research staff, scientists participating on national and international standards-setting committees. NASA has set an agency-wide objective "to improve the external constituent communities' knowledge, understanding, and use of the results and opportunities associated with NASA's programs." The Goddard Space Flight Center has set specific performance goals to contribute to the agency-wide objective:

• Increase the number of visiting scientists, resident research associates, graduate students participating in Goddard programs by more than 10 percent over the 1998 fiscal year level.
• Increase by five percent the number of science research co-op agreements with universities, including predominantly minority universities.
• Enhance the quality of knowledge communicated to the public and to the media by involving principal investigators and project managers in all mission-related media activities.
• Establish collaboratively, with the web-based managers in the technical community, guidelines for enhancing public access to web-based information.

• Partners are involved in planning, conducting and managing research projects through a Consortium Management Committee.
• Personnel exchanges among partners are required as part of the consortium agreement.
• An annual technical symposium involving all participants is held to present results, and discuss with receptors.

• The U.S. Army Research Laboratory's Performance Evaluation Construct recognizes the complexity of assessing the performance of a research organization that conducts a broad range of activity from exploratory to requirements-driven work. The Construct covers three principal areas of interest: relevance (does the work respond to customer requirements?); productivity (are we progressing toward goals at an acceptable rate?); and quality (are we doing world-class work?). It is built on three pillars: peer review, metrics and customer evaluation (see Exhibit 22.10). ARL discourages the use of formulas to calculate performance scores, preferring to use a more qualitative approach. The approach is sufficiently flexible to provide information to senior management, which can be integrated and used in a variety of ways; these include forming part of the performance standards of ARL's senior leaders, thus coupling performance to reward.

22.109 Other examples of constructive initiatives that we identified include the following:

• An evaluation framework is developed at the time that a program is launched, including the identification of measures and targets that will be used to evaluate the program at a later date.
• Research organizations are increasingly relying on ongoing and periodic reviews of projects and programs to ensure progress toward providing relevant, timely and significant expertise and research findings to its dependent constituencies.

Conclusion

22.111 Our intention in doing this work was to inform Parliament about attributes of well-managed research organizations and provide guidance to federal research managers. We expect that the attributes presented here will be further refined over time as researchers, research managers, evaluation specialists and auditors discuss and use them. We hope that our work will catalyze further development of these attributes. We invite others to build on our work by developing more specific performance measures and by defining ranges of performance levels for each attribute so that organizations can monitor improvement in their performance.

About the Study

Objectives

• to inform Parliament about attributes of well-managed research organizations that can be used to assess the management of research activities in federal departments and agencies; and
• to provide guidance to federal research managers on ideal outcomes of good management together with examples of practices followed by respected research organizations outside the federal system to achieve the performance ideals described by the attributes.

Scope

The organizations we visited were drawn from the public and private sectors and from important sectors of the economy. They represent the range of government research activities (knowledge creation; wealth creation; policy, regulatory and security functions; and the provision of major research infrastructure).

Study Team

For information, please contact Peter Simeoni.