This Web page has been archived on the Web.

1995 October Report of the Auditor General of Canada

Assistant Auditor General: David H. Roth
Responsible Auditor: Eric Anttila

Main Points

12.1 Organizations today view the successful introduction of technology as being critical to their success. They are looking to the significant potential offered by information technology to reduce costs and improve service. The Canadian federal government is no exception.

12.2 The implementation of systems development projects, in both the private and public sector, is characterized by risk and uncertainty. Accordingly, it is imperative that the risks be identified, evaluated and effectively managed.

12.3 We found that only one of the four systems under development that we reviewed - Transport Canada's Integrated Departmental Financial and Materiel Management System (IDFS) - is currently being managed in a way that deals effectively with the risks associated with the project. Of the remaining three, the Public Service Compensation System (PSCS) has been terminated by Public Works and Government Services Canada; the Common Departmental Financial System (CDFS), being developed by Public Works and Government Services Canada, has only a small number of committed users, and the Income Security Program Redesign (ISPR) requires continued corrective action by management of Human Resources Development Canada to reduce project risks.

12.4 In its attempt to develop PSCS, Public Works and Government Services Canada has spent approximately $61 million, of the estimated total project cost of $119.5 million, for which the planned benefits will not be realized.

12.5 Private sector research indicates, and our own risk assessment supports the view, that the likelihood of large multi-year systems development initiatives being completed on time, within budget and with the desired functionality (what the system should do for its users) in the public and private sectors is extremely low.

12.6 To improve the likelihood of successfully introducing information technology, the government will need to focus on implementing its long-term information technology strategies through smaller, more manageable components, each of which provides an improved capability (efficiency and/or effectiveness) to the organization.

12.7 Among the factors critical to managing the risks that affect the successful introduction of information technology are:

D effective project sponsorship by an individual who can ensure that the organization understands and achieves the planned benefits from the systems development project;

D clearly defined functional and system requirements;

D effective user involvement and commitment to the success of the project; and

D the expertise and experience of resources dedicated to the project.

12.8 Departments and the Treasury Board Secretariat (TBS) have acknowledged the difficulties and risks in delivering large information technology projects, and the Secretariat has undertaken a major review of the factors contributing to the difficulties. TBS is currently working with departments on a variety of changes to project initiation, management and monitoring processes.

Background

12.9 The Canadian federal government, like many organizations, is making significant investments in the development of information technology, with the objective of reducing the cost of government and delivering programs more efficiently and effectively. Through its Blueprint for Renewing Government Services Using Information Technology, the government has made the successful introduction of information technology a key priority. Currently 25 large systems development projects, having a combined budget of more than $2.1 billion, have been identified to Treasury Board as having been initiated to realize the benefits available from the successful introduction of technology.

12.10 In recent years, the amount spent on developing large-scale computer systems, in both the private and public sectors, has risen dramatically. Historically, many of these systems have failed to meet the needs of prospective end-users and/or have not incorporated sufficient financial and managerial controls.

12.11 Within the federal government, the Public Accounts Committee, this Office and a number of deputy ministers, through the Treasury Board Secretariat Committee on Information Management Systems, have expressed concern over the difficulties experienced by the government in implementing systems development projects.

12.12 A recent United States research study completed by The Standish Group, which included both private and public sector organizations, reported that "a staggering 31.1 percent of projects will be cancelled before they ever get completed" and that "52.7 percent of projects will cost 189 percent of their original estimates." This study noted that "on the success side, the average is only 16.2 percent for software projects that are completed on-time and on-budget." In larger companies, "only 9 percent of their projects come in on-time and on-budget."

12.13 Nevertheless, most organizations have become either largely or totally dependent on their computer systems. With current technologies, systems are becoming significantly more integrated and complex. For the remainder of this decade, emerging technologies will place even greater demands on all organizations - including the federal government - to introduce technology in a careful, well-managed way.

12.14 Within the federal government, a number of organizations have specific responsibility for portions of the systems development life cycle. Exhibit 12.1 lists these organizations and their respective responsibilities. Exhibit 12.2 provides an overview of key steps in the systems development process.

Audit Scope

12.15 The audit examined four major systems under development:

  • Public Service Compensation System, being developed by Public Works and Government Services Canada;
  • Income Security Program Redesign, being developed by Human Resources Development Canada;
  • Common Departmental Financial System, being developed by Public Works and Government Services Canada; and
  • Integrated Departmental Financial and Materiel Management System, being developed by Transport Canada.
12.16 Brief descriptions of the four systems are provided in Exhibit 12.3 .

12.17 We chose these specific systems because they:

  • are complex, long-term initiatives that will take several years to implement and will affect a significant number of Canadians;
  • either affect the infrastructure of government, or have an important impact on the operational capabilities of a department; and
  • represent a significant investment on the part of the government, particularly during a period of restraint.
12.18 We recognize that these four systems may not be representative of all systems development initiatives in government. Their total value of approximately $500 million, representing nearly 24 percent of the government's current spending on the largest information technology projects, makes them significant in their own right. However, the conclusions of this audit should be of interest to all departments involved in systems development projects.

12.19 The audit also considered the impact of environmental factors, such as changing priorities and objectives of the government, new technologies, and changes in organizational structure that affect the implementation of such systems.

12.20 In addition, we examined the role of central agencies in the approval and funding of these projects, because the independent review and analysis of data provided by departments, in support of a request for approval and funding of a project, is a key control within the government's expenditure management system.

12.21 The scope of this audit did not include a review of the procurement of information technology, as this is the subject of a separate audit to be reported in 1996. In addition, we did not audit the role of the contractors in these projects, since it is the responsibility of departmental management to monitor and manage the contractor's performance.

Audit Approach

12.22 As we reported previously, our Office is interested in the wise investment in technology to reap the benefits that it offers. The current audit is part of a series of chapters that will examine the implementation of large-scale systems development projects. The Office's approach was to be proactive and constructive, which involves reviewing systems while they are being developed, rather than waiting until they have been implemented. This approach allows us to alert both management and Parliament to the need to take corrective action early in the systems development process.

12.23 As the basis for our audit, we conducted a risk assessment, a process through which professional judgment is applied in considering certain basic questions about virtually any project, including systems under development. These questions include:

  • What can go wrong?
  • What is the probability of it going wrong?
  • What are the likely consequences if it does go wrong?
  • Who will be affected if it goes wrong? and
  • What procedures has management implemented to control risks and to mitigate potential consequences?
12.24 Our audit focussed on these questions by assessing the risks associated with implementing systems development projects through a framework that considered the processes and practices that management had put in place to identify:

  • the potential risks that could hinder or prevent the successful implementation of a system, and manage those risks;
  • the complexity of the technical and business solutions being implemented; and
  • the linkage between the projects' objectives and departmental business lines and priorities.
12.25 To provide a basis for comparing the results of this audit with the experience of the private sector, we also reviewed a number of recent research studies that examined the reasons why systems development projects fail.

Observations

Systems Status and Risk Management

12.26 The implementation of systems under development projects, in both the private and public sectors, is characterized by risk and uncertainty. Accordingly, it is imperative that the risks be identified, evaluated and effectively managed.

12.27 In our current audit, we found that only one of the four systems reviewed, the Integrated Departmental Financial and Materiel Management System, is presently being managed in a way that deals effectively with the risks associated with the project. Of the remaining three:

  • the Public Service Compensation System (PSCS) has since been terminated by Public Works and Government Services Canada(PWGSC);
  • the Common Departmental Financial System(CDFS), after six years of development, has obtained the support of only a small number of users. At the time of the audit, only one department, PWGSC, and three small agencies had committed to switch from the financial accounting system, which CDFS was designed and built to replace; and
  • the Income Security Program Redesign (ISPR) project needs continued corrective action by departmental management to reduce project risks.
12.28 In particular, we noted the following factors with respect to ISPR, which require immediate attention to reduce project risks from the present high level:

  • As of June 1995, the project was at month 15 of a 35-month contract. Despite the fact that 20 months remained until completion of the contract, the project had used up the entire contingency provided for schedule slippage.
  • To help recover past schedule slippage, a decision has been made to perform a number of tasks in parallel, which were originally planned to be completed in a sequential manner. In addition to helping mitigate current information technology risks, this also has the effect of shifting the dates for completion of some project tasks until much later in the life of the contract, thus increasing project risks in the latter stages of the project.
  • At present, the project office receives data from the contractor with respect to percentage of a task completed. However, it does not receive information on actual versus planned levels of effort expended by the contractor to complete these tasks, or estimates of the levels of effort that will be required to complete them. In our view, such information is important for effective project management as it allows more accurate prediction of levels of effort to complete the project.
12.29 A major commitment and sustained effort on the part of project management will be required to reduce these risks to a manageable level.

12.30 Our risk assessments for all projects represent a snapshot in time of a project's status and the risks that project management faced at the time of the assessment.

12.31 The findings that have emerged from this audit are not new. Although we have made similar observations in past audits, the issues that have led to these observations persist and continue to impair the ability of the government to successfully introduce information technology. Although management has generally identified those risk factors that continue to cause difficulties in implementing systems, it has not been effective in reducing the risks associated with large multi-year projects.

12.32 In comparing the results of this audit and previous audits with the results of private sector research studies, the factors that influence the risks associated with systems development initiatives in the Canadian federal government are similar to those reported in the private sector.

12.33 Current approaches to developing and managing systems development projects must be changed. If current practices are continued, and private sector results are applied, a significant number of the 25 large-scale systems, with total planned expenditures of $2.1 billion, will experience serious difficulties, resulting in some cancellations and/or significant cost overruns.

12.34 We recognize that, while there are similarities in the factors that affect the risk of systems under development, there are also differences in the ways in which the private sector and the federal government deal with projects experiencing difficulties. Traditionally, there has been a tendency within the federal government to reorganize and restructure projects and apply additional resources rather than cancel projects.

12.35 Whether projects are cancelled or restructured, the impact of project cancellations and/or cost overruns is significant. In the case of the PSCS project, Public Works and Government Services Canada (PWGSC) had spent approximately $61 million, of the estimated total project cost of $119.5 million, by the time it terminated the project. These figures reflect only the Department's expenditures for purchased services and do not include the cost of departmental resources dedicated to the project, or such elements as the cost of money.

12.36 There is little doubt that PWGSC did gain improvements in its departmental infrastructure as a result of the system development work that was done; however, a significant amount has been spent for which the planned savings will not be realized. The Secretariat has noted that the government will still receive a return on its investment in the PSCS system, as the funding levels of PWGSC have already been permanently reduced by the savings that were expected as a result of PSCS implementation.

12.37 The focus of our report is to identify those variables that influence the risks associated with systems under development. Our observations and findings are presented on subsequent pages under the following headings:

  • Project Management and Monitoring;
  • The Nature of Large Information Technology Projects; and
  • Environmental Factors.

Project Management and Monitoring

12.38 The pressures of deficits and debt have prompted the government to embark on a number of initiatives aimed at improving service to the public and reducing costs for program delivery and overhead. One key initiative has been to introduce new information technology systems.

12.39 We noted a number of management-related weaknesses associated with the introduction of these systems. These include:

  • inadequate analysis of underlying business issues;
  • inconsistent support from management and project sponsorship;
  • a lack of experienced resources on project teams;
  • inconsistent user involvement and acceptance; and
  • a lack of effective ongoing monitoring of systems under development.
Inadequate analysis of underlying business issues
12.40 If information technology is to be successfully implemented in a department or the government as a whole and contribute to modernizing its business processes, the organization's business needs must dictate the requirements for the type of technology to be used. To ensure that this occurs, it is critical that departments with primary responsibility for the development of supporting business cases carefully analyze the processes or procedures that are to be modernized. In the case of central and common or shared systems, this responsibility is shared by the Treasury Board Secretariat. Such analysis would include:

  • identifying alternatives to the existing processes that would meet the business needs of not only the department but the government as a whole;
  • defining the respective roles, responsibilities and accountabilities of both departments and central agencies in relation to the business processes;
  • determining exactly why the current processes are not working as well as they might; and
  • setting clear performance expectations for the improved business processes.
12.41 Our 1987 audit of the then Department of Supply and Services indicated that the federal government's cost of compensation administration was, at that time, approximately twice that of comparable jurisdictions. In response to this observation, the Department initiated the PSCS project in an attempt to reduce the cost of compensation administration within the federal government by automating the existing processes.

12.42 The PSCS was designed to upgrade and modernize existing public service compensation systems within Public Works and Government Services Canada, with a view to improving service levels and reducing the cost of compensation administration within the Department.

12.43 A number of organizations play a significant role in the business processes for compensation administration. These include the Treasury Board Secretariat, which, on behalf of the Treasury Board, is responsible for developing and monitoring the human resource policy framework, under which the rules are set for payment of public servants, and Public Works and Government Services Canada (PWGSC), which administers compensation within the federal government.

12.44 As part of our audit we attempted to determine whether the approach to modernizing the compensation administration process within the federal government included sufficient analyses of the underlying business issues that influence the cost of compensation administration. We expected to find a thorough analysis of these issues by both PWGSC and the Treasury Board Secretariat.

12.45 It was recognized from the outset that the compensation environment was complex. The Secretariat also realized that the necessary legislative and collective bargaining changes needed to simplify the complexity would have taken years to complete. The Secretariat has stated that the need for a fully integrated compensation system such as that proposed by PWGSC was too great, and the significant savings to which PWGSC committed too alluring, for the project not to proceed. It appears, however, that the complexity of the compensation environment was not fully appreciated at the outset by either organization.

12.46 Public Works and Government Services Canada undertook the development of PSCS without the benefit of redesigning all components of the existing business procedures or processes. In this instance, the complexities of the compensation policy framework contributed significantly to the difficulties encountered in trying to develop this system.

12.47 The government anticipated that the cost of administering compensation would have been reduced had the PSCS initiative been successful. However, the system would not have achieved the full cost reduction potential because of the failure to analyze and resolve underlying business issues, which were a root cause of the high cost of compensation administration within the federal government.

12.48 If the government is to deal effectively with its administrative costs, it must take the current opportunities to re-engineer its business processes and consider the administrative cost of its policy framework.

12.49 We did not find any indication that the Secretariat had specified expected overall levels of performance for the government-wide cost of compensation administration.

12.50 We note that the Secretariat has recently initiated a pay and benefits re-engineering project to address some of these issues.

Inconsistent support from management and project sponsorship
12.51 Introducing information technology projects successfully requires the commitment and support of a project sponsor who is a member of senior management. We recognize that the term "project sponsor" is not currently defined or used within the government's project management and information technology policy framework. Instead, the term "project leader" is defined as the person who is responsible for the management and implementation of the project in accordance with its stated objectives. While this responsibility is of critical importance, it is not, in our view, sufficient. The concept of project sponsorship is more encompassing. An effective project sponsor is responsible for ensuring that the organization understands the value and importance of the project in contributing to the achievement of the organization's objectives and, ultimately, for realizing the predicted benefits of the project.

12.52 In our audit we found that the effectiveness of project sponsorship varied from one project to another.

12.53 The Integrated Departmental Financial and Materiel Management System (IDFS) project illustrates effective project sponsorship and commitment from senior management. At Transport Canada, the project sponsor played a strong and active role. The sponsor's involvement contributed to overcoming significant difficulties encountered shortly after the project's inception.

12.54 Soon after Transport Canada signed the contract, it was realized that the project was in severe difficulty. The contract did not accurately reflect the requirements of the project, the project had slipped in its planned schedule, and costs were expected to increase. The sponsor of the project responded to these problems by:

  • establishing limits on project costs, setting firm schedules, and defining the critical paths and the performance criteria for the system;
  • restructuring the project management team and enlisting experienced managers who had strong contract administration and project management skills; and
  • establishing an effective accountability framework that clearly defined responsibilities and accountabilities for managing the project.
12.55 The sponsor's success was largely due to the effective teamwork that was fostered between his managers and the contractor. As a result, they assumed a professional, results-oriented approach to project management.

12.56 The Common Departmental Financial System (CDFS) project illustrates the consequences that can follow when roles and responsibilities within a project are not clearly established.

12.57 The CDFS development was initiated in 1989 to replace an existing government financial system (FINCON). CDFS was intended to meet the functional needs of a group of users, most of which were small-to medium-size departments, while satisfying the requirements of the central agencies. At the same time, the Treasury Board Secretariat initiated its Financial Information Strategy, a project that was intended to allow the Government of Canada to renew its financial system infrastructure and to develop a new set of guiding principles for financial systems and management.

12.58 In an attempt to reduce the number of financial systems within government, it was agreed that CDFS would be a preferred optional system of the Secretariat's Financial Information Strategy.

12.59 Under terms of an agreement signed by Public Works and Government Services Canada(PWGSC) and the then Office of the Comptroller General (now part of the Treasury Board Secretariat), as joint developers of CDFS, some of the respective roles and responsibilities for each organization were established. The agreement failed, however, to clearly establish a project sponsor. The Comptroller General was responsible, in concert with the system's users, for defining functional requirements (what the system must do to satisfy users' needs), setting priorities, approving plans and monitoring the delivery and performance of the system. PWGSC, as product manager, was responsible for developing, implementing and maintaining CDFS.

12.60 By not clearly establishing the respective roles and responsibilities in this instance, each of the two lead organizations or joint developers looked to the other as being responsible for the successful introduction and acceptance of the project.

12.61 Many viewed the Office of the Comptroller General (OCG) as being responsible for the CDFS project because it was initiated under the auspices of the OCG's Financial Information Strategy. This view was reinforced by actions of the OCG, which, in the initial stages of the project, initiated and funded an independent review of the project to assess project status and risks. As well, the OCG established and co-chaired a CDFS Management Board, which was intended to provide a forum through which advice and guidance could be provided to PWGSC and through which user inputs could be provided.

12.62 By the end of 1992, the CDFS project had slipped significantly behind schedule, and the functional requirements had not yet been agreed upon. Furthermore, the government's Financial Information Strategy, which CDFS supported, had still not been finalized. (We note that at the time of this audit, the Secretariat had renewed its efforts to develop the Financial Information Strategy.)

12.63 After it became apparent that difficulties and delays were being encountered in completing the project, the Office of the Comptroller General, as one of the joint developers, quietly reduced its support for CDFS at the end of 1992, and did not actively encourage departments to consider the system. However, it did so without advising either PWGSC or the federal government financial community of its decision.

12.64 In this instance, decisive action was required to either support the continued development of CDFS or to cancel it. As a consequence of the failure to establish clear roles and responsibilities for the successful implementation of the system, approximately $10 million has been spent since late 1992 on developing a system for which key elements of sponsorship are missing and for which user confidence has eroded.

A lack of experience on project teams
12.65 An experienced project team greatly increases the likelihood that an information technology project will be successfully implemented. Members of the project management team need to have experience, knowledge and expertise commensurate with the size and complexity of the project. They also need strong project management and leadership skills, a sound knowledge of the organization's business, and experience with the technologies being implemented.

12.66 The problem of lack of experienced project management is not one that is unique to government. Research studies conducted within the private sector on causes of systems development failures have cited the lack of experienced project management as a significant factor that contributes to project risks.

12.67 In all four of the projects reviewed as part of this audit, the project teams either did not, at the outset, have experience with projects of similar size and complexity, or lacked experience with the technologies associated with the project. Although the teams developed experience as the projects progressed, their lack of experience at the start contributed to increased project risks.

12.68 In two of the four projects, the Integrated Departmental Financial and Materiel Management System(IDFS) and the Common Departmental Financial System(CDFS), key personnel were replaced at key points during the project. By the time this occurred, both projects had slipped significantly behind schedule. However, in the case of IDFS, those changes helped to turn a project that was in difficulty into one that is currently being effectively managed.

12.69 While changes in personnel within the CDFS project are helping to deliver a final product, they have not offset the delays that had already occurred. The delays, together with a rapidly changing business and technological environment, and a project for which roles, responsibilities and accountabilities were not clearly established, have meant that the developers cannot build the system quickly enough to catch up with the evolving expectations of those who may eventually use it.

Inconsistent user involvement and acceptance
12.70 The people who will use a system must buy into or accept the system as a prerequisite to its successful implementation and use. Acceptance can be achieved through ensuring that the system's users actively participate in the development process. For example, they should be involved in activities such as identification of needs and the review, approval and acceptance of completed work at key stages. Involving users in this way will translate to a commitment on their part to the successful implementation of the system. As well, particularly in multi-year projects, user commitment must be regularly reinforced to ensure continuing support and to offset the impact of staff turnover in user groups.

12.71 In our audit we noted that the involvement of users varied from project to project.

12.72 The Income Security Program Redesign(ISPR) project provides an example of effective user involvement. Management of the ISPR project has taken seriously the issue of involving users, getting their buy-in and involving them in decision making. Realizing that the project would not be successful without careful management of organizational change, extensive consultation with end users was planned. For example, to involve users from the outset, management took a number of steps such as:

  • establishing ongoing two-way communication among headquarters, regional implementation managers and end users to deal with concerns as they were raised;
  • developing a mechanism for training and assisting staff in managing the effects of organizational change; and
  • taking a demonstration installation to each regional office early in the project to allow all staff to see the new system and to ask questions of the contractor and members of the project team who accompanied the prototype. (This exercise allowed staff to see the "vision"of the kind of system ISPR would eventually be.)
12.73 The result of these efforts has been widespread support for the project among users at all levels.

12.74 In the Common Departmental Financial System (CDFS) project, the nature and extent of user involvement has fluctuated. At its inception, CDFS had two objectives. One was to replace an existing financial management system. The other was to meet all the requirements of the Financial Information Strategy being developed by the then Office of the Comptroller General. Today, because of the lack of user commitment and support, CDFS has only a small number of users. At the time of the audit, Public Works and Government Services Canada was the only department and there were three small agencies (of the approximately 20 users of the current system that CDFS was designed to replace) electing to use CDFS.

12.75 The primary forum for co-ordinating users' input and providing guidance to PWGSC for the development of the system has been the CDFS Management Board. However, Board members whom we interviewed as part of this audit do not believe that they have had any meaningful involvement in or commitment to the project. Generally, potential users remain unconvinced that CDFS will provide them with the same level of performance and functionality that they have with existing systems. Although the department that conducted the pilot test of CDFS had committed significant human and financial resources to assessing the system, it decided not to implement the system. Deficiencies cited by potential users as reasons why they chose not to use the system include concerns over functionality and performance such as:

  • slow response time;
  • the lack of horizontal integration with other systems, such as materiel management; and
  • the inability of the system to prevent issuing duplicate cheques without resorting to manual processes outside the system.
12.76 Although the identification and correction of performance problems is a normal part of systems development projects, and although project management believes the performance concerns have been addressed, some potential users have adopted a wait-and-see attitude, while others have indicated that they will want to run their own pilot test before deciding whether or not to use CDFS.

12.77 As well, some potential users have indicated a desire to assess the outcome of the current initiative of the Treasury Board Secretariat to assess a number of commercial software packages.

12.78 Effective user involvement and commitment to the systems development project is critical to the project's success. The CDFS project illustrates two common problems. First, the rate of change of technology in today's environment invariably makes potential users more demanding. Accordingly, a project must have the flexibility to respond, where appropriate, to evolving expectations. Second, successful systems development projects and, in particular, shared systems, require continued user commitment or buy-in.

A lack of effective ongoing monitoring of systems under development
12.79 Successful implementation of large multi-year projects requires effective ongoing monitoring. Primary responsibility for ensuring delivery of projects and for adequate monitoring of their progress rests with the department managing the project. Traditionally, monitoring these projects has consisted of addressing such questions as:

  • Will the system be delivered on schedule and within budget?
  • Will the system deliver the expected benefits?
  • Will users accept it?
12.80 While the answers to these questions are important, they do not, in themselves, cover all issues. Monitoring must also deal with issues such as the currency and continued relevance of the business case for developing the system. As noted previously, monitoring this area is essential because the operating environment is changing rapidly, and there is no certainty that the original assumptions on which systems were built will remain constant.

12.81 Accordingly, it is important that management has effective processes for managing systems under development. Such processes include:

  • measuring and communicating project status with respect to cost, schedule and performance of the project;
  • conducting risk assessments;
  • tracking and monitoring all project costs; and
  • monitoring changes in the business environment that may alter basic assumptions on which original business decisions were made.
12.82 The information provided by the consistent and effective use of such processes provides project managers with:

  • a basis for communicating a project's status;
  • a basis for focussing attention on areas that need corrective action; and
  • a method for detecting trends (whether adherence to planned costs and schedules is improving or getting worse over time) in the project status of multi-year projects.
12.83 We found that, in general, project management has processes that provide data on project status and identify those areas where corrective action is needed. We also noted that these processes were not always fully implemented and effectively operating until some months after projects began. In our view, it is important that such tools be part of the management infrastructure from the outset of the project. In all four of the projects reviewed, information on project risk was available. In some instances, specific internal audits of the systems development initiatives had been conducted, and in others, independent studies had been commissioned by management. However, despite the availability of information on project status, management has generally not been effective in reducing the risks associated with these large multi-year projects.

12.84 We are also concerned over the time it takes to initiate corrective actions. The Income Security Program Redesign project illustrates this point. While project monitoring identified delays in the information technology component of the project in July 1994, effective corrective action, in terms of revised plans and schedules, was not completed until April 1995.

12.85 As discussed below, we noted that improved procedures are required for monitoring time used by project team members in completing tasks.

12.86 We found that the extent of tracking of project hours varied from one project to another. In most instances, Crown time devoted to a project was not tracked and, in some instances, information on contractor time was not provided to project managers. Under fixed price contracts, such as those in place for the systems we reviewed, this information does not affect the price that the Crown is required to pay. However, the nature of the contract has been used by contractors as a reason not to provide project management with full information on the time taken to complete specific tasks. In our view, such information is an essential component of sound project management.

12.87 Complete information on time used for a project provides management with information on levels of effort expended to achieve the results to date. It is also required to predict whether more or fewer resources will be needed to complete the project on time.

12.88 Such information is critical to assessing the continued relevance of the original business decisions and is a necessary first step to respond to the changing business environment.

12.89 Monitoring by Treasury Board Secretariat. While departments have primary responsibility for adequate monitoring of the progress of their projects, there is also a responsibility on the part of the Secretariat, because of its overall responsibility for the financial management of government, to monitor projects of significance and risk.

12.90 We expected that the following key areas would be monitored:

  • project status relative to planned progress, in terms of time, costs and schedule;
  • achievement of expected benefits;
  • continued relevance of original assumptions on which projects were approved; and
  • risk factors that might affect the project.
12.91 We also expected that the information flowing from these monitoring efforts would be used, when necessary, to recommend and, if necessary, initiate corrective action.

12.92 We noted that the Secretariat receives information on project status from departments, and in general, that this information is reviewed and analyzed. In some instances, the Secretariat has, either on its own or in concert with other departments, initiated corrective action. For instance, in response to concerns about delays in the CDFS project, the Secretariat initiated and funded an independent review of the status and risks associated with the project. In other instances, for example with respect to PSCS, while concerns were identified, actions were either late in coming or ineffective in resolving the concerns noted.

12.93 We also reviewed the nature and extent of the Secretariat's monitoring of the effectiveness of its information technology policy framework in contributing to the successful introduction of systems development initiatives.

12.94 We anticipated that such monitoring would consist of the tracking and analysis of information technology spending within the federal government, and an assessment of the reasons for past successes and failures. In our opinion, such data and analysis are essential for assessing the effectiveness of the information technology policy framework.

12.95 We found that while the Secretariat has data on individual departmental projects, such data are not analyzed, from a government-wide perspective, to assess the effectiveness of the policy framework. The Secretariat has not regularly captured and disseminated lessons learned from past successes or failures as a basis for future improvements.

12.96 We noted that in response to concerns raised by departmental deputy ministers, the Secretariat has begun an examination of all aspects of the management framework for large information technology projects.

The Nature of Large Information Technology Projects

12.97 Our audit noted several factors related to the inherent nature of large multi-year, multi-million dollar projects that influence the risk of successfully implementing information technology projects. These findings are set out in subsequent pages under the following headings:

  • Project size;
  • Technical complexity; and
  • Risks associated with uncertainties about functional requirements.
Project size
12.98 The implementation of any systems development project carries with it a degree of uncertainty and risk, which can be magnified many times over as the size of the project increases. Studies have indicated that the likelihood of a system either falling significantly behind schedule or failing to be implemented increases with the size of the project. For this reason, it is important that the analysis of projects in their planning and approval stages includes measuring their size and complexity.

12.99 One such measurement technique used in the software development industry is called Function Point Analysis (FPA). This technique provides management with a relative estimate of the size, complexity and risks associated with a project.

12.100 In FPA terms, any project containing more than 5,000 "function points" is considered complex and entails significantly more project risk. A function point can be considered to be a task that the computer software must perform. Research on FPA carried out in the United States has shown that systems development projects that have more than 10,000 function points have about a 50 percent chance of being cancelled before they are implemented.

12.101 The use of software metrics does not, by itself, guarantee success in implementing information technology. However, the use of such techniques is an integral component of sound project management, which provides management with a benchmark for measuring the size and complexity of a proposed initiative, as well as an indicator of the levels of risk that a project may face.

12.102 We noted that such measurement tools had not been used during the development of the business cases for any of the systems included in this audit. For two of the four systems, management had completed a function point count only after the projects had begun:

  • The count revealed more than 25,000 function points for CDFS, as it was originally envisaged in 1990. After the project was scaled down in 1992, it contained about 16,000 function points.
  • PSCS, as originally planned, contained about 14,000 function points.
12.103 While no function point analysis was completed for the ISPR and IDFS systems development projects, our review of these systems indicated that the size and complexity of these systems were about the same as PSCS and CDFS. We noted that in the case of ISPR, management did use alternative measurement techniques to assess the size of the project in functional terms.

12.104 By comparison, some private sector organizations that use function point analysis will not, because of the associated high risk, undertake systems development initiatives that exceed 5,000 function points.

Technical complexity
12.105 The risk of being unable to successfully implement a systems development project increases with the technical complexity of the system itself. Technical complexity refers to the design features or architecture of the system, such as the system's links to networks, its data conversion features, and the extent to which it is designed to accommodate multiple users working from remote terminals.

12.106 The complexity of the four systems reviewed as part of this audit varied, as did the reasons for their complexity.

12.107 For example, Transport Canada acquired a commercially available software package for the IDFS project and modified it to meet the Department's needs. In this instance, the modifications involved developing interfaces with existing departmental systems and the government's central accounting systems and customizing some of the system's reporting features.

12.108 While the acquisition and installation of a commercially available software package should have been the least complex of the systems we reviewed, the extensive modification of the software to meet the needs of the Department and to be integrated with central accounting systems significantly increased the project complexity.

12.109 As well as adding complexity, such modifications may make the implementation of new releases of the software more difficult and costly.

12.110 The use of commercially available software offers the potential for cost savings. A significant portion of the savings comes from limiting the extent of changes to the commercial software package. Within the private sector, a guideline often used for changes to commercial software is to limit them to less than 10 percent of the functionality (what the system will do for its users) of the software package.

12.111 In contrast, the remaining three systems had aspects that are, or were, significantly more complex.

12.112 For example, in the case of CDFS, the developer faced the task of building a shared system that would interface with many different departmental systems and architectures, all of which added to the general complexity of the project.

12.113 In such instances, project complexity and cost could be reduced through greater adherence, on a government-wide basis, to common standards and architectures. Such increased standardization across departments would also facilitate achievement of the government's objectives.

Risks associated with uncertainties about functional requirements
12.114 In each of the four systems reviewed, we noted that the functional requirements (what the system must do to satisfy the needs of those who will use it ) had been articulated at the time the project had been approved and initiated. However, as all four projects progressed, it became clear that the requirements, as originally stated, needed to be either restated, refined or more clearly communicated.

12.115 The impact of any uncertainty arising from imprecise requirement statements affects two key areas:

  • project budgeting; and
  • project approval.
12.116 Project budgeting. We noted in our audit that initial estimates of project costs prepared by departments often significantly differed from the ultimate cost of a project. The change in estimates can affect a project in one of two ways. The first possibility is that project costs will rise as the understanding of the project improves. For example, as the refinement of the PSCS design progressed, the original cost estimate of $55.1 million rose to $90.3 million and, eventually, to $119.5 million.

12.117 However, at the same time, the expected savings in departmental operating costs, after successful implementation of the system, rose from an initial estimate of approximately 300 person-years to 781 person-years.

12.118 The second possibility is that in order to keep project costs stable, the project requirements will be reduced to match the funds available. This means the system will perform fewer functions than were originally planned, which may increase the risk that the system will fail to meet users' expectations or needs.

12.119 The IDFS project provides an example of a project where scope was reduced to stay within the original budget. In order to maintain the $60 million ceiling for the project and to avoid an estimated $20 million increase in contract price, management decided to redefine and prioritize the original scope of work to eliminate some items, while maintaining what management considered essential functionality.

12.120 A decision to reduce a system's scope is sometimes reversed after it is discovered that the "downsized" system will not adequately meet users' needs. In these cases, subsequent contracts or enhancements can add significantly to the project cost. This happened in the case of the CDFS project. In 1992, to facilitate earlier delivery of some components of the projects, approximately one third of the system's original requirements were eliminated. While the project scope was reduced, the contract price remained constant. This functionality has since been reintroduced at an approximate additional cost of $10 million.

12.121 We also noted that estimates of project cost do not include all costs associated with implementing the system. Estimates include only the costs of obtaining goods and services from suppliers or contractors, and not the substantial departmental resources expended to develop and implement the system.

12.122 Approval process. We are concerned that failing to clearly define the scope and systems requirements for the projects that we reviewed has had implications for the approval process. From the above-noted observations, it is apparent that the inherent nature of large, multi-year projects does not allow project sponsors to predict with confidence what the ultimate cost of the project will be.

12.123 A similar concern was expressed by Treasury Board ministers at the time of the second Treasury Board submission for PSCS when estimated project cost had risen by $35 million.

12.124 Within the scope of our audit, we wanted to assess the nature and extent of Treasury Board Secretariat analysis of the business case submitted by departments in support of their project initiative. Because of the strategic importance of information technology in reducing costs and improving services, we had expected that the Secretariat's analysis of departmental submissions would include a thorough analysis of full project costs, projected savings and assessment of the risks associated with the project.

12.125 However, in our view, the analysis was not sufficient for the purposes of providing a recommendation or advice to Treasury Board ministers as to whether a project should be approved and funded at the level requested.

12.126 The analysis of the submission for PSCS illustrates this weakness. In this instance, the Secretariat did review and comment on the submission and asked the Department, PWGSC, for clarification on the draft submission. However, we found little evidence of extensive analysis of estimated project costs to assess the reasonableness of the requested funds or analysis of projected savings to assess their reasonableness.

12.127 Finally, the analysis done by departments and by Treasury Board Secretariat should be sufficient to provide as complete information as possible on which to base the system investment decision. This analysis should provide assurance that:

  • project benefits are realistic when project approval and funding are sought;
  • users' expectations are realistic and affordable; and
  • realistic estimates of project costs and competence required are presented, along with the impact of these on the scope of work.

Environmental Factors

12.128 The business environment within which we operate is continuously changing, with the only seeming certainty being that the rate of change will continue to accelerate. In recent years, the number of changes and the pace at which they are occurring, both globally and within government, has sometimes outpaced the ability of systems developers to respond to them.

12.129 Key environmental factors include:

  • a changing business environment; and
  • rapid advances in technology.
The changing business environment
12.130 For each system that we reviewed, the business environment, under which the initial decision to build the system was made, is no longer relevant. The examples below demonstrate the risk associated with developing large systems that take years to develop, in a situation where the business environment is continuously changing.

12.131 Changes to the business environment have affected Transport Canada's Integrated Departmental Financial and Materiel Management System (IDFS). When the Department initiated the system in 1990, it employed 20,000 people and included airport, aviation and coast guard operations. The size and operations of Transport Canada in 1995 and beyond differ greatly from what they were when the project started. Today, Coast Guard operations have been transferred to another department, air navigation services are proposed to be transferred to a not-for-profit organization outside government, airports are being privatized and projections reflect a Department of 3,000 to 5,000 employees. All these changes equate to a very different business environment than when work on IDFS began.

12.132 As a consequence of the many factors that have changed the operating environment of Transport Canada, the Department had significantly more licences for its commercial software than would be needed for its new operating environment. To mitigate the impact of the changing environment, the Department, in conjunction with PWGSC, has negotiated a transfer of surplus licences to four other departments, one of which is implementing IDFS.

12.133 In the case of the Income Security Program Redesign (ISPR), some of the basic assumptions that governed the decision to build the system have changed. Initially, it was assumed that the system would administer the three elements of the social security framework: Family Allowance, Old Age Security and the Canada Pension Plan. Since the project was initiated, the Family Allowance program has been replaced by the Child Tax Benefit program and is administered by Revenue Canada through the taxation system. The federal Budget of February 1995 indicated that the government would conduct a review of the public pension system and was making changes to the Old Age Security program. The results of this review may impact on the original assumptions under which ISPR began.

12.134 To mitigate the impact of a rapidly changing business environment, it is important that large systems development initiatives be broken down into smaller, more manageable components, with each component delivering an improved capability (efficiency or effectiveness). Under such an approach, if environmental changes affect systems development requirements, the government has still achieved a benefit and payback from the implementation of earlier components.

Advances in technology
12.135 When the Common Departmental Financial System (CDFS) was initiated in 1988, users' expectations focussed on vertical integration of the system with the central accounting systems. Since initiation of the system, users' expectations have dramatically evolved in response to a changing environment and the availability of new and emerging technologies. Today, users' expectations of what constitutes an acceptable financial accounting system require not only vertical integration with the central accounting systems but also horizontal integration with other departmental business applications, such as materiel management and human resource information systems. Although the expectations of users have changed, the system's developers have not been able to respond quickly enough. Therefore, the system does not provide the benefits of horizontal integration, which users are demanding.

Conclusions and Recommendations

12.136 The government has placed significant emphasis on its ability to successfully implement information technology as a means of reducing its costs and improving service to the public.

12.137 In our audit we have identified a number of risks that influence the successful implementation of systems development projects:

  • weaknesses in project management;
  • the lack of effective ongoing monitoring and/or appropriate corrective action;
  • the nature of large information technology projects; and
  • the impact of environmental factors.
12.138 We have observed the significant impact that occurs when these risks are not effectively managed. In its attempts to develop PSCS, Public Works and Government Services Canada has spent approximately $61 million, of the estimated total project cost of $119.5 million, for which the planned benefits will not be realized.

12.139 Many factors affect the successful development and implementation of systems. Among the most critical to the successful management of project risks are:

  • effective project sponsorship;
  • clearly stated requirements;
  • effective user involvement in and commitment to the success of the project; and
  • the expertise and experience of resources dedicated to the project.
12.140 An integral component of any new approach must focus on the implementation of long-term information technology strategies through smaller, more manageable components, each of which provides an improved capability (efficiency and/or effectiveness) to the organization.

12.141 Such an approach provides certain key benefits:

  • It is easier to more clearly define requirements(what the system should do for its users).
  • The likelihood of successfully completing the project is greater.
  • It is possible to develop more complete, accurate estimates of costs and schedules.
  • It is easier to obtain project resources with appropriate levels of experience.
12.142 To increase the likelihood of successfully implementing systems development projects, departments and Treasury Board Secretariat should give immediate attention to improving current practices for approving, funding and managing such projects.

12.143 We encourage and support the Treasury Board Secretariat's efforts to renew the current management framework for systems under development.

12.144 Future audits will monitor the progress of this initiative and will continue to examine issues that affect the successful implementation of systems development projects.

Public Works and Government Services Canada response: The Department shares the concerns of the Auditor General with respect to the risks of systems under development.

As part of its recent evaluation of Common Purpose Procurement (CPP), PWGSC conducted its own investigation into the difficulties experienced with large-scale information technology systems and how these might be overcome. The results of this work indicated that improvements could only, meaningfully, be realized through a co-operative effort between ourselves, Treasury Board Secretariat and, of course, the sponsoring departments for the projects involved.

This perspective was endorsed when our research was presented to the Treasury Board Subcommittee on Information Technology (TIMS) on April 28th of this year. At that time, a co-ordinated, interdepartmental Action Plan to address areas of major concern and from which improvement efforts are expected to result in the greatest benefits was agreed to.

Public Works and Government Services will continue to strengthen the skills our staff bring to the contract and risk management associated with information technology and systems integration (IT/SI) projects; examine the potential for providing project management expertise for such initiatives; consult with stakeholders regarding how to assess the achievement of value for money in IT/SI projects; and collaborate with Treasury Board Secretariat in examining issues concerning partnering arrangements.

Treasury Board Secretariat has undertaken to ensure the adequacy of policies, standards and tools for the management and delivery of IT/SI projects, including risk, project and contract management; to determine acceptable partnering relationships for IT/SI projects; and to collaborate with PWGSC and the broader community to resolve project management issues and to further develop the successor to CPP.

Sponsoring departments will be obliged to meet the requirements set out by Treasury Board, including the assurance that sound business cases and superior project and risk management skills are brought to major systems projects.

More specific to the systems reported in the OAG audit, the measures taken by the Department to reduce risks include:

1. Responsibility for design and construction of the system was contracted out to major systems integration firms (PSCS and CDFS).
2. Risks of unclear user requirements or overly optimistic construction estimates were mitigated by the use of competitive tenders, independent reviews of project size (PSCS and CDFS), as well as funding of the design phase to two systems integration firms (PSCS).
3. Detail reporting was required from the contractor on time and performance (PSCS).
The fact that these were not successful, as expected, in reducing risks, highlights the need for the Crown to understand the industry track record as well as to manage the risk that contractors will fail to perform well on complex fixed price software development projects. With hindsight, the Crown placed excessive confidence on the industry's ability to deliver. Consequently, we agree with the Auditor General that organizing smaller projects would reduce risks.

Recent work on the CDFS product has in our view addressed any remaining concerns with the functionality of the system. Nevertheless, in light of the environment described by the Auditor General, the Department has decided to suspend active marketing of the system to other government departments and to focus its efforts on implementing CDFS within PWGSC and on supporting its committed clients. While this client base does provide adequate returns to the Crown for its investment in the system, more consistent support from the Treasury Board Secretariat would undoubtedly have helped to increase its return on the investment.

Finally, while the Department is convinced that the notion of "project sponsorship" is critical to the success of common shared financial and administration system development, current circumstances and the accountabilities of the central agencies, departments and the common service department, preclude the effective application of this concept.

In the future, the Department will continue to adhere to professional standards of project management, take a balanced approach to achieving cost reductions through a combination of organizational, procedural and technological change, and review the opportunities for smaller, and shorter duration projects.

Treasury Board Secretariat response: The Treasury Board Secretariat generally concurs with the findings of the Auditor General. These findings are consistent with a separate analysis that TBS has conducted. As both studies point out, technology is complex and ever-changing and its acquisition carries inherent risks. Nevertheless, information technology is a crucial part of the government's thrust to help improve quality service to the public while reducing the cost, and information acquisition projects must continue. While the risks cannot be eliminated, whether in the public or the private sector, we agree they can and must be reduced.

The Treasury Board Secretariat has begun the introduction of changes to the management framework for large information technology projects. These changes will potentially affect all aspects of the project environment, including project size, approvals, risk assessment, project management experience and training, procurement vehicles, change management, reporting and monitoring. We are confident that the improvements will significantly improve the success rate of government informatics projects. Treasury Board Secretariat will be monitoring high risk projects closely to ensure effective implementation of the new approach.

We thank the Auditor General's Office for its support and look forward to its continued involvement in this work.

Human Resources Development Canada response: The Department generally concurs with the findings of the Auditor General and shares the concerns expressed with regard to the risks inherent to systems development projects. The Department will continue to manage the risks associated with the ISP Redesign project, placing additional emphasis on those areas identified in the report. We would also endorse the observation that "large multi-year system development" projects are unlikely to be successful, unless the project is broken down into smaller, more manageable components with significant user involvement at all times.

Audit Team

Greg Boyd
Guy Dumas
Ira Greenblatt
Christine Kelly
Laurel Lowry
William Moeller
Gerry Montigny
Richard Quesnel
Hugh Rodgers
Marvin Schwartz
Bruce Sloan
Barbara Taylor
Peter Taylor

For further information, please contact Eric Anttila, the responsible auditor.