1996 November Report of the Auditor General of Canada
Chapter 24—Systems under Development: Getting Results
Responsible Auditors: Eric Anttila and Hugh McRoberts
Part I - Key Issues for Success24.1 To realize the benefits that large information systems promise, risks must be taken but managed effectively. Of the four projects reviewed this year, only one has an effective risk mitigation process in place, although project delays are still expected.
24.2 We concentrated our audit on service delivery systems rather than the mostly administrative systems examined last year and found many of the same problems. We are concerned that investments of some $3 billion in these projects will not achieve the results targeted by government.
24.3 We focussed our report on six issues we considered key to project success: taking charge of the project, defining requirements, improving software development processes, setting priorities, measuring the status of a project and implementing new government guidelines. Issues of procurement for information technology will be reviewed separately in a later report.
24.4 We estimate cost overruns of $250 million for expenditures of $1 billion in two of the projects. There is an urgent need for attention and involvement by senior management in the development of large information technology projects.
24.5 Some scheduled delivery dates have slipped by as much as two years. A major factor is the inability of departments to specify the functional requirements of the system. Given the rapid pace of change, risks are further increased when the government enters into long-term fixed-price contracts with requirements that are essentially ambiguous, poorly defined, or changing frequently.
24.6 Where software development is involved, processes need to be well defined and well managed. Tools exist to aid in identifying and managing risk, sizing the project and assessing the organization's capabilities to deliver the project. They need to be used.
24.7 To overcome schedule slippages, cost overruns and the relaxation of original requirements, it is important for senior management, at the start of the project, to establish clear, realistic and precise priorities for time, cost and requirements, as well as the results to be achieved. These priorities have to be reviewed regularly throughout the project.
24.8 In the projects we reviewed, measurements of project progress were not used or were used ineffectively to determine the status of the project, to support effective management decision making and to report on the project.
24.9 In April 1996, the Treasury Board approved an Enhanced Framework for the Management of Information Technology Projects and, as of June 1996, departments are being directed to comply with its requirements. However, action plans are needed from the departments to apply the Framework to the more than $5 billion worth of investments in existing and future projects.
Part II - CAATS: Some Matters of Concern24.10 One of the projects we reviewed, the Canadian Automated Air Traffic Control System was in difficulty by late 1994. It appeared that the project could not be completed either on time or on budget. Transport Canada changed the Project Director and the reporting relationships to ensure the ongoing involvement of senior departmental management in the remainder of the project. Under this guidance, a rebaselined contract was negotiated.
24.11 Changes to the project resulting from the renegotiation of the contract resulted in an increase in costs to the Crown of $217 million. In preparing Part III of its Estimates, the Department followed the convention of reporting for Major Crown Projects the following: the effective project approval ($659 million) and only those additional costs that would be incurred to the end of the main contract ($75 million). However, in doing this, the information provided suggests that the renegotiated contract was achieved with no material increase in the cost to the project. As a consequence, this presentation results in Parliament not being fully informed of the real cost of the decision to rebaseline the CAATS project.
24.12 In renegotiating the contract price, the price was increased by $73 million, from $427 million to $500 million. Our analysis of the documentation prepared to justify the price increase shows that the government has agreed to a contract price of $500 million on a contract where a consideration of the price of the items removed from the original contract and the cost of the items added would only justify a price of $282 million. Additionally, in our opinion, the material prepared for consideration by the government in approving the revised contract should have clearly shown that the revised contract price of $500 million included a component that represented the cost to the Crown of putting the contract in good standing, settling all past differences and disputes among the parties to the contract, and receiving from the contractor a formal release to that effect. While it is not possible to calculate the exact amount, our review of departmental documentation, the price justification and discussion with management leads us to conclude that the settlement cost was approximately $200 million.
24.13 Prior to rebaselining, the role of the CAATS Project Office in bringing closure to requirements definition was largely responsive. There was no formal plan to ensure that this work was concluded in an orderly and timely fashion. At rebaselining, a significant number of important design issues were unresolved. Since rebaselining, integrated project teams were set up to tackle design issues, most of these have been resolved and the Project Office has initiated formal project monitoring and risk management practices.
24.14 Milestones represent an important control in a project of this size. However, in this project, milestones have acquired considerable flexibility. We are concerned both about past practices in assessing the completeness of milestones and about the apparent practice of renegotiating milestones every time a significant milestone is not met.
24.15 Verification is the set of activities that permits the Project Office to conclude that the delivered system meets the specifications in the contract and is safe for use. CAATS is a safety-critical system and this imposes a higher onus on verification activities.
24.16 We conclude that the Project Office has not established the degree of assurance it needs in terms of either system performance or system safety from its verification activities. In addition, the Project Office is not able to determine the level of assurance it will receive from the activities that it and the prime contractor are and will be carrying out. Extensive testing of the system may aid in gaining assurance that the required functions exist in the system, but it is unlikely to provide adequate assurance of the system's overall safety.
Part I - Key Issues for Success
- more resources to monitor large information technology projects;
- an annual report by the Treasury Board Secretariat to the PAC on the status of departments' investments in systems under development;
- steps by the Secretariat to ensure that there is complete ownership and accountability for large information technology (IT) projects; and
- the division of projects, if possible, into manageable components.
24.19 We were concerned that the existing large IT projects would not or could not make adjustments to reflect this revised framework for some time. Accordingly, we continued the work started last year with a view to underscoring the urgent need for improving the way the existing projects are being managed. Further details on the scope and approach of this year's audit are included at the end of the chapter in the section About the Audit .
24.20 In Chapter 12, we referred to the Treasury Board Secretariat's list of 25 large government IT projects with a combined budget of more than $2.1 billion. Our work has shown that this list is incomplete: it does not include all the projects and all the costs.
Status of the Systems Reviewed24.21 We found that the Tactical Command, Control and Communication System (TCCCS) project at the Department of National Defence, the largest of the four systems reviewed this year ( Exhibit 24.1 ), was the only one whose associated risks were being managed adequately. In this project, there were serious risks associated with delays in software development modules that could cause serious financial problems. However, the project team paid close attention to the risks identified. A concerted effort to promote open communications and to effectively delegate decision making throughout the project organization facilitated the identification of the risks affecting the project and offered management the opportunity to act accordingly.
24.22 In the three other projects that we reviewed, project risks were not being managed effectively:
- The schedule for the Canadian Forces Supply System Upgrade (CFSSU) at National Defence started slipping very early in its development. However, management is attempting to bring under control the risks of overrunning the schedule and budget by taking some specific actions regarding key management and technical risks.
- In 1994, the Canadian Automated Air Traffic System (CAATS) at Transport Canada ran into very serious time and cost overruns that led to a major renegotiation of the contract in 1995. In 1996, the project remains at very high risk and is not expected to meet the renegotiated schedule. For events subsequent to August 1996, see paragraph 24.137.
- The Real Property Services (RPS) system development framework contains 11 separately planned projects. In attempting to develop nearly all the projects at the same time, the developer of the system, Public Works and Government Services Canada (PWGSC), encountered difficulties that have introduced a high level of risk. The complete package of projects was not submitted for approval to the Treasury Board and relied solely on internal budget allocations by PWGSC. Two projects have already delayed their implementation date by six months.
Taking charge of the project24.23 Last year, we noted the consequences of a lack of consistent support for a project from senior management and emphasized that effective sponsorship for a project is critical to its success and achievement of intended results.
24.24 The Public Accounts Committee endorsed the notion of effective project sponsorship in its May 1996 Report. The Treasury Board also endorsed this concept in April 1996 in its Enhanced Framework for the Management of Information Technology Projects.
24.25 We have observed that the chance of success is greater when the project leader, sponsor and manager are trained and experienced, and have proven project management capability ( Exhibit 24.2 ).
24.26 Strong leadership, effective communication and appropriate delegation of responsibility in the TCCCS project helped to mitigate the risks caused by the geographical separation of the project teams and by the complexity of the project. Given the nature of the project, senior management's support to the project team came from the project leader. Overall, the project team in place had a high level of technical expertise and displayed strong management skills. The project team encouraged and created an environment of open and honest communication among all the members of the project and with the stakeholders. There was also a judicious delegation of responsibility to the technical staff.
24.27 By contrast, for the first five years of the CAATS project, the project leader and project sponsor were only minimally involved in the decision-making process. As a result, senior managers did not take a central role in decision making. In December 1994, the Deputy Minister assumed the role of project sponsor, the Assistant Deputy Minister assumed the role of project leader and the project adopted a more businesslike approach.
24.28 CAATS is an extremely complex software development project ( Exhibit 24.3 ) that, like TCCCS, entails managing geographically dispersed teams. However, in the case of CAATS, effective communication and appropriate delegation of responsibility to the project staff were missing prior to December 1994. Although improvements are under way, communication is still not effective between the Project Office and the Air Navigation System (ANS), the client group for CAATS.
24.29 The lack of appropriate delegation of responsibility to the CAATS technical staff to resolve technical issues as they arose made decision making cumbersome and slow. In all cases, decisions affecting work being done in British Columbia had to be referred to senior project managers in Ottawa or to Transport Canada headquarters. Beginning in late 1994, a higher priority has been put on expediting decisions.
24.30 In the spring of 1995, the Deputy Minister took an active role and the Assistant Deputy Minister took a hands-on approach by changing the reporting relationship of the project manager to a direct one. This has been instrumental in closing some of the open items and shortening decision cycles. However, problems persist as a more results-oriented management approach is put in place.
24.31 As part of this audit of systems under development, we reviewed the U.S. government's professional development program that offers senior project executives training in the procurement and the management of large system projects. The Treasury Board Secretariat is working on implementing a professional development program aimed at Canadian project managers. Such a program could help create a pool of experienced and expert project managers in Canada. However, we have concerns whether the Canadian program will address the need for qualified project leaders at the assistant deputy minister level to lead large development projects.
24.32 In the U.S., recent policy and practice do not allow a large development project to proceed until a qualified project leader is appointed to lead the project full time. In Canada, there is no such policy. Government departments generally assign part-time project leaders who have no specific training or experience in managing large information technology projects.
Defining requirements: the essential first step24.33 Last year, we stressed that business needs should dictate the requirements for technology. Similar to our 1995 findings, we found in this year's audit that the government started developing systems before it had clearly defined system requirements for reasonable and realistic time horizons.
24.34 The government runs unnecessary risks when it enters into long-term, fixed-price contracts before a clear understanding of what will be built is reached by all parties. The fixed-price development contract ought to be let only when the government and the developer have reached a common understanding and definition of the requirements, the criteria to be used to determine success and the milestones for the work to be done. Combined with the "manageable component" method described last year, this approach can help avoid the high risks currently experienced. Similar to long- and short-term financial planning, projects need to be planned as a whole; but project phases and contracts need to be planned in more detail for 6 to 18-month planning horizons to allow a higher level of control in achieving results.
24.35 We selected two projects to illustrate what can happen when a contract is awarded before requirements are clearly understood.
24.36 In the case of the Canadian Forces Supply System Upgrade, $15 million was divided among three firms for what was called a "funded definition phase". This approach was a deliberate risk reduction strategy to enable potential contractors to develop a full understanding of a very complex requirement and thus develop precise proposals for the solution. The three firms worked competitively and concurrently to prepare their bid. The winning firm received an additional pre-implementation contract of $2.5 million to update its implementation proposal. However, this process of using a series of contracts to define the project failed to produce a clear set of requirements.
24.37 After being awarded a four-year development contract, the winning contractor needed the first nine months to confirm with National Defence its understanding of the requirements. By the end of that time, it had become clear that the project would take longer and cost more than expected unless the requirements were reduced. Establishing and confirming the system's requirements over shorter periods of time and identifying the specific deliverables before awarding the final contract would have helped the Department design a better contract and establish better development plans.
24.38 In the case of CAATS, the project evolved from another project started in 1986, the Flight Data System Modernization Program. The contract was signed in 1989. A year later, at the first major milestone, the contractor presented a description of what the new system would look like and do. However, Transport Canada rejected the document because it did not correspond with its own understanding of the requirements.
24.39 In June 1994, the contractor and the Department had not yet reached a common understanding of the requirements and, as the requirements for the latest milestone had not been met, the government stopped all progress payments ($230 million to that point). In May 1995, an agreement was reached on a reduced set of requirements and new approval was obtained from the Treasury Board in September 1995. Resolving a number of high-risk functions and other items that were removed from the contract is now the Department's responsibility. A further description of the renegotiated contract can be found in Part II - CAATS: Some Matters of Concern.
24.40 In May 1996, six years into the contract and two years before the system's final delivery date, there are still 11 unresolved issues relating to some operating concepts and to what will actually be built according to the renegotiated contract.
24.41 In both the CFSSU and CAATS projects, many dedicated people in the project management offices have struggled to keep the projects moving ahead, only to have to redo their work because requirements weren't understood or were poorly communicated. Clearly, all parties must thoroughly understand the system requirements as a prerequisite to managing the risks associated with any software development project. While it is true that requirements can legitimately change over time and that some cannot be detailed in advance, it is important that all parties agree to specified requirements for reasonable and realistic time horizons. Action plans can be created to deal with any vague or outstanding need that extends beyond predictable time horizons.
Improving software development processes: the Capability Maturity Model24.42 In all eight of the IT projects that we have examined over the past two years, the risk of falling behind schedule and running over budget has been very high. To deliver high-quality products, on time and within budget, developers of large software systems need to implement standards of quality in the processes they put in place. Whether departments acquire or develop their own software, they need a framework that encourages continuous improvement in software development practices must exist. To date, except for new directives issued in June 1996 with the Enhanced Framework, we have seen no evidence that the government has provided the methodologies necessary to help support the continuous improvement of software development.
24.43 What limited success there has been in developing government information technology projects has been the result of outstanding contributions by individuals. However, when those individuals leave, the ability of an organization to repeat the success is often lost. By contrast, a "mature" organization possesses an organization-wide ability to manage software development, software acquisition and lifetime software maintenance. Its success does not depend on chance or on individuals.
24.44 Mature software development organizations work hard to implement standards of quality that are well accepted by industry and governments, such as the International Organization for Standards models (ISO 9000) or, more specifically in the area of software development, the Software Engineering Institute's Capability Maturity Model (CMM). The Treasury Board Secretariat has included in the Enhanced Framework for the Management of Information Technology Projects the requirement to use the Capability Maturity Model ( Exhibit 24.4 ). The Secretariat sees the Model as the cornerstone of an approach to identifying and managing project risks in software development projects. In the longer term, the Model may help the government to improve software development capability by supporting continuous improvement in development processes. In the Framework, the Secretariat further suggests that the achievement of a specified maturity level eventually become mandatory for government departments and contractors. As organizations meet higher standards of quality in their development processes, the probability of obtaining the intended results in large information technology projects is increased.
Setting priorities24.45 To achieve a desired outcome or result in implementing large information technology projects, three key variables must be managed effectively: requirements, time and cost. In all projects examined, we found that compromise of these variables began immediately after the project was started. Generally, the first variable to be compromised is requirements, with the usual result that the system will ultimately do less than originally envisioned.
24.46 Typically, the next variable to be compromised is time. Either the time allowed for completing the project is increased or the project is implemented in stages or phases, which, in effect, extends the original completion date.
24.47 The third variable, cost, is usually the last to be compromised because the Treasury Board must normally approve any change in funding. Attempts to adhere to the project budget often translate into more compromise of time and requirements.
24.48 To adhere to the original project budget approved by the Treasury Board, costs are often adjusted by transferring direct project costs to other departmental budgets. For example, when the CAATS project was redefined in 1995, Transport Canada assumed a number of responsibilities that previously had been part of the CAATS contract. Equipment that was to be purchased as part of the original contract amount is now to be leased by Transport Canada at a total additional cost of $176 million over seven years. These new costs will be paid out of departmental budgets.
24.49 All the projects that we examined faced critical decisions on the compromises necessary to complete them. The time to decide which is the most important element - requirements, time or cost - is when the project starts, before major contracts are signed. Based on our work to date, single, large, lengthy, fixed-price contracts are not suitable for large information technology projects because they appear to limit management's ability to adjust to changed conditions. There is an urgent need to improve the procurement and contract processes for information technology projects.
24.50 As reported last year, one way to achieve more flexibility is to break down large projects into smaller, more manageable components. This position was endorsed by the Public Accounts Committee in its report on Chapter 12 of our October 1995 Report and by the Treasury Board Secretariat in its Enhanced Framework.
24.51 Implementing large projects in smaller pieces must be well managed. For example, the set of 11 projects in Real Property Services (RPS), when taken together, represents a very large system development effort. Individually, each project constitutes a manageable component of the whole. However, the benefit of dealing with the entire set of projects in smaller, more manageable components was lost when, for reasons of time and of funding from operational budgets, all the projects were run concurrently. A shortage of experienced and expert staff resulted. RPS senior management has determined that the implementation date (time) of the system is the highest priority, cost is the second priority and requirements is where the greatest flexibility can be exercised. Unfortunately, those priorities were not made clear to all the stakeholders and project teams. When the project priorities are not clearly understood at the start, time and effort are wasted later on clarifying understandings, reducing overlaps and reconciling organizational agendas.
Measuring the status of a project: a key to completing it on time and within budget24.52 Tracking the status of a software development project against its planned cost and schedule is critical to making decisions on managing that project. Accurate and timely use of performance data is crucial to delivering a useful product to the end users on time and within budget. An important characteristic of a mature organization is the existence and use of a balanced set of performance indicators to measure the progress of a project. In software development projects, these indicators are critical because little end-user value is really visible until much later in the project.
24.53 In 1992, the Treasury Board required all project leaders to use a project performance management system to manage all major capital projects (projects larger than $100 million). In 1994, this requirement was extended to all projects. We note that in 1993, government and industry adopted C/SPMS (Cost and Schedule Performance Management System) as the national standard. It conforms to international standards and provides a balanced measure of performance by evaluating performance against both planned cost and schedule. The Treasury Board is strongly recommending that all project leaders use C/SPMS.
24.54 In the case of the CAATS project, progress figures were produced as required under the 1995 renegotiated contract but management did not use the data as a tool for managing the project (for reasons outlined in paragraph 24.61). In Real Property Services, progress indicators such as C/SPMS were not used. Two other projects that we reviewed had mixed experiences with the use of performance data.
24.55 The CFSSU project adopted the C/SPMS standard. Management's analysis of the monthly performance figures showed rising costs and a slippage in schedule. Efforts were made to take corrective action; however, senior management did not recognize the full implications of these figures until our audit confirmed the project management's analysis of the situation. The trends in the data showed that the project would take twice as long and cost over $100 million more than expected if the project were to continue to completion with productivity at the current level. However, the trends showed productivity declining rather than staying constant. This indicated that the project would never be completed unless management took significant corrective action. At the time of our audit, project management was looking at alternative ways to meet the project objectives.
24.56 The Treasury Board Secretariat received the same monthly reports, as a courtesy, from the CFSSU project manager. Secretariat officials attended senior project meetings. However, the Secretariat considered the information gained from the meetings and reports to be "unofficial". Treasury Board Secretariat policy requires that the project leader formally communicate project performance information to ensure that accountability for project delivery clearly remains with the department. Despite having the information and a responsibility to monitor large information technology projects, the Secretariat did not assess departmental performance in meeting the objectives of the major capital projects policy, nor did it escalate the issue of the schedule slippage to the Treasury Board ministers. The Secretariat views its monitoring responsibilities as related to the effectiveness of its policies rather than to project management, which is a departmental responsibility.
24.57 In the case of TCCCS, the project predated the adoption of C/SPMS. Nevertheless, performance data conforming to the standard were provided by the contractor to National Defence.
24.58 The system that generated the performance data for TCCCS was separate from the one used to generate data for internal management purposes. As a consequence, the reporting system and the management system often presented different views of the status of the project. Efforts to change the system would have been costly and the overall project might have been 75 percent complete by the time the changes were made. Consequently, because the reported data were incomplete or contradicted other sources of information, the TCCCS management team had to develop alternative, time-consuming methods of tracking progress. As a result, management obtained performance data that were over two months old by the time they were assembled, which compromised their usefulness for decision making. The TCCCS management team compensated for the time lag with an ongoing analysis of the various project risks it had identified and with technical reviews.
24.59 Two other major software projects that we examined, CAATS in 1996 and ISPR (the Income Security Program Redesign) in 1995, were major capital projects that used the C/SPMS standard but only partially. In both cases, performance was tracked in relation to schedule but neglected in relation to cost. In the case of CAATS, Transport Canada is not using the cost performance indicators, and the method used to calculate the figures for schedule performance has severely compromised their usefulness for management purposes.
24.60 Cost performance is a critical indicator. It can show management that resources are being spent at a rate that will put the project under severe financial pressure before it is completed. We have observed that fixed-price contracts created a false sense of security for some managers and they paid little attention to cost performance indicators. A fixed-price contract does not necessarily mean that costs are managed. Project costs can increase dramatically while contract costs remain fixed. As mentioned in paragraph 24.47, financial pressure can cause compromises in the schedule or in what the final product can do.
24.61 In the CAATS contract renegotiations of 1995, the Crown agreed that the previous five and a half years of work would be deemed to be 100 percent "on time". This approach introduced severe distortions that have hidden the true schedule performance since July 1995. Even serious slippages in the schedule cannot be clearly identified until major deadlines are missed. Status reports from the prime contractor indicate that the CAATS project is very unlikely to be delivered on time and at currently specified cost and requirements (as per October 1995 amended contract). This is further confirmed by contract negotiations under way as of August 1996.
Implementing the Enhanced Framework for the Management of Information Technology Projects24.62 As mentioned, the Treasury Board Secretariat has produced guidelines in the Enhanced Framework for the Management of Information Technology Projects that can help to reduce the risks inherent in such projects. In view of the size of the investments in information technology and the savings possible through better project management, it is imperative that departments develop action plans to implement the guidelines in new and existing projects as soon as possible.
24.63 The first challenge for the Treasury Board Secretariat is to determine the actual size of investments in information technology projects. The Secretariat does not have a comprehensive list of all projects and their full costs. Only two of the systems that we examined this year are on the Treasury Board Secretariat's list of 25 projects: CFSSU and CAATS. The cost of these two projects totals $953.7 million, as approved by the Treasury Board. However, cost is presented only as $710 million on the Secretariat's list, or 26 percent of the total $2.7 billion. The other two projects that we examined, TCCCS and RPS, were not on the list. The Treasury Board approved TCCCS at $1.9 billion, including over $1 billion in communications equipment. The set of projects in RPS has a value of at least $50 million, by our estimate, but has never been submitted to the Treasury Board. A complete listing of all government projects with an estimate of the full life cycle costs does not exist. This information may be useful to inform Parliament of the total investment in these projects, as recommended by the Public Accounts Committee.
24.64 We estimate that technology projects in government represent well over $5 billion in investments over the life of these projects. Using the statistical findings of a recent study in its revised Framework, the Treasury Board Secretariat estimates that changing the way these projects are managed could help save over $1 billion in cost overruns on the $2.7 billion worth of projects on its current list. Assuming the same factors were applied, there could be possible savings of close to $2 billion on the current $5 billion worth of projects.
24.65 The Treasury Board Secretariat plans to develop the policy changes and the government-wide practices required by the Enhanced Framework by April 1997. As of May 1996, departments had been directed by the Treasury Board to implement the new Framework over the next few years. However, we are concerned that the lack of clear departmental action plans will allow many large information technology projects already under way to lose the benefits of the new Framework over those years. Until the Framework is firmly established in the departments, the Secretariat will need to provide support and monitor compliance very closely if the government is to reap the expected benefits.
Report Card on the Four Projects Reported in October 199524.66 In 1995, we reported that the Income Security Program Redesign (ISPR) needed continuing corrective action by senior management. Intensive negotiations in the past year to manage project risks and to accommodate legislative changes have resulted in a proposal to submit to the Treasury Board a revised schedule and budget as well as some changes in the contracted deliverables. The project requires a schedule adjustment of 11 months and an increased budget to accommodate change of approximately $80 million. At the time of writing this report, no decision had been made. Any additional change to government policies and programs could further affect the cost and schedule of ISPR. Risks remain very high that further changes in schedule, cost or requirements will be necessary to complete the project; management needs to continue taking strong action to mitigate these risks as much as possible.
24.67 Since we reported last October, the development of the Common Departmental Financial System (CDFS) has been concluded and the system is being implemented throughout Public Works and Government Services Canada. In addition, Correctional Service Canada and eight small agencies have adopted CDFS as their financial system. Adjustments are still being made to the system to make it fit better with other existing systems and to improve its performance.
24.68 The Integrated Departmental Financial and Materiel Management System (IDFS) has been implemented at Transport Canada. In addition, a group of users has been formed and is aiming to upgrade the system by April 1997. The cluster group includes Transport Canada, the National Film Board, the National Research Council, Environment Canada, Fisheries and Oceans and the Vancouver Airport Authority.
24.69 As reported last year, the Public Service Compensation System (PSCS) was terminated.
Conclusions and Recommendations24.70 Our Office fully supports the use of information technology in government to manage costs and improve service to the public.
24.71 In our audit this year, we focussed on six key factors that we believe greatly influence the success of systems under development in government:
- It is essential to clearly define the requirements of the system for reasonable periods of time prior to awarding long-term fixed-price contracts or starting development work.
- Senior management, including (but not limited to) the project leader, the project sponsor and the project manager, must be closely involved in any systems development project.
- Organizations carrying out major IT projects must have the "maturity" or capability to do so.
- Priorities for time, cost and requirements must be set at the start of the project, be well communicated to all parties and be reviewed regularly so that the planned results can be achieved.
- Accurate, timely and effective performance measurements must be used to manage the project.
- Action plans need to be developed by departments to implement the Enhanced Framework for the Management of Information Technology Projects.
- effective user involvement and commitment to the success of the project;
- the experience and expertise of the staff dedicated to the project; and
- the separation of large projects into smaller, more manageable components, each of which provides an improved capability to the organization.
24.74 We are pleased that the Public Accounts Committee has taken a keen interest in the results of our 1995 audit of systems under development. The Committee stressed three areas for action in systems development projects:
- sponsorship, ownership and accountability by senior management;
- full commitment and true accountability at all levels to carry a project through to completion; and
- implementing systems in smaller, more manageable components.
24.76 To support the work started by the Treasury Board Secretariat and continued by the Public Accounts Committee, the following initiatives should be taken by the Secretariat and the departments:
- All parties should formally agree that requirements for reasonable periods of time have been clearly defined and understood before a fixed-price development contract is signed or development work begins.
- A professional development program for senior project management (project leaders, sponsors and managers) should be implemented.
- Before a department begins a software development project, a suitable rating according to the Capability Maturity Model, as recommended by the Treasury Board Secretariat in the Enhanced Framework, should be considered as a criterion for selecting the system developers.
- Appropriate performance indicators or measures should be put in place for managing information technology projects.
- A project leader at the assistant deputy minister level should take charge and be fully involved in very large projects ($100 million or more) and, as we recommended last year, there should be a project sponsor to work in close support throughout the development process.
- At the start of the project, a document setting out the results to be achieved and the relative importance of time, cost and requirements should be produced. These priorities should be communicated to all parties and reviewed regularly for their continuing appropriateness.
- The Treasury Board Secretariat should co-ordinate with the departments to produce action plans that will implement the principles discussed in the Enhanced Framework for the Management of Information Technology Projects.
Departments are implementing the Enhanced Framework for the Management of Information Technology Projects approved by the Treasury Board. Work is proceeding in a review of the information technology procurement process, in developing a project manager and project leader development and support program and in identifying improved practices, methodologies and tools for managing projects, including project size, project review, monitoring and approval, risk assessment and management, and change management. As mentioned by the Auditor General, the Software Engineering Institute's Capability Maturity Model is being studied as a model for government.
Many existing projects, like those audited by the Auditor General this year and last, suffer from inadequate past practices that cannot be fully corrected retroactively and are being carried out under contracts that limit departments' ability to negotiate changes. Departments are managing these projects closely but they still pose high risks. The Enhanced Framework will help departments to manage their existing high-risk projects. However, its primary benefit is expected to be significant improvement in the government's success for new information technology projects.
We appreciate the continuing support of the Auditor General's Office and look forward to its continuing involvement in the work of improving the government's successful completion of large technology projects.
Department of National Defence's response: The Department generally concurs with the content of this report.
For all major projects, the Department assigns both project sponsors and project leaders in addition to the project manager. For some projects, the project sponsor and leader roles may be assigned to the same person. These people normally have extensive project management experience in addition to the formal training provided to all departmental project managers.
On the CFSSU project, the findings of the Auditor General served to confirm departmental management's assessment of the problem areas identified several months before the audit. Senior management, including the interdepartmental Senior Project Advisory Committee, was fully involved and had engaged a highly regarded consulting firm to assist in confirming appropriate corrective action.
Public Works and Government Services Canada's response: The Department generally concurs with the findings of the Auditor General and shares the concerns expressed with respect to the risks of systems under development.
Public Works and Government Services Canada (PWGSC) agrees with many of the Office of the Auditor General's (OAG) conclusions; however, the proposed remedies are less persuasive, as these recommended improvements have been used in the past. The recommendations focus on processes such as reporting, risk analysis, performance indicators, and scorecards; this belies the human factor in the successful completion of projects. The Department believes that competent individuals and project management skills are fundamental to the success of projects. Therefore, the following paths should be considered by the government:
- development through such proven training in project management at the departments of National Defence and Transport, the Project Management Institute (PMI) and the PWGSC Institute;
- incentives to attract and retrain staff, and management of the incremental progression of these valued resources across departments.
PWGSC continuously aims to improve its software and systems development processes through various initiatives. It is the first department to use the Software Engineering Institute (SEI) software risk assessment methodologies and tools to facilitate systems under development (SUD) audits; furthermore, the full suite of SEI tools is being considered for facilitating audits and for transferring this technology to the business line organization. In addition, PWGSC has adopted this risk management approach to be implemented in each SUD project. The Capability Maturity Model (CMM) mentioned by the Office of the Auditor General was adopted by a departmental branch as a roadmap to improving its software development processes.
More specific to the PWGSC systems reported in the Auditor General's chapter, while there are 11 projects under way within the RPS Systems Framework, only 9 are application development projects. PWGSC agrees that concurrent development presents more risk than if projects are developed one at a time; however, it presents less risk than if the projects had been defined as one mega project. It should be noted that these projects were thought out and planned within a strategic framework, the purpose of which was to mitigate the risk of developing these systems concurrently. Although the timing of the development effort overlapped, it is a result of a need to proceed with these systems within a specified time period in order to meet our business and client needs. In order to mitigate future risks, PWGSC is currently revising its Strategic Information Technology Plan with respect to the RPS systems. At the same time, PWGSC is reviewing the legacy systems and systems under development to rationalize and streamline these initiatives. An initiative is currently under way for situations where departmental staff are involved in a risk assessment audit of a RPS software development project.
The Department recognizes a need for RPS systems to be developed in a timely fashion within a finite budget; however, time, cost and requirements are not mutually exclusive. There are occasions where project timelines may need to be extended to ensure adequate functionality. It is PWGSC's objective to obtain a high-quality product that meets all functional requirements on time and within budget.
With respect to the CAATS project, PWGSC was responsible for the contractual and legal elements of the project.
The renegotiation of the CAATS contract was a complex process that took into account many factors, including contractor claims, outstanding differences between the Crown and the contractor and changes in scope of the work package. As a result, the amended contract was not prepared on the basis of integrating all of the elements in the original contract into the new contract. Rather, it was structured based upon available funds and a redefinition of the required work.
While the cost of putting the contract in good standing could have been presented in a different manner for greater clarity in documentation, PWGSC fully described the complex nature of the negotiation exercise to the government. PWGSC did, to the best of its ability, clearly disclose fully all aspects of the contract restructuring.
Part II - CAATS: Some Matters of ConcernExhibit 24.5 for a brief summary of the project.) The core of the project centres around a contract with a single supplier - hereafter referred to as the prime contractor - charged with assembling the hardware and writing the software. The contract with the prime contractor was let in December 1989 for a price of $377 million, with final delivery scheduled for August 1995. Through successive amendments, the contract now stands at $500 million, with final delivery scheduled for July 1998.
24.78 In March 1995, the Minister of Transport met with the Auditor General to advise him of concerns related to the CAATS project that had recently come to the attention of the Minister. He indicated that, in his view, the project merited our attention. This view was later reiterated publicly by the Minister in statements to the House of Commons. The project, which we were already monitoring, was also high on our list of audit priorities, and the conjunction of our priorities and the Minister's request resulted in the decision to do this audit.
24.79 Because the project was one of the larger systems under development in the government, we made an early decision to link our work with that of the Systems Under Development (SUD) audit team. It, in turn, was interested in examining CAATS as a case for its audit.
Assignment to Nav Canada24.80 During the survey stage of our audit, an agreement was signed to sell the Air Navigation Service (ANS) to a private corporation - Nav Canada. One feature of the agreement was the intent to transfer the CAATS project to Nav Canada as part of the sale, and to assign the contract to the corporation. This meant that on the date of transfer, Nav Canada would assume the assets and liabilities, as at the date of transfer, of the Crown with respect to the CAATS contract and that those responsible for the project would be working in the private sector.
24.81 Accordingly, we decided to focus most of our audit attention on matters for which the lessons to be learned would remain valid for future government projects of this scale. At the same time, there were certain matters related to the project that lay outside the normal focus of our work on systems under development that we believed would be of interest to Parliament. We report on those matters here.
The Growth of CAATS24.82 The CAATS contract was let in December 1989. Within the first year, delays were encountered. By the spring of 1994, there was increasing concern on the part of the CAATS Project Office about whether the work could be completed according to the original schedule. In June 1994, two events transpired. First, the Project Office agreed to accept a milestone, due that month. Acceptance was conditional on certain other events, including a full review of the schedule for the contract. Second, because of concerns about progress, the contract authority (Public Works and Government Services Canada) decided to suspend further payments to the prime contractor pending the results of the review. At that point, $230 million had been paid to the contractor. When the review was completed, late in the fall of 1994, the results indicated that the project was 15 months to two years behind schedule and that it could not be completed within the limits of the price of the current contract, which stood at $419 million. Additionally, the review concluded that there had been significant technical divergence from the direction set out in the original contract. The reasons for this were beyond the scope of this audit.
24.83 These findings led to significant changes in the management of the project. A new Project Director was appointed, the Assistant Deputy Minister (Aviation) was designated the Project Leader and the Deputy Minister assumed the role of project sponsor. From this point on, they assumed ongoing leadership roles in the project, chairing key internal committees and communicating regularly with the senior management of the prime contractor.
24.84 In January 1995, under the leadership of a new Project Director, the Project Office and the contractor began discussions on a possible "rebaselining" or restructuring of the project. In March 1995, at the request of the government, the contractor submitted a proposal indicating that it would require a contract price of $764 million (including the $230 million already spent) to complete the project (the existing contract then stood at $419 million). The requested amount was not acceptable to the government.
24.85 Faced with this situation, the Department believed that it had two options. It could terminate the contract or seek to negotiate a reduced system at an affordable price. An affordable price was one that would stay within the original overall cost ceiling - the effective project approval (EPA) - of $659 million established by the government for the project, and would result in a contract price of not more than $500 million. Should a renegotiated contract be possible, continuing with the project would mean that the Department faced a further cash outlay of $270 million on the contract. The Department believed, however, that there would be a reasonable prospect of receiving a functional system at the end of the contract - albeit, a system with less capability than it had originally hoped to get. Alternatively, if the Department decided to terminate the contract, it believed it faced the prospect of receiving very little of value for the $230 million already spent, and the likelihood of being sued by the prime contractor for an amount equal to or greater than the cost of finishing the project.
24.86 After considering the alternatives, the Department decided to enter into negotiations with the prime contractor. The main objective was to maintain a project that would be operationally sound, while restructuring the contract so that the original overall cost ceiling (the EPA) of $659 million would not be exceeded.
24.87 Those negotiations were concluded in September 1995 with a rebaselined contract. The price for the new contract was $500 million, the total cost of two contracts: the main contract for CAATS with a price of $478 million; and a contract for the voice switch for CAATS, priced at $22 million. The voice switch was part of the original contract and was separated from it during negotiations. When we refer to the post-rebaselining contract, these two contracts are included together.
24.88 Transport Canada reported that, coupled with other changes to the project (discussed in the next section), the revised contract permitted the Department to complete the project while remaining within the overall approval level of $659 million. In Part III of its Estimates for 1996-97, the Department reports the cost of CAATS at this approved value ($659 million) plus an additional $75 million in associated costs for training, installation and leasing.
24.89 Exhibit 24.6 presents a breakdown of project costs. The first column presents the original EPA cost ceiling as approved by the government for CAATS in November 1989. The second column gives the EPA costs as proposed by the Department and approved by the government in September 1995. It also includes certain additional costs to arrive at a fair statement of the total project cost for the reapproved project, calculated on a basis consistent with the original EPA. These additional costs represent two types of costs: first, those costs included in the original contract and thus in the original EPA; and second, costs that must be incurred in order for the CAATS system to operate.
24.90 The most significant item in these additional costs is a pair of leases for the hardware and commercial software for CAATS valued at $176 million (see line 10 and note 3). Under the original contract, the prime contractor was required to design, assemble and supply these items as part of the contract deliverables. Under the renegotiated contract, the contractor was relieved of the obligation to pay for the items as part of the contract and instead the Department was to lease them from a third party. The capital price of the items to be furnished under the main lease was capped at $103 million and the contractor is required to furnish them at that price. According to the Department, should the cost of the items, the requirements of the software or any other changes not explicitly directed by the CAATS Project Office cause the price to rise above the cap, the excess costs would be on the contractor's account. Additionally, the Department has informed us that the lease was structured in three stages in order to minimize the financial exposure of the Crown in the face of both the likelihood of the project being transferred to Nav Canada, and the continued technical risk of CAATS.
24.91 The leased hardware and software are essential to the operation of CAATS and, in our view, the omission of these costs and other like costs from the revised EPA results in the true cost of CAATS to the Crown being understated by $217 million (line 15). In preparing Part III of its Estimates, the Department followed the convention of reporting for Major Crown Projects the following: the effective project approval ($659 million) and only those additional costs that would be incurred to the end of the main contract ($75 million). However, in doing this, the information provided suggests that the renegotiated contract was achieved with no material increase in the cost. As a consequence, this presentation results in Parliament not being fully informed of the real cost of the decision to rebaseline the CAATS project.
Declining content, rising price24.92 As a result of the negotiations, significant changes were made to the scope of the project (changes in what the software would do), the hardware to be delivered and the price. The price of the contract grew by $73 million, from $427 million before rebaselining to $500 million afterwards. Exhibit 24.7 presents, in summary form, the information prepared by the departments (Transport and Public Works and Government Services) for consideration by the government in seeking its permission to enter into the new contract and to justify to the government the increase in the price of the contract. This information is drawn from a more comprehensive document describing a number of aspects of the restructuring of the contract and the leases.
24.93 As indicated earlier, one of the major changes in the contract, resulting from the rebaselining, was to take the purchase of the hardware and some related commercial software out of the contract and to set up a lease for the computers, work stations for the controllers and some of the software. In addition, a number of features were removed, such as the ability of the system to propose possible resolutions to potential conflicts between aircraft. The scope of the contract was also reduced significantly by the decision to implement CAATS at 23 towers instead of the 60 originally proposed. The Department believes that the removal of certain of the features, such as conflict resolution, from the contract, had the benefit of greatly reducing the overall technical risk associated with the CAATS project.
24.94 The items removed were presented by the departments to the government as having a price of $227 million (line 14). In arriving at this figure, the price of the hardware removed from the contract was shown as approximately $92 million (note 2). We were concerned about the extent to which that figure represented the correct price of the hardware removed. When we asked for a listing of the hardware prices, based on the price of the items at the time they were contracted for, we were informed that none was available.
24.95 We would have expected that before entering into negotiations to remove the hardware from the contract deliverables, the Project Office would have established the original price of the hardware in the contract and the price of hardware added though contract amendments. We anticipated that this would have been done so that the Department could ensure that it received full credit from the contractor for the deletion. We found that this had not been done and that the Department did not know the original hardware prices.
24.96 Accordingly, we used available information and a variety of approaches to estimate the price of the hardware at the time it was put into the contract. Our analyses yielded a range of estimated prices and, to be conservative, we have chosen to use the lowest figure ($103 million). This means that the price of the items removed from the contract is understated, we estimate, by $12 million (line 19 and note 2).
24.97 The decision to lease the hardware and certain items of the COTS (commercial off-the-shelf) software rather than purchase them through the contract affected not only the price to be removed from the contract, but also the pricing of several of the new engineering change proposals (ECPs) that were added to the contract. We found that while items priced at $19 million (line 20) were removed from ECPs to be leased instead, the price of those items was not removed from the ECP prices (line 2) that were used to justify the contract price. As a consequence, the price of the new ECPs (line 2) is overstated by $19 million in the material presented for the government's consideration.
24.98 At the same time, changes in the approach and in the needs of the Air Navigation Service led to new elements and costs being added to the contract. These included costs such as $54 million to be incurred as a result of changing the approach to the transition from the current control system to CAATS. In total, the cost of these additions to the contract was reported to the government as $364 million (line 5). However, this amount includes $187 million for an engineering change proposal - ECP 232 (line 3). Our inquiries have shown that during the negotiations for the rebaselined contract, a decision was made to reject this proposal.
24.99 According to the Department, ECP 232 is not properly an ECP, but rather is a claim to be paid an additional amount for the costs of work arising from an earlier engineering change proposal. That proposal (ECP 60R4) had been incorporated into the contract as part of Amendment 10 at a price of $12 million. The work related to the preparation of a series of what were called operational requirements definition documents. In ECP 232, according to the Project Office, the contractor was asking for additional funding as the work had become broader in scope than anticipated.
24.100 The Department has informed us that it rejected the ECP because it did not wish to accept any liability for the charges made by the prime contractor in the claim. Accordingly, the Project Office made no attempt to verify any of the financial data in the claim in order to assess whether the work done justified the price asked. Nonetheless, in the interests of continuing the contract, it decided that it wished to credit the contractor in the price negotiations with an amount equivalent to the price ($187 million) claimed in the ECP .
24.101 This item, the most costly by far of the items justifying the price increase, was presented to the government as part of a list of ECPs representing new value to be added to the rebaselined contract. In fact, it was not a new ECP; it was included in the settlement made without admission of liability and represented part of the cost of resolving the differences between the Crown and the prime contractor.
24.102 In sum, our review of departmental documentation including the contract price justification, prepared by the departments, led us to conclude that the justification did not present the full price of the hardware and COTS software removed ($92 million as opposed to $122 million for a difference of approximately $31 million (lines 19 and 20)). Additionally, the contract price justification did not adequately disclose the nature of the $187 million addition to the price attributed to ECP 232. This means that our analysis suggests that the government agreed to a contract price of $500 million on a contract where a consideration of the price of the items removed from the original contract and the cost of the items added would only justify a price of $282 million (line 22 and note 2).
24.103 In our opinion, the material prepared for consideration by the government in approving the revised contract should have clearly shown that the revised contract price of $500 million included a component that represented the cost to the Crown of putting the contract in good standing, settling all past differences and disputes among the parties to the contract, and receiving from the contractor a formal release to that effect. While it is not possible to calculate the exact amount, our review of departmental documentation, the price justification and discussion with management leads us to conclude that the settlement cost was approximately $200 million.
The Role of the CAATS Project Office24.104 The CAATS Project Office is led by a Project Director who is appointed by Transport Canada. It also includes a number of staff from Public Works and Government Services Canada who are responsible for the legal and contractual elements of the project. In the case of CAATS, the role of the Project Office is to act on behalf of the government and, more specifically, the Air Navigation Service, to acquire the necessary hardware and software to meet the specifications in the contract. Because the contract in this case is a "firm fixed price" contract, the role of the Project Office is a relatively limited one. First, it must validate the design proposed by the prime contractor to ensure that it meets the specifications in a way that is compatible with the larger air navigation system of which CAATS is to be a part. Second, it must monitor the project and control the flow of cash to the contractor. Finally, it must verify that CAATS, as built, meets the specifications and is safe to use for air traffic control. We looked at the extent to which the Project Office fulfilled certain aspects of each of these areas of responsibility.
Design validation24.105 There was little by way of a system design at the outset of the contract. The capabilities that CAATS was to achieve were established, but the design issues of how those capabilities were to be achieved were not specified at that point. These issues, however, were a key part of the early stages of the contract. In the system design process, it is the role of the Project Office to do two things. First, it must communicate the specifics of the client's (the Air Navigation Service) needs to the prime contractor. That is, it must be able to clarify to the contractor, for example, what a conflict prediction function would mean in the context of Canadian air traffic control practices and the management of Canadian air space. Second, the Project Office must assess the design proposals prepared by the prime contractor to ensure that they will fully meet the contract requirements and will integrate safely and functionally into the existing system.
24.106 We observed that throughout the project, the approach of the CAATS Project Office to the design validation was responsive, based largely on reacting to the proposals and questions of the prime contractor. Given that the lead role in the contract lies with the prime contractor, who is responsible for preparing and implementing the design, we would have expected a certain level of responsiveness in the role of the CAATS Project Office. However, the Project Office should also have had a plan to work with the prime contractor to achieve closure on the design issues by clearly communicating the requirements of the air navigation system to the prime contractor. When we asked for such a plan, staff indicated that a strategic plan for the Project Office had never been prepared.
24.107 The CAATS project had been under way for nearly five years in the fall of 1995. Yet at that time there were still six major (and many more minor) unresolved design issues. According to the prime contractor, the failure to achieve closure on these issues had significant effects on project progress. The unresolved issues included matters as fundamental as the interface between the radar data processing system (part of the existing and ongoing Air Navigation Service) and CAATS. Given that one of the fundamental purposes of the CAATS project was to integrate the information from the radar data processing system with flight plan information, the failure to resolve such a central issue for the whole system, and similar issues, was an indication of the problems of the Project Office in achieving its objectives in design validation.
24.108 To respond to this problem, the CAATS Project Office began, in late 1995, to establish what it called integrated product teams. This involved assigning Project Office staff to work in tandem with their prime contractor counterparts to try and expedite the resolution of key design problems.
24.109 We observed that in the period since rebaselining, the CAATS Project Office has had some success in resolving many, but not all, of these design problems.
Monitoring progress24.110 Another key role of the CAATS Project Office is to monitor the progress of the project. This involves two types of activities. First, the Project Office needs to work with the prime contractor to develop a performance measurement system that can be used to monitor progress against both cost and schedule. Second, in projects such as this, which involve significant technical and production risk, the Project Office is expected to have in place a process to monitor project risks on an ongoing basis and to identify steps that it can take to mitigate identified risks. Before rebaselining, there was little formal activity carried out in the Project Office to address either responsibility. However, since rebaselining, steps have been taken to respond to both.
24.111 Performance monitoring . The prime contractor has put in place a project performance monitoring system called an "earned value system". According to the system description provided to the Project Office by the prime contractor, the contractor's earned value system has most of the critical attributes of the types of systems recommended by the Treasury Board Secretariat for such projects. The Project Office receives from the contractor monthly reports on progress against plans for the project as a whole and for subcomponents of the projects. In addition, individual managers receive progress against plan information on a weekly basis. However, the overall project information provided to us by the Project Office was often between six and eight weeks out-of-date by the time it was received and the weekly information appeared to be two to three weeks out-of-date. Project Office staff informed us that they have performed extensive checks on the reliability of the information coming from the system and are satisfied that the information is sound.
24.112 Risk assessment . We were informed that before rebaselining, risk analysis was not part of the management process in the Project Office. However, since rebaselining, the Project Director has appointed a risk manager who is producing a monthly series of risk analyses of the project.
Managing the cash24.113 Managing milestones. The original contract established a series of milestones, or technical achievements, with corresponding dollar values. The milestones were to be verified by the CAATS Project Office using predefined checklists. While the prime contractor was working toward a milestone, it was to be reimbursed for costs incurred up to 75 percent of the total value of the milestone. If the milestone was not met at the scheduled time, payment was to be stopped until it was completed. At that time, a further 20 percent of the milestone value could be paid, with the remaining 5 percent to be paid on completion of the contract.
24.114 Under these circumstances, control by the Project Office over the milestones was critical to ensure some degree of correlation between proven progress and the Crown's growing financial exposure. However, problems with milestones were evident from the start of the contract. Milestones were consistently allowed to be passed with significant lists of outstanding items attached. Indeed, some items were passed from milestone to milestone without ever being resolved.
24.115 The Department was to complete a checklist to assess the completion of each milestone. For the first milestone, System Design Review (for which $24 million was paid by January 1991), we were informed that the Department did not use the 38 pages of checklists required for this review, but paid for the milestone nonetheless.
24.116 A similar situation arose at the second milestone. Its checklist shows that half of the milestone, relating to interface requirements, was not completed. However, a certificate of completion was issued and the 20 percent milestone payment was made.
24.117 This practice leads to a "bow wave" effect whereby each successive milestone becomes more difficult to achieve than was intended, and the list of postponed work grows, increasing the likelihood of project failure. At the same time, the gap between the actual value of the work achieved and the amount of work the Crown had paid for grows as well, further increasing the risk of project failure.
24.118 On four significant occasions prior to the rebaselining, 75 percent of the milestone value had been paid for, yet the milestones were not delivered. On each occasion, the government stopped making payments to the contractor. However, the resolution in each instance was ultimately to amend the contract and to restructure the milestones by shifting more of the requirements from the current to future milestones, thereby permitting payments to resume and the project to proceed.
24.119 The initial stop-payment was issued in October 1991, when the prime contractor had reached the 75 percent payment limit of the second milestone. At that time, some $56 million had been paid. The stop-payment was issued in October 1991 and the second milestone was not met until November 1992 when $16 million, representing the balance of its assigned value (less holdback), was paid. This payment was made possible only because of an amendment to the contract that changed the milestone schedule.
24.120 From November 1992 on, the CAATS Project Office began a pattern of paying for one milestone and, at the same time, paying 75 percent of the next milestone because the prime contractor's progress claims already exceeded that amount. An example is the case of the second milestone, as discussed in paragraph 24.119. In November 1992, a payment of $16 million was made based on its completion. At the same time, a payment of $9.1 million was made representing progress claims for 75 percent of the price of the third milestone, which was also eventually late. This payment process continued until 1994 when, after payments had again been suspended, it was decided that the whole contract had to be renegotiated. The cumulative amount paid then stood at $230 million, out of a contract price that was then $419 million.
24.121 Milestones restructured . Under the rebaselined contract, milestones remain the critical point for payment control. As before, each milestone has a pre-established value. However, between milestones, progress payments are based on dividing the value of the milestone into roughly equal monthly amounts. These amounts are then modified by applying an index intended to reflect progress in the work. If progress for the payment period is less than planned, the payment is reduced and the amount of the reduction is held back until the milestone is completed. Once a milestone is missed, payment is stopped until the milestone is completed. In lieu of a cash holdback, the rebaselined contract requires the prime contractor to maintain an irrevocable letter of credit in favour of the Crown in an amount equal to five percent of the cumulative amount paid under the contract.
24.122 Since rebaselining in the fall of 1995, the prime contractor has successfully met a series of smaller milestones. The most recent - milestone 10 - was delivered slightly late, at the end of March 1996, and was deemed complete. However, significant elements of that milestone remain to be resolved. Finally, in the summer of 1996, when it became clear that the next milestone would be missed by several months and that the contracted delivery date of the final product was again in danger, contract renegotiations were begun again. One feature of the discussions was a further restructuring of the milestones.
Verification and safety24.123 Verification is achieved by various means . The third key function of the CAATS Project Office is to verify that the system, as built, fully meets the specifications and is safe for use in "live" air traffic control applications. To achieve the required degree of assurance in a system of this size (over 600,000 lines of computer code) and complexity (involving 23 commercial off-the-shelf (COTS) software products), various approaches are necessary. Most experts in the field of software testing now believe that for systems of this size, assurance cannot be achieved by testing alone; the possible set of permutations and combinations of events is so large as to be practically infinite. As a result, verification is usually achieved by combining approaches, such as ensuring that the computer code is written in a disciplined and well-documented way; applying analytical techniques to the computer code that has been written; and testing programs at successive stages of development, from the subroutine level up to the fully integrated system.
24.124 Although we discuss verification as the last function of the Project Office, it is by no means only an end-of-the-project task. If proper assurance is to be achieved, we would expect that the verification process would begin in parallel with design validation, and continue through the development of a verification plan by the Project Office. Such a plan would classify program functions by their criticality, establish accordingly the degree of assurance that the Project Office desires from the verification of each function or combination of functions, and establish the actions and tests to achieve that assurance.
24.125 No overall plan. When we asked the Project Office for its plan to achieve such assurance, we were informed that it did not have one. We were further informed that the prime contractor follows a disciplined and well-documented software development process that includes a series of reviews as each unit of code is designed, written and tested. Project Office representatives are permitted to witness all of these activities and do so on a selective basis. The contractor has also developed a series of other plans for testing as program integration takes place, testing once the system is assembled and, finally, testing once the system has been installed. The Project Office representatives review, suggest changes to and approve these plans, and again are permitted to observe all aspects of these tests.
24.126 The CAATS Project Office had not established the desired level of assurance for determining the degree to which the system fulfills the specifications, nor could it show how its review and observations would ultimately lead to achieving such assurance. Accordingly, we conclude that the Project Office has not established the degree of assurance it needs to achieve from its verification activities, nor is it currently able to ascertain what level of assurance it will receive from the activities that it and the prime contractor are, and will be, carrying out.
24.127 Importance of verification for safety-critical systems. Verification is an important activity of the Project Office for any software development project to ensure that the Crown receives value for the money paid. However, the importance of verification rises sharply when the software, or parts of it, are critical to safety. CAATS is a safety-critical system and thus assurance of system safety is of paramount importance. In operation, CAATS will be the controllers' sole source of much of the information they need to safely route air traffic and to prevent losses of separation between aircraft. If that information is wrong, for any reason, then the likelihood of an accident and the potential for loss of lives increases.
24.128 In spite of the fact that CAATS is a safety-critical system, the CAATS Project Office has not been able to provide any evidence that it has initiated any independent safety process or hazard mitigation plan of its own. Although the expertise in air traffic control systems and their safety resides within the Department, we were informed that the Project Office has no documented plan to deal with system safety. While it has a representative on the System Safety Working Group, which is part of the contractor's safety process, the Project Office is relying almost completely on the prime contractor for safety of the delivered system.
24.129 Management support needed. Good safety practices must be supported by top management to foster the culture necessary for an effective safety program. One of the ways management demonstrates its commitment is by appointing a manager with the authority and mandate to ensure that safety is given the necessary attention and priority. In a project of this scope and complexity, we would have expected the appointment of a full-time safety manager, reporting directly to the Project Director.
24.130 Following the August 1995 retirement of the Project Office staff person responsible for performance engineering and system performance, performance and safety responsibilities were transferred to the Software Engineering Manager, and the responsibilities for reliability, maintainability and availability and for quality assurance were transferred to the Systems Engineering Manager. With this division of responsibilities, two individuals working in separate parts of the Project Office have each spent part of their time working on safety-related issues. However, in September 1996, the CAATS Project Office announced the appointment of a Safety Manager.
24.131 Some efforts made. System safety has not been wholly ignored. In 1990, the prime contractor developed an initial safety plan, titled the "Reliability, Maintainability and Availability (RMA) Program Plan". In 1992, the contractor engaged a consultant to evaluate the existing safety of the CAATS project, and to propose a new direction for future work in this area. The consultant's report provided an evaluation of safety for the CAATS project, a short description of state-of-the-art safety analysis, a list of recommendations on how to perform future RMA tasks, and a list of suggestions on how to modify the 1991 version of the RMA Plan. The report also described a number of items that the consultant considered to be responsibilities of the CAATS Project Office and identified specifications that the prime contractor needed from the Project Office in order to define the program safety requirements better. These included such things as: formal specifications for the definition of functions; data flows and data storage; the identification of critical functions with the safety objectives stated for them (by stating controllers' responsibilities, definitions of hazards and severity of hazards, and by performing a hazard analysis on the air traffic control system); a procedure to test the critical functions, verifying that the controllers would actually use the system for its intended purpose; and an "exhaustive" review of documentation to ensure consistent, unambiguous specification of critical functions.
24.132 The CAATS Project Office was made aware of this report in 1993, and we expected that it would have responded by producing the analyses recommended by the consultant. However, the Project Office has not responded in any way to the recommendations.
24.133 The prime contractor's RMA Plan has been amended since the consultant's evaluation, with the most current version (according to the Department) being the August 1995 revision now titled "Safety Case". Our review of that plan suggests that it is quite complete, based on the project description of the day. However, the rebaselining of the project was only partially completed at the time the document was prepared. Several key areas in the document contain notes to the effect that it will need to be revised to reflect the new baseline when that is established. Unfortunately, the rebaselining has substantially changed many parts of the project with the consequence that the 1995 plan will need to be updated to apply to the project as now defined.
24.134 Use of COTS software products increased . The planned use of COTS (commercial off-the-shelf) software products has escalated tremendously as the CAATS project has been redefined. Most COTS software products are designed for non-safety-critical use and come with very limited warranties. Further, many of these products are designed as either stand-alone programs with little, if any, consideration of operability with other applications or as programs designed to operate in a limited environment with a few other programs. It will be extremely important that the individual and integrated responses from performance testing of COTS software products are carefully studied initially and, for each subsequent version released, before implementation in the functioning system. Evaluations of COTS software products must be frequently revisited, since the project duration has already outlived some of the products that were available at the first writing of the RMA Plan.
24.135 Conclusion . The current approach being taken by the CAATS Project Office to system verification and safety appears to be relying very heavily on observing the testing by the prime contractor. Extensive testing of the system may aid in gaining assurance that the required functions exist in the system, but it is unlikely to provide adequate assurance of the installed system's overall reliability and safety.
24.136 The CAATS Project Office has not described any supplementary process to gain assurance that the system will operate reliably, consistently, over a long duration, and at an acceptable level of performance for critical functions.
Subsequent Events24.137 On 19 September 1996 the CAATS Project Office and the prime contractor completed their negotiations in the most recent attempt to bring the contract into good standing. Their agreement, if approved, will become Amendment 18 to the contract. The most significant change is the moving of the planned delivery date back six months from July 1998 to January 1999. If this date is met, CAATS will be delivered some 41 months later than originally planned. There are minor price increases in the contract, for engineering changes, and in price of the equipment to be leased to accommodate increases in equipment quantities. To improve payment control, the number of milestones has been increased and their value, on average, decreased. The assignment of the contract to Nav Canada remains a matter to be resolved between Nav Canada and the prime contractor.
Transport Canada's response: The Department definitely agrees that CAATS has to date been a very problematic contract. When it was apparent that the CAATS project had encountered serious problems, Transport Canada took decisive action to correct them and is still doing so. While active consideration was given in early 1995 to cancelling the contract, we made a business decision to proceed with a fixed-price contract. This approach was designed to limit costs while still ensuring that there would be a functional product at the end of the contract.
The audit only refers to the role of the Department in managing the contract and does not refer, directly or indirectly, to the role of the contractor. It is the view of the Department that a share of the problems belongs to the contractor. The Department has to date needed to take corrective action not only to get its own house in order but also to ensure timely and effective delivery of milestones and key products by the contractor.
The audit attempts to make a direct comparison of the prices and costs in the contract prior to its renegotiation in October 1995 with the prices in the revised contract. The Department took a fixed-price approach and negotiated a global price within which all aspects of the revised contract had to be considered. The adjustments that had to be made by both sides were defined as a package and, as a result, it is extremely difficult to make a direct comparison of the numbers before and after the rebaselining of the contract.
We acknowledged that the costs associated with putting the contract in good standing could have been more clearly delineated in the contract documentation. However, we believe we fully disclosed the complex changes to the contract, including the settlement of claims.
With regard to the issue of system safety, it is essential to recognize that system safety engineering for air traffic management systems such as CAATS is a new and evolving discipline. As the software functionality is developed, it will be tested against established safety criteria. The bottom line is that the system will not be implemented unless we are assured that it is safe.
In general, we agree with many of the conclusions reached in this audit but we strongly object to the allegations that the facts were misrepresented, and that all the blame seems to belong to the contractee and none to the contractor.
About the Audit
PART I - Key Issues for Success
ScopeThis year's audit examined four more major systems under development:
- the Canadian Forces Supply System Upgrade (CFSSU) at the Department of National Defence;
- the Tactical Command, Control and Communications System (TCCCS) also being developed by National Defence (including four separate but interdependent software projects that we reviewed in more detail);
- Transport Canada's Canadian Automated Air Traffic System (CAATS); and
- the Real Property Services (RPS) set of projects at Public Works and Government Services Canada, which consists of 11 separate but interdependent projects, including the Real Property Inventory system that we reviewed in more detail.
- the Income Security Program Redesign (ISPR) at Human Resources Development Canada;
- Transport Canada's Integrated Departmental Financial and Materiel Management System (IDFS);
- the Common Departmental Financial System (CDFS) at Public Works and Government Services Canada; and
- the Public Service Compensation System, also at Public Works and Government Services Canada.
ApproachOur audit approach was the same as last year's, which was described in Chapter 12 of the October 1995 Report. In examining the implementation of large-scale systems under development, we again conducted a risk assessment of the management of individual projects to determine what hinders or facilitates their successful completion and, as a corollary, to determine what lessons can be learned and shared among other projects.
PART II - CAATS: Some Matters of Concern
ScopeAs required by the Treasury Board, CAATS - as a Major Crown Project - has a Project Office known as the CAATS Project Office (CPO). The CPO is led by a Project Director from Transport Canada and assisted by staff from Public Works and Government Services Canada (PWGSC), who are respectively responsible for acting as the technical and contract authorities for the project. Our audit work focussed on the operations of the CPO and on how it was carrying out the key responsibilities assigned to it. As for all such projects, a central part of the role of the Project Office is to act as an interface between the prime contractor and the departmental client for the project, and our work inevitably touched at many points on the work of the prime contractor. However, we did not audit the prime contractor and we have no opinion to offer on its actions. Where our work touched on that of the prime contractor, we were only concerned with the appropriateness of the government's actions.
ObjectivesWe believe that it is important for Parliament to know both what is being delivered by CAATS and what the full cost will be. Accordingly, one objective was to attempt to determine and report what Transport Canada anticipates will ultimately be delivered as CAATS and what the cost of those deliverables will be.
The role of the CAATS Project Office is to have built, on behalf of the Air Navigation Service (ANS), an automated air traffic control system that meets the functional needs of the ANS and that is safe and efficient to use. We have identified three critical jobs that the Project Office must do to accomplish this. First, it must communicate the needs of the client to the prime contractor and assess the contractor's design proposals for their technical and operational suitability and compatibility with the rest of the ANS system. This task is sometimes referred to as design validation. Second, as the government's representative in dealing with the prime contractor, the CAATS Project Office must monitor the progress of the contractor to ensure that it is satisfactory and ensure that funds are paid to the contractor only when they are due under the contract. Finally, the Project Office must be able to assure the Air Navigation Service that the product delivered is safe, fit for use, and fully meets all of the specifications of the contract. This is sometimes called verification.
We examined these areas to be able to report on matters that, in view of the circumstances surrounding CAATS, are of significance to Parliament.
Quantitative information. The quantitative information in this chapter has been drawn from government documents. Historical financial information has been checked for reasonableness but not audited. Cost and future-oriented financial information has been checked against official sources.
Audit TeamBernard Battistin
For information, please contact Eric Anttila (Part I) or Hugh McRoberts (Part II), the responsible auditors.