This Web page has been archived on the Web.

2007 May Report of the Auditor General of Canada

Main Points

What we examined

Canada and the United States are partners in the air defence of North America under the North American Aerospace Defense (NORAD) agreement. Under this agreement, Canada and the US operate an integrated and interoperable air surveillance and control system to secure North American airspace. In 1997, the two countries agreed to upgrade and modernize this system. Originally, the Canadian government approved funding for about $93 million to modernize the Canadian portion of the NORAD system—the Canadian Air Defence Sector Air Operations Centre. Since the original approval for modernization, the project has undergone several changes and cost increases. We examined how the modernization has progressed, what it has cost National Defence, and what has been delivered.

Why it's important

Heightened security demands since September 11, 2001, have put emphasis on the need to modernize the NORAD surveillance and control systems. National Defence has a significant investment in the modernization project and needs to implement it in a way that is cost-effective and ensures that operational requirements are met. In May 2006, Canada and the United States renewed the NORAD agreement to continue operations in both countries.

What we found
  • Modernization was originally expected to cost National Defence $87 million, plus about $6 million for a definition phase, and was to be completed by 2001. However, we estimate that the Department has spent about $125 million so far, and bringing the modernization to completion is expected to cost another $18 million. Other related expenses of about $13 million will bring the expected total cost to about $156 million. The project took longer than planned to deliver and has cost more than initially expected because of problems with system development and project management. Recently, the Department successfully installed a new air surveillance and control system at its sector air operations centre, and it reports that the system is meeting initial operating requirements. However, anticipated savings of up to $16 million a year in personnel and operating costs have not materialized.
  • Neither National Defence nor the government made appropriate use of mechanisms available for managing large, high-risk projects like this one. Early signs that the project was in trouble and that costs were climbing did not prompt moves to strengthen its oversight. We asked why, for example, a decision was not made at the time to designate this a Major Crown Project. Treasury Board Secretariat officials initially declined to provide us access to the working papers on this issue because they viewed these as cabinet confidences. At the conclusion of the audit and after a search, Treasury Board Secretariat officials assured us that there were in fact no Treasury Board Secretariat working papers to provide to us.
  • Information provided to the government for increased project funding was not always timely or accurate, and did not always reflect the risks of the project or how its scope had changed.
  • National Defence entered into an agreement with the United States Department of Defense as an equal partner, during the development phase of the first effort to implement a new air surveillance and control system, and on that basis it agreed to pay 50 percent of common costs. Yet it could not provide us with evidence to demonstrate that it had in fact shared control of the project. Later, problems appeared in the Department's ability to control its own development of a second, replacement air surveillance and control system, which it was working on for the Canadian Air Defence Sector. Work on this second system was stopped in favour of a third system developed for NORAD, mainly with US Department of Defense input into the requirements. This air surveillance and control system has now been installed and is operating at the Canadian Air Defence Sector air operations centre. Future modernization and upgrade work is planned, but National Defence still has not clearly set out its own Statement of Operational Requirement and needs to review requirements before continuing.

National Defence has responded. The Department agrees with our recommendations and is proposing actions to address concerns.

Introduction

6.1 Canada shares the defence of North American airspace with the United States through the North American Aerospace Defense Agreement (NORAD). NORAD is the foundation for Canadian–US aerospace defence cooperation. Its primary mission includes surveillance, detection, monitoring, validation, and warning of air attacks against North America. Since it was formally established in 1958, NORAD has evolved to meet a growing variety of security threats. In 1991, the NORAD mission was expanded to include counter-narcotics air surveillance operations. After the September 11, 2001 terrorist attacks, NORAD has increased its role in the security of internal domestic airspace.

6.2 NORAD is a combined command established by mutual agreement between Canada and the United States. Based on available information, NORAD provides warning and assessment of air threats to each nation's responsible authorities.

Background

6.3 NORAD air defences are built around a network of radars connected to a command and control system for tracking targets and deploying fighter jets to intercept and, if necessary, engage threats. NORAD command and control is based at the Cheyenne Mountain Operations Center in Colorado Springs, Colorado. Surveillance and control of North American airspace is conducted from Canada, the continental United States, and Alaska. Exhibit 6.1 shows the NORAD regions and sectors. The Canadian NORAD Region (CANR) Headquarters is in Winnipeg, and the sector air operations centre is at Canadian Air Defence Sector (CADS) in North Bay, Ontario, which provides the tactical command and control for Canadian airspace.

6.4 A crucial element of NORAD's network of sector air operations centres is the computerized air surveillance and control system. It displays data gathered from sensors located across North America and, when they are deployed, from surveillance and reconnaissance aircraft. Each sector's air operations centre relies on the computerized system to assist in processing information, identifying targets, and guiding interception to the targets. Critical information generated in North Bay is forwarded to CANR Headquarters in Winnipeg, and to the Cheyenne Mountain Operations Center in Colorado Springs.

6.5 NORAD began operating the AN/FYQ-93 (Q-93) system in 1983, at which time it was expected to last 20 years. In the early 1990s, National Defence and the US Air Force began planning how to replace the Q-93 system at the end of its 20-year life span. As a partner in NORAD, Canada was involved in the decision to modernize and replace the system. However, initial plans and project scope for modernizing the NORAD computer system were not realized, and since that time, National Defence has been involved in developing two other replacement systems, one of which—the Battle Control System–Fixed or BCS-F—has recently become operational.

6.6 As part of the modernization, National Defence expected to generate savings. Some savings would be realized through a reduction in the number of personnel required to run the new air surveillance and control system. Further savings would be realized through a move from the existing underground complex in North Bay to a new, above-ground complex that would house the modernized system. Since 1995, costs to National Defence for modernizing the NORAD computer system have amounted to about $125 million in spending on system development and new facilities. The Department estimates that another $18 million in expenses will be necessary to bring the project to completion, plus about $13 million in related costs.

Focus of the audit

6.7 Our audit examined the National Defence components of the project to modernize NORAD's Q-93, the air surveillance and control system, to determine whether capability improvements had been developed and installed, and whether they met Canadian capability requirements. The audit examined project management, costs, timelines, and what was delivered. We also examined whether project activities complied with government policies, directives, and guidelines.

More details on the audit objective, scope, approach, and criteria are in About the Audit at the end of this chapter.

Observations and Recommendations

Approval to modernize the NORAD system

6.8 The North American Aerospace Defense (NORAD) Canadian Air Defence Sector Air Operations Centre has recently installed the newly developed Battle Control System–Fixed (BCS-F), an air surveillance and control system. The contractor for BCS-F began installation at North Bay in early 2006, and the Canadian Air Defence Sector Air Operations Centre switched from the old system (Q-93) to BCS-F operations in October 2006.

6.9 BCS-F represents the third attempt by National Defence to install a modernized air surveillance and control system in the sector air operations centre, a process that has spanned almost 12 years. Work to replace the old system began in 1995 when the government approved $6.4 million in funding for National Defence to participate, in cooperation with the United States, in the definition phase of the Region/Sector Air Operations Centre Modernization Project. In 1996, a Memorandum of Understanding was signed between National Defence and the United States Department of Defense to jointly modernize NORAD's air surveillance and control system. The aim was to develop a common NORAD system that could improve the coverage of North American airspace. Throughout the project, the two countries would fund an equal share of their common costs; they would also have a shared software maintenance strategy. As equal partners, Canada and the US were to share decision making, data rights, and industrial participation.

6.10 National Defence personnel and US Air Force personnel were assigned to the project management office to ensure that each partner's concerns and specific requirements were represented. A bi-national steering committee was set up to review progress and resolve any issues that might arise throughout the life of the project.

6.11 In February 1997, after completion of the project definition phase, the government approved the Department's request to spend $59.5 million on developing a new system. It was expected that the new system would be in place by September 1999 and that work to get it to full operational capability would continue until fall 2001. The Department supported its request for funding approval by showing that it expected to save about $16 million per year in operating costs. Savings were to begin around 2004, at which time the Department had planned to shut down its underground complex. The Department also anticipated a change in operations, with a reduction in personnel.

6.12 A few months later, the government gave approval, with conditions, to spend an additional $27.5 million on the following:

  • a deployable capability study;
  • an upgrade of the internal communications system; and
  • construction of an above-ground complex to house the new air surveillance and control system, allowing National Defence to close the underground complex.

6.13 In total, it was expected that the modernization would cost National Defence about $87 million, in addition to the $6.4 million spent on project definition, but that the possible savings would allow the Department to have its investment paid back within four to five years after project completion.

6.14 In March 1997, a contract to develop the proposed new air surveillance and control system for NORAD was awarded to a US company, and development work began. Although a system concept was defined at this point in time, system development components and costs had not yet been clearly set out.

There were early indications that the modernization project was in difficulty

6.15 In early 1997, National Defence told the government that the risk of not completing the project was low. However, by July 1998, it became apparent that costs were escalating. The bi-national steering committee was informed that projected costs for completing the modernization project were increasing. National Defence personnel, who were monitoring costs and overseeing progress, now considered it a high-risk project.

6.16 In February 1999, the Department requested an increase in funding to complete the modernization project. The Department reported that it was very confident that additional funds would enable the project to achieve operating capability, even though there were concerns at that time about the contractor's slow progress and escalating costs. National Defence received approval for an additional $36.7 million for this project, bringing the total expected cost for the modernization to about $130 million at that time.

6.17 We noted that the government had expected the Department to demonstrate progress on the modernization project and that funding would be approved in two phases. Funding for the second phase was conditional upon successfully achieving the first phase. However, because of delays and cost overruns, the government approved a change in project scope so that money set aside for phase two could be spent on continuing phase one in order to achieve core operational requirements, even though there was no evidence that the first phase was successfully achieving objectives. Hardware and software modules were not ready and significant development remained to be done.

6.18 In October 1999, despite assurances from the Department that increased funding would enable the modernization project to succeed, the contract for developing a new system was cancelled by the US Department of Defense. Concerns that progress was slow, along with cost projections that were more than triple the initial expected project completion costs, led the US Department of Defense to halt any further work.

National Defence had spent $65 million when the contract was terminated

6.19 National Defence has estimated that it had spent $65 million on this contract and on other related expenses by the time the project was terminated. Our review found that the Department had paid about $57 million for contractor development work, and that the Department's other project management costs brought the total to about $65 million. Despite its $65-million investment, the Department was left with the same system and operations that it had had before. The Department did receive about $1.5 million in hardware, as stipulated in the contract. Some of this hardware was used for training, and some remained in storage.

6.20 Department documents indicated that despite its 50–50 partnership in this project, and notwithstanding concerns about its difficulties, National Defence did not believe it could stop the process sooner or pursue an alternative course of action.

6.21 During this time, National Defence was also upgrading its internal communications system. The Department put that work on hold when the air surveillance and control system development contract was terminated. It had spent about $7.7 million on upgrades when the work was stopped. The Department was able to derive some benefit from this previous work when the internal communications systems work was later continued.

National Defence chose to unilaterally develop its own air surveillance and control system

6.22 After the contract was terminated in 1999, NORAD officials determined that the AN/FYQ-93 (Q-93) air surveillance and control system would not be replaced immediately, and that NORAD could continue to use it until 2009. Nevertheless, National Defence informed NORAD that it needed to save money, and that it therefore still planned to move its operations to a new above-ground complex, reducing personnel in the process. Although the Memorandum of Understanding with the US stated that both countries would jointly develop an interoperable air surveillance and control system, National Defence decided to develop and install its own system in the above-ground complex, as a short-term solution until a NORAD-wide system could replace it. In part, National Defence questioned the long-term viability of the Q-93 system at its sector air operations centre, and was concerned that a lack of spare parts might prevent Q-93 from remaining operational and reliable until 2009.

6.23 At the time of our audit, we found no issues related to spare parts shortages affecting Q-93 system operations. An analysis conducted by the Department in 2000 forecast that enough spare parts would be available to keep the system operational until 2010.

6.24 In September 2000, National Defence chose NATO's Multiple AEGIS Site Emulator (MASE) system. It planned to modify MASE to be compatible with the Q-93 system, which was still in use in the US. The Department began working on MASE system development and entered into agreements with other agencies to modify the system. However, it did not submit a request for increased spending authority to the government until June 2003, nor did it seek approval to change the scope of the project. We found that internal delays slowed the approval process. The Department initially expected to have a viable system ready by fall 2003, but by the time the submission was made to the government, the date had slipped to July 2004.

6.25 The Department supported its MASE-led approach with possible reductions of about 30 to 40 percent in operations and maintenance staff. The Department forecast that these reductions would lead to savings on the order of $10 to $12 million annually. It told the government that modifying the MASE system was a medium- to high-risk strategy for replacing the Q-93 system. In June 2003, the government gave approval to increase the funding to about $143 million (GST excluded). This was done in order to allow modification work to continue on the MASE system and to provide funding for other ongoing project requirements, such as the construction of the above-ground building. The government stipulated at this time that National Defence was to begin submitting annual reports on

  • progress,
  • changes in costs and schedules, and
  • updates on how risks were being managed.

6.26 Exhibit 6.2 shows the project costs that were expected as of June 2003.

The Department stopped working on the MASE system to join a new US-led option

6.27 Although it was working to modify the Multiple AEGIS Site Emulator (MASE) system, National Defence remained committed to a bi-national air surveillance and control system for NORAD. Shortly after receiving approval for its MASE-led solution, National Defence was approached by the US Air Force to participate in the development of the Battle Control System–Fixed (BCS-F) as the NORAD-wide system. In January 2004, senior management decided that the Department would join the BCS-F option, but would continue working on the MASE system as a fallback, should BCS-F prove unworkable. However, by mid-2005, it had become clear that problems with modifications to the MASE system would be difficult to overcome and that target dates would not be met; however, BCS-F was progressing. Therefore, National Defence stopped work on the MASE system to instead pursue BCS-F as the system for NORAD. It was expected that the BCS-F system would be installed, tested, certified, and operational at the sector air operations centre by early 2006.

6.28 The Department estimated that it would spend about $13 million to install and test the BCS-F system. It did not request an increase in spending authority from the government, because MASE system expenses were proving to be considerably lower than initially anticipated, and because the Department believed that, with the funds remaining, BCS-F system costs could be absorbed into the already approved $143 million.

6.29 The Canadian Sector Air Operations Centre began using the BCS-F system in October 2006, and is currently operating that system with the rest of NORAD.

Replacing the NORAD system plus related expenses brings total costs close to $156 million

6.30 At the time of our audit, we found that the Department had spent about $125 million of the approved $143 million for this project. Further expenses of about $18 million were expected for ongoing work to finalize this part of the modernization project. We found other costs that had been incurred by the Department as a direct result of this project, but that had been funded separately. These costs were incurred at several stages: during construction of the above-ground building at North Bay, at the time that the systems were moved from the underground facility to the above-ground building, and after the system was installed. Ensuring security of the facilities was expected to incur further expenses. However, no cost estimate was available to us at the time of this audit. We estimate that total costs for modernizing the NORAD system in Canada will amount to about $156 million.

Anticipated savings have not been realized

6.31 Despite expectations that there would be savings of as much as $16 million annually by 2004, at the time of this audit, the modernization project had not been able to demonstrate that any savings have been made or will be made. The Department based its projected savings on analyses that included closing the underground complex, building a small above-ground operations centre, and reducing personnel requirements. However, at the time of our audit in early 2007, we found that

  • the underground complex remained open for operational reasons, thus incurring the expenses necessary to keep it functional and safe;
  • the above-ground complex has grown from the small operations centre envisioned in 1998 to the current larger two-storey building; and
  • the Department has not yet been able to determine whether it will have any reductions in operational personnel under the new system.

6.32 Recommendation. National Defence should update its analysis of the NORAD system modernization project's expected savings, and should determine when such savings will be realized.

The Department's response. Agree. In response to a meeting of the Senior Review Board conducted on 11 December 2006, the business case for the Region/Sector Air Operations Centre Modernization Project was updated on 15 February 2007. As a significant portion of the original business case was predicated on the possible relocation of the operational elements, much of the infrastructure-related savings originally envisioned will not be realized. The current savings projections are approximately 18 percent of the original 1996 business case estimate; further studies on the actual savings are ongoing.

Although the underground complex has been declared surplus and is currently moving through the surplus property process, two additional challenges have impacted plans for closure of the underground complex. First, the underground complex has been identified as a heritage site. This will entail providing a minimal level of support to the underground complex until a permanent solution is found. Second, due to residual security concerns, a small number of systems have yet to be moved from the underground complex. This should be rectified over the next few months.

Project management and oversight

The Statement of Operational Requirement has not been updated since 1997

6.33 In 1997, National Defence drafted a Statement of Operational Requirement that outlined its needs for modernizing the North American Aerospace Defense (NORAD) system. A Statement of Operational Requirement is a mandatory document for all National Defence equipment projects. It defines expectations and guides decision making. National Defence policy requires that statements of operational requirement be updated as a project progresses and as requirements change. We expected that the Department would have reviewed the 1997 Statement of Operational Requirement as its policy requires, and before proceeding with other options.

6.34 However, we found that the Statement of Operational Requirement document has not been updated since 1997, even though

  • NORAD's responsibilities have changed since 1997;
  • advances in information technology since 1997 may have changed the risks or created new opportunities that are not addressed in the original Statement of Operational Requirement; and
  • the bi-national partner for this modernization, the US Air Force, has twice updated its version of a Statement of Operational Requirement.

6.35 We found that because National Defence had proceeded with the Multiple AEGIS Site Emulator (MASE) system development and with acquisition of the Battle Control System–Fixed (BCS-F) system without first re-examining its own Statement of Operational Requirement, it did not have an updated assessment of its needs. It was therefore unable to provide us with assurance that the systems would perform as needed, and was also not positioned to evaluate future system performance. As modernization and upgrades continue for the Canadian Air Defence Sector, an updated Statement of Operational Requirement is a key element for guiding the department's further analysis and decision making.

6.36 Recommendation. National Defence should ensure that any further modernization and upgrade of Canada's air surveillance and control system is supported by a current and clearly defined Statement of Operational Requirement.

The Department's response. Agree. Canada and the US have continued to evolve the system modernization requirements over time. With the full participation of Canada in earlier NORAD requirements documents, our requirements were incorporated into the Battle Control System Operational Requirements Document and the System Capabilities Document. In addition, as the Department determines the need for follow-on effort, all capability deficiencies will be validated as part of the Statement of Requirements for the Canadian Air Defence Sector Upgrade Project.

Some key project management components have not been completed

6.37 We examined the process followed by the Department to manage modification of the MASE system. The Department had entered into agreements with other agencies to modify the software, but did not identify start and end dates for the work. As a result, the work was not treated as a priority by the other agencies, despite National Defence timelines, and project completion was postponed.

6.38 We examined whether the Department had analysed options and assessed cost and feasibility. Government policy, along with generally accepted good management practices, require that both the expected benefits of any system and its estimated life cycle costs be analysed and included in the decision-making process. While few options may have existed for National Defence as a NORAD partner other than to accept the BCS-F system as its sector air operations centre system, we nevertheless expected the Department to have identified system benefits, as well as any associated risks, and to have assessed future costs. The Department could not provide us with evidence that it had done this analysis.

6.39 We expected that the Department would have a business continuity plan for its system operations, as required under the Government Security Policy. A business continuity plan is developed by assessing the threats and risks that could disrupt critical services. It identifies what needs to be done to protect against system failures. Although we did observe that some steps had been taken by the Department to protect against events such as a major power failure, we also expected to see a planned response, on the part of the Department, to a major disruption. Such a planned response would identify which systems take priority and the need for system back-up and system redundancy. National Defence was unable to provide us with a business continuity plan that addressed these key elements.

6.40 Recommendation. The Department should conduct a threat and risk assessment for the air surveillance and control system, to determine the extent to which the Department will provide system back-up and redundancy, and to develop a business continuity plan as necessary.

The Department's response. Agree. The Department agrees that threat risk assessments and business continuity plans are important and necessary. For the Region/Sector Air Operations Centre project, a threat risk assessment was prepared for the first system and revised for the MASE system. For the current BCS-F system, a threat risk assessment is being performed as part of the Department of Defence Information Technology Security Certification and Accreditation Process. Within this certification and accreditation process, back-ups, redundancy, and continuity will all be addressed. The requirement for site redundancy will be further examined under the Canadian Air Defence Sector Upgrade Project.

National Defence did not inform the government of project scope changes in a timely fashion

6.41 Government policy requires departments to seek authority from government before changes are made to a project in areas that affect the project's scope, such as key deliverables, risk management, timing, contracting strategy, special requirements, and project management. We found that over the course of the modernization project, the Department did not officially inform the government in a timely way of significant changes in some of these areas.

6.42 We found that although government approval of the project called for a joint effort to share in the development, acquisition, and software maintenance of a new air surveillance and control computer system common to Canada and the US, the Department acted unilaterally to modify the MASE system for its own use. The decision by National Defence to change its key deliverable and adapt the MASE system was made in 2000. However, the Department did not officially inform the government of this decision until 2003. The Department has informed us that government officials were "informally" notified in October 2000 of this change.

6.43 We examined whether this decision resulted in a change to the deliverable, the timing, and the contract strategy. Government officials informed us that the decision was considered to change only the delivery method. They stated that since the ultimate goal was to replace the Q-93 system, the decision to adopt MASE did not represent a change in project scope that would require government approval. It is our view that National Defence did, in fact, change the scope of the project, from developing a bi-national system in common with the US, to implementing a short-term solution for Canada alone.

6.44 Despite the fact that, according to the Department, the project's risks were changing, we were not provided with evidence that this was communicated to the government in a timely way.

6.45 The Department had estimated that implementing its MASE system solution would cost an additional $9.5 million. Officials maintained that the Department retained the authority to proceed with the implementation, because costs were within the already approved spending authority (that is, within the amount remaining after approximately $65 million had been spent on the cancelled contract), and because the Department's approach still fell within the general scope of modernizing the NORAD system. Ultimately, expenses were considerably lower than planned—about $440,000—due to problems delivering the MASE modifications.

6.46 In the Memorandum of Understanding signed with the US, Canada was to be an equal partner in the joint development of an interoperable air surveillance and control system, sharing the common costs, the decision making, the data rights, and the participation of industry. When the Department joined in the acquisition of the BCS-F system, it did not inform the government of this change in scope from being a partner to becoming a customer buying a product developed mainly from US Air Force stated operational requirements.

Oversight would have increased by designating the modernization project a Major Crown Project

6.47 Given the cost escalations, delays, and risks, we found that sufficient warnings were available to the Department and the government to consider designating the modernization project a Major Crown Project. Government policy on Management of Major Crown Projects identifies costs, risks, and a department's capacity to manage a project as key factors in determining whether it should be designated a Major Crown Project. Such a designation would have required National Defence to conform to a more defined and rigorous structure for reporting to the government. In 1998, National Defence officials asked Treasury Board Secretariat officials whether the modernization should be designated a Major Crown Project. The modernization was not designated a Major Crown Project, but Treasury Board Secretariat officials initially declined to provide us with their working papers setting out the reasons for this decision, because they viewed these as cabinet confidences of a type that were excluded from our right to access under a 1985 Order in Council. At the conclusion of our audit, Treasury Board Secretariat officials assured us, following a search, that working papers on this topic did not in fact exist. Accordingly, we are unable to assess the rigour and quality of the government's oversight.

6.48 National Defence management policy states that increased senior management scrutiny is required for Major Crown Projects. Although this project was under scrutiny by a senior review board at the Department, we found that the board met only once a year, which is the minimum requirement under the Department's defence management system. Had the modernization been designated a Major Crown Project, the project would have been required to report to a Special Project Advisory Committee. As well, the Statement of Operational Requirement would have been passed to the Joint Capability Requirement Board (made up of the chiefs of the Army, Navy, and Air Force staff, along with the Assistant Deputy Ministers) for endorsement.

The Department did not take advantage of already existing controls to provide better oversight of this project

6.49 The Department's Assistant Deputy Minister (Information Management) is the National Defence authority for information technology projects and initiatives. Each National Defence service (Army, Navy, Air Force) is responsible for development of projects categorized as weapons system projects, and each may ask the Assistant Deputy Minister (Information Management) for advice if need be. Because National Defence categorized the modernization project as a weapons system for sector air operations, the project's development remained under the control of the Air Force.

6.50 Although it is clear that the Air Force is best positioned to state what it needs for NORAD operations, the Assistant Deputy Minister (Information Management) is concerned with the development of all information technology systems in the Department and oversees how they integrate with each other. The Assistant Deputy Minister (Information Management) recommended that the C4ISR Oversight Committee monitor the modernization project, because the new air surveillance and control system would have to integrate with other defence systems, and recommended that the modernization project not be managed independently. Although senior management noted these concerns, the Department did not submit the project to the Oversight Committee as recommended.

Construction of the above-ground complex

Costs and design have grown considerably since first approved

6.51 National Defence originally planned to build an above-ground complex that was much smaller than the current building. Initially, in spring 1998, the Department proposed to the Minister that it would build a small (4,500 square metre) operations centre for about $6 million. Shortly thereafter, however, the Department recognized that it had underestimated space requirements in its initial business case. The Department then made revisions that expanded the size of the proposed building to about 7,000 square metres, a change that would raise the cost to about $9 million. Later, in October 1998, as part of an effort to reduce infrastructure at the Canadian Forces Base at North Bay, plans for the building changed. Originally conceived as a small operations centre, it would now be a larger (9,560 square metre) two-storey building large enough to house the majority of operations on the base. Costs for the new plans were estimated at $12 million. This estimate was later revised to $18 million. When the contract for the first air surveillance and control system development was cancelled in 1999, National Defence put the building construction on hold.

6.52 In 2003, after the Department decided that it would install a modified version of the Multiple AEGIS Site Emulator (MASE) system as a temporary NORAD system, the Department started construction on the 9,560-square-metre complex. By this time, construction costs had increased to about $23.9 million, and we estimate that at the time of this audit the costs were close to $28 million. We also found other costs that resulted from the new construction, but that were not included in the Department's building costs. Examples of such expenses are costs for moving systems into the new building and costs related to site security. We estimate that overall, National Defence will spend about $38 million on expenses related to the above-ground complex, for constructing it, moving into it, and addressing some of the facility's other requirements.

6.53 The Department supported moving into an above-ground building by stating that doing so would generate savings. The Department cited these potential savings as part of the rationale for pursuing development of the MASE system. However, we were not provided with evidence showing how savings were calculated or how costs would be reduced despite the growth in size of the building. (See photograph)

Use of the building is limited by design and construction irregularities

6.54 We found that planning and approval for the construction of the building had proceeded without a clear understanding of the operational requirements. Originally, the building was designed to accommodate the replacement for the old Q-93 system. Later, it was decided that the building would also house most of the base operations. As built at present, the above-ground complex cannot accommodate base operations as originally planned. There are now concerns about the extent to which the base can decommission other buildings and reduce its infrastructure.

6.55 We found that the Department had not completed a review of the building security requirements prior to construction. National Defence requires that a security review checklist be completed for new buildings, to ensure that security concerns are identified and addressed. Department officials told us that due to time and budget constraints, this step was not taken. Several security concerns did arise during construction that have led to questions about the building and the subsequent feasibility of operating in it. These questions specifically concern

  • the blueprints and purpose of the building, which were placed in the public domain when they were made available to interested contractors;
  • the physical control of the building and access to the site during construction, which were limited; and
  • the workers on site, none of whom were security cleared to work there.

6.56 As a result, the Department is currently examining any extra steps it must take to ensure that the building can be used for the purposes intended, and to determine the necessary costs. At the time of this audit, the Department had not yet decided on its course of action, and hence was not able to provide us with this assessment.

6.57 As a result of concerns over the facilities in the above-ground building, the Department has had to maintain some operations in the underground complex longer than expected. At the time of this audit, despite plans to save costs by closing the underground complex and operating out of one facility above ground, National Defence was still operating both facilities.

Conclusion

6.58 Because of the way this modernization project was structured and managed, it has cost National Defence significantly more than originally planned. Decisions were made without complete information or an understanding of real savings. We found that projected savings are not being realized, and that costs for the construction of the sector air operations centre, and for developing and installing a new air surveillance and control system in the Canadian Air Defence Sector, are almost double the original estimate.

6.59 We examined whether government policies, directives, and guidelines were followed by the Department as it replaced the old AN/FYQ-93 (Q-93) system. We found that the Department did not report on project difficulties or changes in scope in a timely way. This, in our opinion, contributed to the modernization project having less oversight than would be expected for a project entailing such cost, risk, and complexity. We found as well that some information presented to decision makers about the progress of the modernization project did not adequately portray the risks. There were early indicators that the project was experiencing significant cost overruns, and that it was failing to deliver the intended product—an operational air surveillance and control system. Sufficient indicators were evident to warrant more oversight by the Department and by the government. There were also several indicators that this project would have benefited from being designated a Major Crown Project.

6.60 We also examined whether the new system met National Defence requirements for air surveillance, but we found that the Statement of Operational Requirement had not been updated to reflect a review of the current environment or of the options. While we recognize that North American Aerospace Defense (NORAD) is a bi-national arrangement that requires compatibility of systems and operations, we nevertheless expected that National Defence would have demonstrated that its requirements were being met, by clearly identifying what those requirements were and how the Battle Control System–Fixed (BCS-F) system responds to them.

6.61 We examined whether the construction of the above-ground complex was supported by a sound rationale and business case. We found that the intended purpose for the building changed over the course of the project. It was originally planned to be a small operations centre, but it grew to be a larger facility. Part of the rationale for this construction was to save costs, but we found that the Department could not provide us with evidence to support how it had determined that there would be savings, how the changes to the building continued to support operational savings, or when savings would be realized.

6.62 We are concerned that the Department is proceeding with the modernization of its air surveillance and control system without having clearly reviewed the Statement of Operational Requirement needed to help define the necessary characteristics of the system, and to guide the evaluation and performance standards of the system. The Department has not completed a business analysis that scrutinizes all costs as it moves forward. Nor has it prepared a business continuity plan. We are concerned by the fact that a statement, analysis, and plan are not yet in place, because the Department is forecasting further expenditures. These expenditures will be incurred for future components of the new air surveillance system. The Department has identified a project to examine this—the Canadian Air Defence Sector Upgrade Project.

About the Audit

Objectives

The objective of this audit was to assess National Defence's development, implementation, and costs for modernizing the air surveillance and control system operated by the sector air operations centre under the North American Aerospace Defense(NORAD) agreement. Specifically, we assessed to what extent National Defence

  • complied with government policies, directives, and guidelines in managing project costs, timelines, and delivery;
  • complied with government's policy and with its own policy on risk management in developing systems to replace the old AN/FYQ-93 (Q-93) system;
  • had a sound rationale for building an above-ground complex for housing a new air surveillance and control system to replace Q-93; and
  • analyzed options to ensure that the proposed replacements for the Q-93 system do in fact meet the NORAD and Canadian NORAD Region (CANR) operational requirements for the defence of North America.

Scope and approach

We conducted our audit at National Defence Headquarters, and we conducted field visits to the Cheyenne Mountain Operations Center (CMOC) in Colorado Springs, the CANR Headquarters in Winnipeg, and the Canadian Sector Air Operations Centre (SAOC) located at 22 Wing in North Bay.

The audit team interviewed personnel responsible for the project at National Defence Headquarters. The team also interviewed personnel in the following groups: Assistant Deputy Minister (Information Management), Assistant Deputy Minister (Materiel), Assistant Deputy Minister (Policy), Assistant Deputy Minister (Infrastructure and Environment), Assistant Deputy Minister (Finance and Corporate Services), and Chief of the Air Staff. At each of these organizations, the audit team examined relevant files and documents. We interviewed personnel at the Cheyenne Mountain Operations Center in Colorado Springs, at the Canadian NORAD Region headquarters, and at the Sector Air Operations Centre.

Expenditures were tracked using data from the Financial Management and Accounting System and from department reports.

We attempted to reach a clear understanding of the Regional/Sector Air Operations Centre (R/SAOC) Modernization Project. We examined project management, costs, and timelines.

All government funding approvals exclude the Goods and Services Tax (GST).

Criteria

We expected that National Defence would do the following:

  • Comply with government policies, directives, and guidelines for contracting;
  • Have a completed option analysis and approved plans for addressing capability deficiencies as a way of meeting the requirements of the modernization project and the NORAD mission;
  • Demonstrate the savings realized by moving to the above-ground complex; and
  • Have analyses that support the decision to move the NORAD installation into the above-ground complex.

Audit work completed

Audit work for this chapter was substantially completed in January 2007.

Audit team

Assistant Auditor General: Hugh McRoberts
Principal: Wendy Loschiuk
Lead Director: Pierre Hamel
Director: Tony Brigandi
Director: Daniel Thompson
Audit Project Leader: Craig Millar

For information, please contact Communications at 613-995-3708 or
1-888-761-5953 (toll-free).


Definitions:

Statement of Operational Requirement—A mandatory National Defence project document that describes the characteristics of the operational requirements to be used by technical and procurement staff. It contains critical performance criteria for evaluating technical options and system performance. (Return)

A project may be designated a Major Crown Project when the estimated cost of the project will exceed $100 million, and when it is assessed as being high risk. Projects that have a total expected cost of less than $100 million, but that are assessed as high risk, may be designated as Major Crown Projects by the government. (Return)

C4ISR stands for command and control, communications, computing, intelligence, surveillance, and reconnaissance. National Defence created the C4ISR Oversight Committee in 1999 to respond to the need for more coordination of requirements. The Committee provides a strategic perspective and leadership on all C4ISR related matters. (Return)