This Web page has been archived on the Web.

2002 September Status Report of the Auditor General of Canada

Assistant Auditor General: Hugh McRoberts
Principal: Wendy Loschiuk

Key Message

4.1 National Defence has entered into a $2.8-billion contract to train pilots over 20 years as part of the NATO Flying Training in Canada program. During the first two years of program implementation, National Defence used only about 41 percent of the training capacity that it paid for. Because of the restrictive terms of the contract and the problems encountered during start up, about $65 million in training costs were paid for training that was not obtained. The Department has informed us that there are contractual opportunities to recoup some of this expense. These opportunities need to be pursued now.

The Department has responded. National Defence agrees with most of our conclusions but feels that because of the unique nature of this program, many of the issues we raise were start-up challenges that were to be expected. Its responses, presented at the end of the chapter, elaborate on the actions it will take to address our concerns and recommendations.

Introduction

4.5 This follow-up on the NATO Flying Training in Canada program is an audit of National Defence's implementation and management of the $2.8-billion contract for a 20-year term for military pilot training. Our 1999 audit reviewed the contracting process up to the signing of the contract in May 1998. Pilot training was planned to start in February 2000. This follow-up reports on the results achieved during the first two years after planned start-up.

4.6 In 1999, we reported that the contract was awarded without competition, that the profit mark-up was not consistent with current Government of Canada guidelines or supported by adequate analysis of contractor risk, and that the chosen financial arrangements increases some risks (Appendix A). Departmental officials responded that a major advantage of moving to a training system supported by a contractor was the shifting of risk to the contractor. However, they had not quantified this risk before awarding the contract.

4.7 In our December 2001 follow-up report on National Defence—Alternative Service Delivery we reported that National Defence had changed its approach to alternative service delivery projects and that employee training was now available. We mentioned that National Defence did not address alternative service delivery projects in its Report on Plans and Priorities and that the government's work on procurement reform had failed to address our original concerns about large service contracts for multi-year terms.

4.8 This follow-up focusses on the NATO Flying Training in Canada program. We assessed whether National Defence had used the levels of service paid for from February 2000 to December 2001 and whether it had received value for money. As well, we followed up on our 1999 recommendations to the Treasury Board Secretariat and Public Works and Government Services Canada's continuing work on procurement reform.

Focus of the follow-up

4.9 The objectives of this follow-up were to report to Parliament on the implementation and management of the NATO Flying Training in Canada contract from February 2000 to December 2001. During the contracting process, the Department determined how many pilots would be trained by the system for the next 20 years. We did not audit the planning process or the numbers that were agreed to by the Department. Rather, our audit examined whether National Defence used the services that it paid for and whether it obtained value for money.

4.10 We did not audit the activities of the prime contractor. Our audit was limited to National Defence, Public Works and Government Services Canada, and the Treasury Board Secretariat. We reported facts regarding contract delivery, but limited our comments and observations to whether the actions of the government were appropriate. Further details can be found in About the Follow-Up at the end of the chapter.

Observations

The NATO Flying Training in Canada program

4.11 The program provides flying training to military student pilots to fly the Canadian Forces' helicopters, multi-engine aircraft and the CF-18 Hornet jet fighters. The program trains pilots who have already passed primary flying training by offering four additional phases of training—basic Phase IIA, basic Phase IIB, advanced Phase III, and fighter lead-in training Phase IV (Appendix B). The first three training phases are taught at Moose Jaw, Saskatchewan and fighter lead-in training is taught at Cold Lake, Alberta. Those who will fly the CF-18 Hornet take all four phases of the program. The pilots who will fly helicopters and multi-engine aircraft go on to a separate training program in Portage la Prairie after they graduate from basic flying training Phase IIA.

4.12 The Harvard II aircraft is used for basic flying training. The Hawk aircraft is used for advanced Phase III and fighter lead-in training Phase IV. (See photographs)

4.13 As the prime contractor, Bombardier Inc. co-ordinated start-up activities with several international firms. The prime contractor is responsible for providing and supporting aircraft and simulators. They also provide ground school training for phases IIA, IIB, and III in Moose Jaw, classroom training systems, and maintenance services. The prime contractor provides the aircraft by leasing them for the life of the program from a not-for-profit company created at the start of the NATO Flying Training in Canada program.

4.14 National Defence and the prime contractor share some responsibilities: design of ground school training, scheduling of student facilities and aircraft, and the provision of operational and base support services.

4.15 National Defence is responsible for overall program management including operational control of training, providing instructors for flight training, providing infrastructure and military flying training areas at Moose Jaw and Cold Lake, and for the design, development, and conduct of flying training. It is also responsible for ground school training for the fighter lead-in phase, quality assurance, and student administration.

4.16 National Defence selected this as an alternative to training its pilots on the aging Tutor aircraft. It was one of three options for training pilots that the Department studied in the early 1990s (Appendix C). The other two options were to either purchase fighter pilot training off-shore or to upgrade the Tutors and extend the original program.

International participation is essential

4.17 Government approval of the program depended on the participation of other nations to help offset the costs of new aircraft and simulators. Without international participants, National Defence would be responsible for all the costs of the program. The government directed that the established fees charged for foreign participants would be on a full cost recoverable basis. As of December 2001, Canada and the air forces of four other countries were participating in the program—Denmark, the UK, Italy, and Singapore. The Hungarian Air Force has also signed on to the program and will start training pilots in 2002.

4.18 National Defence and the participating countries pay for a specified number of training slots each year. For example, National Defence has contracted for 141.6 Phase IIA basic flying training slots each year. In addition, other nations have purchased training slots based on the number of years they have committed to the program. The levels of service the prime contractor is required to provide are spelled out in the various contracts and agreements under the program (Exhibit 4.1).

National Defence states that the contractor is not providing the required levels of service in basic flying training Phase II

4.19 During the first two years of the program (from February 2000 to December 2001), National Defence states that it was only able to use 105.4 of the 265.6 Phase IIA basic flying training slots it purchased, or 40 percent. It was expected that during this time National Defence would have graduated about 216 student pilots from Phase IIA (assuming 15 percent attrition). Due to delays and cancellations, the expected graduation numbers were revised to 160 Canadian pilots but by December 2001 only 61 Canadian student pilots graduated from this phase of the program. Many that should have finished in 2001 will now finish in 2002.

4.20 Prior to January 2000, the Canadian Forces trained its own pilots on the Tutor aircraft. Once the NATO Flying Training in Canada contract was signed, the Department decided to shut down its existing program prior to the start-up of the NATO Flying Training in Canada program. The Tutor training system had produced 64 basic flying training graduates in one year (1999) from an intake of 78 students. By December 2001, the NATO Flying Training in Canada program had not reached this level of performance after two years of operation and when National Defence is short of pilots for its helicopters, multi-engine, and fighter aircraft. (See photograph)

4.21 The first four courses of the NATO Flying Training in Canada program were originally scheduled to start between February and May 2000, but were cancelled because of late delivery of the aircraft and simulators. Problems in acquiring the Harvard II aircraft and technical data from the United States due to International Trade in Arms Regulations (ITAR) issues, aircraft engine problems, and certification problems with the aircraft and aircraft maintenance operation meant that the program did not start training students until June 2000.

4.22 In August 2000, two months after basic flying training had started, a problem with the engine oil cooler caused National Defence to ground the Harvard II aircraft because of concerns about pilot safety. Training was delayed for three more months until the problem was fixed and one course was cancelled.

4.23 In 2001, three more courses were cancelled. Overall 8 of the 17 courses planned for 2000-01 were cancelled and only 3 of the 9 courses that did take place were fully loaded with the contracted number of students (16). Furthermore, courses which should have been completed in 5 to 7 months took on average about 9 to 10 months to complete (Exhibit 4.2).

4.24 The student intake for 2002 has been reduced to 105 students overall, about 41 below the contracted course-load level, in order to match students and available training capacity.

The program is not providing enough training flights per day to meet requirements

4.25 The contract states that on average 81 sorties, or training flights, could be scheduled each day by National Defence so that pilots can complete their flying training on schedule and move on to the next phase as planned. The prime contractor is responsible to provide and maintain enough aircraft and simulators to meet the specified sortie rates. Departmental analysis showed that 17 serviceable aircraft flying five flights per day must be available to generate this rate of flying activity. However, on average only 14.5 serviceable aircraft have been available during the first two years of the program to maintain the sortie rate.

4.26 Throughout 2001 National Defence did not receive the contracted sorties to meet its demand. Since the program has been in operation, National Defence has only scheduled an average of 62 sorties per day and has only flown an average of 42 sorties per day. Although there has been some improvement, the actual number of sorties flown per day is still far below anticipated levels and therefore the backlog continues to grow (Exhibit 4.3).

4.27 According to the Department, aircraft availability has been an ongoing problem and has resulted in training delays and cancellations. Departmental documents indicate that most of the problems with the program over the first two years of operation relate to Harvard II aircraft availability. The Department has stated that the focus is on problem-solving rather than attributing blame and has worked closely with the prime contractor to get the most training possible under current circumstances.

National Defence needed more instructors than originally planned

4.28 In 2001, studies conducted by the flight schools in Moose Jaw and Cold Lake found that the number of instructor pilots provided by National Defence was enough to meet the initial planned staffing levels, but this initial number was not adequate to meet the training demand. The flight school in Moose Jaw has recommended increasing the number of instructors to 80. In early 2002, the Canadian Forces took action to remedy the shortage by increasing the number of instructor pilots at Moose Jaw from 60 to 71 but a decision of the final level of staffing remains outstanding.

Problems and delays in the basic flying phase of the training program have caused shortfalls and cancellations in subsequent training phases

4.29 Overall, the start-up challenges in the basic flying training phases have meant that National Defence has been unable to use the full capacity in the advanced and fighter lead-in training phases.

4.30 Advanced training Phase III. The Department identified that advanced flying training would be underused throughout 2001 since delays in basic flying training meant that no students were ready to start the advanced work.

4.31 By December 2001, National Defence had used only 8 of the 29 basic flying training Phase IIB positions it purchased and only 13 of 36 advanced Phase III slots. In 2001, Canada sold 5 Phase III slots that they could not use to the United Kingdom and Italy. On average advanced courses have been one month longer than planned and one course was cancelled in 2001.

4.32 Fighter lead-in training Phase IV. The first course was cancelled because of delays in delivering the Hawk aircraft and simulator, and preparing the course material. However, the next three courses ran with more students than originally planned to make up for the cancelled course. These students had not completed Phase IIA or Phase IIB of this training program but were either previously qualified Canadian pilots being retrained to fly the CF-18 Hornet or international students. (See photograph)

4.33 One critical shortfall that was identified by National Defence in March 2001 was the late delivery of the centre line fuel tank for the Hawk aircraft, an external fuel tank which increases the range of the aircraft and is required for fighter lead-in training. The fuel tanks were late being built and once they were received, the Department needed eight months to complete certification test flights. This meant that the course syllabus had to be adjusted with shorter training flights. In some cases this lack of equipment extended the course by as much as two to three weeks beyond the planned graduation date. At the end of March 2002, the centre line tank was cleared for use and has now been installed on the fleet.

4.34 By December 2001, National Defence had sold off 4 Phase IV slots to Denmark and Singapore, reducing its capacity to 20.2. National Defence used only 5 of its remaining 20.2 slots in the fighter lead-in training phase and had sold off 5 additional slots to the United Kingdom and Italy. The Canadian pilots that did graduate from Phase IV had actually received their basic training on the Tutor aircraft. As the program grows, by 2004 National Defence will have 29 Phase IV training slots, but projections to 2004 indicate that National Defence is only going to use 9 slots per year and continue to sell 6 which would leave 14 slots unused. The first Canadian pilot to be trained from Phase IIA to Phase IV through the NATO Flying Training in Canada program is scheduled to graduate in July 2002.

More pilots were awaiting training and were waiting longer

4.35 The number of Canadian pilots awaiting training and the wait time for training has increased. In September 2001, 161 students were awaiting basic flying training. Behind this group were another 109 candidates completing basic officer training, second-language training, or initial degree requirements whose names would be added to the pilot training waiting list. On average, students were waiting 18 to 22 months before starting pilot training.

4.36 The Department has indicated that in July 2002, the number of Canadian pilots awaiting training had decreased to 131 and the average waiting time had fallen to 14 months. They expect this trend to continue.

4.37 National Defence has stated that the operational impact has been manageable since the system was producing as many pilots as could be accepted into the helicopter and the multi-engine aircraft school or the various operational training units. Nevertheless, departmental documents show the situation was difficult for the students facing delays and frustrating for persons involved in the program. (See photographs)

National Defence and PWGSC have been working on fixing problems for over two years but cannot say when problems will be resolved

4.38 National Defence and Public Works and Government Services Canada (PWGSC) have identified a number of problems with implementing the contract. They are working with the prime contractor to fix them. For example, a steering committee, co-chaired by National Defence and the prime contractor was set up to address issues like the low sortie rates. However, many of the issues identified early in the program have yet to be resolved.

4.39 The contract gives National Defence the right to sell some of its unused training capacity to other countries under a memorandum of understanding. Fees from these sales help offset some of the fixed cost to the Department. However, the contract limits the number of slots that can be sold. National Defence can only sell off 11 basic, 15 advanced and 15 fighter lead-in slots per year. The prime contractor nevertheless has waived this contractual limit during the time frame of the follow-up in order to allow National Defence to sell 20 slots to Hungary that cannot be used because of production delays in the basic flying phase. This enables the Department to recoup some of its costs.

National Defence has paid about $65 million for unused training capacity

4.40 National Defence has not used the full amount of training purchased in any phase of the program. From the start of the program to December 2001, National Defence only used 41 percent of the planned student slots but still paid most of the anticipated cost (Exhibit 4.4).

4.41 National Defence has paid a total of $179.5 million, which includes costs for both students and instructors, of the anticipated $206.8 million training costs.

4.42 National Defence calculated a cost per student for the foreign participants based on the target capacity of the program. If the program had been operating at its total capacity during the first two years, National Defence would be paying the same per student as other nations. During the first two years of the program, we estimate that National Defence paid about $65 million for training that it did not use. International participants are using all of the training slots they bought while Canada is not. This means that National Defence is paying the financial impact of inefficiencies in the program and the lack of productivity in Phase II (Exhibit 4.5).

4.43 In some cases, the minimum amount of equipment which the program needs has residual capacity that can train more students as the program expands, for example; the simulators for the Hawk aircraft. Because foreign fees were based on participants paying a fair share of equipment total capacity, until the program expands Canada is absorbing this as a business risk which the Department hopes to recover in the future. As well, Canada pre-pays the fees in advance of the training program reaching its full capacity. As of December 2001, the value of these two factors was estimated to be about $25 million (Exhibit 4.5). However, the department feels that as the program reaches full capacity, much of this value will be recouped. The final result will depend on how much the program finally expands.

4.44 In our 1999 report on Alternative Service Delivery, we found that inflexible contract arrangements resulted in payments for unused training capacity at the Meaford Area Training Centre and the Canadian Aviation Training Centre. For example, the Meaford Area Training Centre operated under a $40 million, five-year, fixed price contract and had used only 43 percent of its capacity. The Canadian Aviation Training Centre flying training program at Portage La Prairie, operating under a $165 million contract, was consistently underused during the first six years of operation.

About 79 percent of the contract fees are paid regardless of how many pilots are trained

4.45 National Defence pays tuition fees to the prime contractor for the services detailed in the contract. Tuition fees are based on a schedule of payments in the contract and are composed of fixed and variable fees. About 79 percent of the total $2.8 billion cost of the program covers overhead and equipment costs and is fixed. This amount must be paid no matter how many pilots are trained. The remaining 21 percent of the fees are variable and are based on the actual number of pilots trained and their use of the aircraft and supplies. The contract also calls for National Defence to pay transition fees to the prime contractor for work done before the program was implemented (Exhibits 4.6 and 4.7).

Firm fixed payments are unconditional and irrevocable

4.46 A non-share, not for profit corporation was established to purchase equipment for the NATO Flying Training in Canada program. It issued a $720 million bond to obtain the capital required to buy the aircraft and simulators. (See photograph)

4.47 National Defence must make 40 semi-annual payments of $31.4 million over 20 years (December 1999-June 2019) which go to the not-for-profit corporation to cover the cost of the principal and interest on these bonds and its operating costs. These payments are unconditional and irrevocable regardless of whether National Defence has the use of the aircraft. National Defence must pay even if the prime contractor is placed in default and the contract is terminated. However, should this occur, National Defence would continue to have access to the aircraft.

4.48 In December 1999, National Defence paid $31.4 million but the aircraft were late in being delivered and, therefore, could not be used for training. National Defence cannot recover this payment despite the late delivery of the aircraft. There are provisions in the contract to ensure that National Defence retains access to the aircraft and simulators if the program is extended to provide the missed training. However, it is not clear at this point whether National Defence will have to pay some of the expenses necessary to keep the aircraft or whether this is the responsibility of the prime contractor. At the end of the program, the licence agreement states that National Defence can purchase the aircraft at fair market value.

4.49 In addition, National Defence paid the prime contractor $15 million in firm and variable fees to meet its contractual obligations even though the first four courses had to be cancelled. The contract required that National Defence pay these fees as per the agreed schedule. The prime contractor has returned $2.5 million of the $15 million advance payment, the pre-paid variable components of the fees, since no students were trained.

4.50 The contract does state that training not provided can be made available at some future date. The contractor is willing to provide the training but, until it does, it has been paid $12.5 million for training that has not yet been conducted. As well, problems with program delivery must be resolved first in order to clarify how the provisions in the contract will apply.

National Defence has not yet determined whether it has under or over paid the variable fees to the prime contractor

4.51 National Defence makes semi-annual prepayments for variable and reimbursable fees based on an estimated amount of consumption. These fees are to be adjusted at regular six month intervals based on actual use. To date, this has not been completed. National Defence and the prime contractor have yet to finalize the number of flying hours used.

4.52 National Defence has paid $44.2 million to the prime contractor in variable and reimbursable fees of which almost $11 million is for international participants. It has not used all of the training slots and is entitled to a refund for these advanced payments. National Defence is still reviewing a proposal from the prime contractor for a refund of $6.6 million for the training period from 1 January 2001 to 30 June 2001. This number remains in dispute.

4.53 The contract states that the prime contractor will provide an information management system, but this has not yet been put in place. As a result there have been delays in finalizing the reconciliation, since all records must be reviewed and accounted for manually. In September 2001, the contractor and program participants agreed to change the method of reconciliation. National Defence personnel on site in Moose Jaw who were expected to provide the input for the reconciliation were not trained in the process. Until the reconciliation of flying hours has been finalized, National Defence cannot determine whether it has under or over paid the prime contractor for these fee elements.

4.54 The agreements with the other countries involved with the NATO Flying Training in Canada program require a similar type of reconciliation at regular six month intervals. Since National Defence has not reconciled with the prime contractor, it cannot reconcile variable fees with the other nations. The result is that National Defence does not know if it has over or under charged other nations for the training that they have received.

Payments are not tied to performance

4.55 National Defence's payments under the contract are based on schedules rather than performance milestones. In the event that the prime contractor is non-compliant with the terms of the contract or where the prime contractor does not provide the specified levels of service, there are no financial incentives that can be used by National Defence. Conversely, there are no performance incentives to reward or encourage the prime contractor for exceptional service.

4.56 The only remedy available to National Defence under the contract would be to place the prime contractor in default of the contract. Short of putting the contractor in default, National Defence may not withhold payments regardless of the quality or quantity of the service provided. A National Defence study of the Alternative Service Delivery program (May 2001) concluded that for the NATO Flying Training in Canada program "the only real financial incentive [to ensure that the prime contractor provides full delivery of services] is the threat of termination of the contract."

4.57 Although the NATO Flying Training in Canada program has been providing training to students since June 2000, the prime contractor and National Defence have never reached an agreement as to how they will measure the performance. There is still a disagreement between the parties as to what constitutes availability of the aircraft. While the prime contractor may have the required number of aircraft operational on a particular day, factors such as slow turnaround of the aircraft may cause the schedule to slip such that it results in the cancellation of sorties. These missed sorties lead to differences between the required and actual number of sorties provided and it also raises questions as to the level of contractual compliance. This issue has been outstanding since the beginning of the program and while progress has recently been made, there is still no agreement on what constitutes successful performance by the prime contractor.

There are no mechanisms in the contract to allow for changes

4.58 Unlike many standard government contracts, this contract does not include any specific clauses describing allowable ways to change the contract. Even though it is not possible to foresee all situations that may occur, particularly given the long-term nature of the agreement, we would have expected the contract to include a clause that would outline the procedures for implementing changes. The personnel of Public Works and Government Services Canada that are involved in the management of the contract have recognized the need for a change process. They have begun work on getting an agreement to outline a procedure for changes to the contract.

4.59 In its 1999 report on Examining the Value for Money Deals under the Private Finance Initiative, the United Kingdom National Audit Office states that in long term contracts, "change is inevitable as the Department's needs and priorities will alter over time." The program does allow for some adjustment by selling training capacity between countries and drafting of memoranda of understanding, but any contract changes would involve re-negotiations.

The NATO Flying Training in Canada program did not include management practices consistent with a program of this size and risk

4.60 The government recognizes the risk associated with managing large, complex projects and has a policy in place to identify and manage them. Projects that exceed $100 million and that are assessed as high risk are considered major crown projects. The NATO Flying Training in Canada program is a complex project valued at about $150 million per year for 20 years. Although the NATO Flying Training in Canada program exhibits many characteristics that are common to major crown projects, it was not managed as one. Treasury Board Secretariat officials told us that, at the time, the policy was only applied to capital acquisitions while this project was considered to be a service delivery contract. It was subject to a high-level of management reporting and oversight.

4.61 Nevertheless, due to similar characteristics, we expected to find similar management practices as part of this program's initial governance arrangements, such as an integrated risk management process and a performance measurement system. As a result of some of the difficulties realized during the first two years of implementation, Public Works and Government Service Canada officials have now brought in some of the management structures described in the Treasury Board Manual section on major crown projects.

4.62 The staffing and personnel requirements for the program were based on the assumption, made in the absence of a satisfactory risk assessment, that the contract was a routine service contract, despite its large dollar value and multi-year nature. The program was not given the same level of resources that a major crown project would receive. Public Works and Government Services Canada have recently hired a person to act as the local contract officer in Moose Jaw and provide a link between the site and the project office.

The program is slow in devolving a management framework to the users

4.63 Administration of the contract and dealings with the prime contractor have been centralized in the project management office at National Defence. The program was planned so that the operation would be producing the contracted levels of service by February 2002. However, given the number of outstanding contractual issues that have not been resolved the project management office will remain in place for an additional two-year period.

4.64 The 15 Wing Moose Jaw 2002 business plan identifies the lack of a clear governance and accountability chain to allow timely decision making and effective management. Until very recently, limited information or decision-making authority have been devolved to the local training personnel, even though they are the ones who have to deal with prime contractor on a daily basis and are responsible for ensuring that the program meets the needs of the students. It also identifies constraints on their ability to plan and operate effectively, including shortfalls with the NATO Flying Training in Canada program and contract and misunderstandings with the prime contractor, such as

• the command and control structure is too complicated,
• there is not a clear understanding of responsibilities, and
• there is a lack of visibility of available resources which has resulted in cumbersome and protracted planning and decision making cycles.

4.65 Officials of National Defence and Public Works and Government Services Canada have recently started developing a governance and accountability framework as part of the NATO Flying Training in Canada program's transition to the steady state level of operations. Public Works and Government Services Canada has created a director level position to oversee the project. This has assisted in bringing some major issues to the foreground. (See photograph)

4.66 In an internal departmental study released in April 2002, National Defence identified many of the issues being reviewed in this audit, specifically

• no single point of accountability for the program,
• poorly defined roles and responsibilities with no person or position clearly responsible for dealing with unique governance issues,
• lack of communication between all levels and slow response to local base concerns,
• disjointed or non-existent business planning,
• lack of understanding of prime contractor responsibilities,
• a limited number of personnel who understand the terms of the NATO Flying Training in Canada contract,
• lack of skilled personnel to address contracted service problems,
• lack of succession planning, and
• poor Air Force control over the pilot production process.

4.67 The report does note some strengths of the existing program, including the strong central control and effective contract management performed by the project office, and the benefits of co-locating the technical authority, contract management and Public Works and Government Services Canada personnel within the same office.

National Defence is looking into new management arrangements for the program and lessons learned

4.68 There are various governance studies currently under development within National Defence which have identified particular concerns with the management and accountability framework. These governance studies come well into the third year of operation of the program. We expected that the project team would have had a plan to guide the transition to the steady state level of operations. Still, we are encouraged that National Defence and Public Works and Government Services Canada are addressing these issues at this time.

4.69 Given the unique structure and the creative approaches used in the provision of services under this agreement, there is increased risk for the Department but also a significant opportunity for learning and innovation. Since National Defence is looking to enter into similar agreements in the near future with the re-tendering of flight training services at Portage La Prairie and other service contracts, we expect that the lessons learned from the NATO Flying Training in Canada program would be documented for use both across the department and across government. However, formal lessons learned have not yet been recorded on this project either by National Defence or Public Works and Government Services Canada.

4.70 National Defence has recently started work on a major service delivery framework. Although this initiative is in the early stages of development, the aim is to provide a framework for large-scale contracts for the delivery of complex services. Its purpose is to provide National Defence with easy access to sources of information and set the stage to develop and use an integrated set of policies, procedures, processes, and tools.

Procurement Reform initiatives are underway but progressing slowly

4.71 In our 1999 Report, we recommended that, as part of its on-going work on procurement reform, guidelines and training be developed for large multi-year service contracts and that these should address key issues of how competition is to be addressed where long-term "partnering" would be beneficial to the government. In 1999, the Treasury Board Secretariat indicated that they were developing a guide and a database on alternative service delivery. The Treasury Board Secretariat was also leading an interdepartmental initiative on procurement reform, which would include guidance for large, multi-year service contracts. Public Works and Government Services Canada also indicated that they would be working with the Treasury Board Secretariat in this regard.

4.72 In our December 2001 follow-up on alternative service delivery we reported that the Treasury Board Secretariat, Public Works and Government Services Canada and National Defence were working on a framework and best practices guide on long-term complex contracts. They have recently completed a paper on Strategic Direction and Best Practices in Complex Procurement. The paper documents the strategies and techniques that have been developed by Public Works and Government Services Canada and National Defence on dealing with an increasingly complex procurement environment and Treasury Board Secretariat officials said that it will be used as a guide for future procurement.

4.73 In April 2002, the Treasury Board Secretariat released a new Policy for Alternative Service Delivery. While the new policy provides some limited guidance on the governance and performance monitoring for alternative service delivery projects, it provides little direction for long term, large dollar value service arrangements. The planned alternative service delivery database is not yet operational.

4.74 Public Works and Government Services Canada is developing a training course covering complex procurement. Department officials informed us that the program has been piloted and is nearing completion of the development phase.

Conclusion and Recommendations

4.75 National Defence did not train the contracted number of student pilots for which it paid in the NATO Flying Training in Canada program. During the first two years of this program it used only 40 percent of the basic Phase IIA flying training slots, 28 percent of the basic Phase IIB training, 50 percent of the advanced Phase III training and 58 percent of the Phase IV fighter lead-in training for which it has contracted. We estimate that as of the end of December 2001, the Department had paid about $65 million more than the value of training received. This amount is growing and will continue to grow as long as the program remains underutilized.

4.76 National Defence recognizes that it has had program management problems. Studies are underway to determine how best to manage the NATO Flying Training in Canada program but there are outstanding issues that must be resolved. Developing a solid governance arrangement is the first step in improving overall management.

4.77 Recommendation. The Department should resolve the program management issues and implement a revised management framework as a matter of urgency.

National Defence's response. The Department will continue to evolve the governance structure to meet complex needs and a steady state structure will be put in place over the next year. NATO Flying Training in Canada is a unique and complex program being managed in a constructive and evolutionary fashion. There has been significant oversight at senior management levels from two government departments during the difficult start-up period. Now responsibilities are being slowly, carefully, and appropriately delegated to various levels within both the Department and Public Works and Government Services Canada. Although there has been much discussion and some differences of opinion over the program's steady state governance structure, we maintain that identifying and then resolving issues reflects an appropriate management approach.

4.78 We have reported on underused training capacity in the past and have concerns that problems may be repeated as new contracts are negotiated. This program does not have the flexibility needed in long-term contracts to accommodate changing needs over time or the measures to ensure performance meets expectations.

4.79 Recommendation. Lessons learned from the NATO Flying Training in Canada program should be documented and reported to senior management with an action plan.

National Defence's response. The Department agrees that formal documentation of lessons learned is important. Moreover, lessons learned in the NATO Flying Training in Canada Program are already being applied in real-time to the Contracted Flying Training and Support project that is being managed within the same project office. The Department's intent is to formally document all applicable lessons this fiscal year and then promulgate them through appropriate media.

4.80 Recommendation. New contracts of a similar nature to the NATO Flying Training in Canada program that the Department enters into should ensure that

• payments are tied to performance and value received, and
• a project management framework commensurate with the risk and design of the program is in place at the outset.

National Defence's response. It is unlikely that many of the unique risk-sharing features of the NATO Flying Training in Canada contract will arise in the foreseeable future. However, as was the case with this particular contract, the Department will continue to tailor project management structures and contract terms and conditions to the unique circumstances of each contractual arrangement.

4.81 Recommendation. The Department should ensure that the services purchased through a long term service contract are aligned with the ability to use them.

National Defence's response. The Department agrees that this is an appropriate goal but disagrees with the implication that this was not the case with NATO Flying Training. We believe that the under utilization of capacity observed by the Office of the Auditor General was a reflection of the challenges inherent in starting up a new program of this magnitude and complexity. The Department remains confident that purchased training capacity will be more fully utilized in future as the program matures. In the NATO Flying Training program, the possibility of reduced training demand over time was considered and there is a contractual provision for Canada to sell off 50 percent of our "jet pilot" training capacity to other nations. Further, since the NATO Flying Training in Canada program was designed to be marketed, there is a built-in marketing infrastructure to dispose of non-required capacity should that occur.

National Defence's comments. The NATO Flying Training in Canada Program is a unique program of risk-taking and risk-sharing, which, despite some start-up hurdles that are being overcome, represents a huge leap forward in training technology and training philosophy. The Department agrees that training production in Phase II has been less than expected in the first two years, but asserts that any missed training will be made up later in the contract. The contractor has an obligation to make up all the contracted training over the life of the program and the contract is being enforced.

This program represents the best long-term option for the training of Canada's military pilots. NFTC is a success with over $1 billion of training sold internationally. NFTC is now the recognized benchmark against which future pilot training systems are being compared.

About the Follow-Up

Objectives

This follow-up audit had two objectives:

• to examine the implementation and management of the NATO Flying Training in Canada program and to determine whether the service contract was meeting the needs of the department in an economical and efficient way, and
• to review whether the Treasury Board Secretariat and Public Works and Government Services Canada had made any progress in responding to our recommendations about procurement reform and "partnering" as stated in the 1999 chapter on alternative service delivery chapter.

Scope

Our audit focussed on the management framework and processes with respect to the NATO Flying Training in Canada program in place at National Defence, the Canadian Forces, 1 Canadian Air Division and the Canadian Aerospace Training Program office. In addition, it looked at the management and delivery of contracted services at 15 Wing, Moose Jaw, and 4 Wing, Cold Lake.

We assessed the implementation and management of the NATO Flying Training in Canada program from the time the contract was signed in May 1998 until March 2002. This represents a very short period in the overall life of the 20-year contract worth $2.8 billion. While our audit focussed on this time period, we have made certain comments about the way the planning was done in the pre-contract stage.

We also focussed on work done to date by the Treasury Board Secretariat and Public Works and Government Services Canada on implementing our 1999 recommendations.

We did not audit the activities of the prime contractor and, as a result, do not comment on them. We do bring forward facts regarding their activities but limit our comments to the whether the actions of government officials were appropriate or not.

Criteria

We followed up on the recommendations for procurement reform made to the Treasury Board Secretariat and Public Works and Government Services Canada in our 1999 audit on alternative service delivery.

We expected that National Defence would ensure that the goods and services contracted for would be delivered as stated in the contract. We expected that National Defence would act in a way compatible with the Financial Administration Act to ensure that only money earned is paid.

We also expected that the Canadian Forces would be able to train the number of pilots required as indicated in the contract in a cost-effective way.

Ratings

We assessed the action of departments/agencies against our original audit recommendations (see Key Message at the beginning of the chapter). We used the following ratings:

• Completed. Corrective action has been fully implemented.
• Satisfactory progress. Progress is being made at a satisfactory pace.
• Limited progress. Some progress is being made, but the pace or scope is not satisfactory.
• No progress. No evidence of progress although the department or agency accepted the recommendation from the original audit.
• Rejected. The department or agency did not accept the recommendation from the original audit.
• Unknown. Status of progress is unknown or information is not available.

Audit team

Assistant Auditor General: Hugh McRoberts
Principal: Wendy Loschiuk

Richard Delano
James Harris
Christopher MacDonald

For information, please contact Communications at (613) 995-3708 or
1-888-761-5953 (toll free).