This Web page has been archived on the Web.

2008 May Report of the Auditor General of Canada

Main Points

What we examined

Transport Canada's responsibilities for air transportation safety include promoting safety; developing regulations; and overseeing compliance with them by airlines, aircraft maintenance companies, manufacturers, airports, air traffic control, and other sectors of the industry. The Department is now adopting a new approach to oversight, based on the implementation of safety management systems (SMS). The approach will require aviation companies to have in place a system for managing the safety risks linked to their operations. Transport Canada's oversight role will change from one focused solely on conducting inspections and audits to one of assessing the processes that companies have in place for ensuring safety—although direct inspections and audits may still be carried out if necessary.

We examined how Transport Canada has managed the transition to the new approach for the first sectors to make the shift: airline operators and associated aircraft maintenance companies. We did not examine the level of air transportation safety in Canada. Nor did we look at security—that is, protection against deliberate acts such as terrorism.

Why it's important

The civil aviation sector is a key element of Canada's transportation network and its economy. In 2006, air transport in Canada carried 99 million passengers, up 6 percent from 2005, and the number is expected to grow 40 percent from 2006 to 2015. According to the International Civil Aviation Organization (ICAO), the rapidly expanding aviation industry and the limited resources of oversight authorities make it increasingly difficult to sustain the existing approach to managing safety. ICAO has stated that by 2009, each member country must establish a safety program requiring aviation companies to implement a safety management system acceptable to the country's regulating authority.

For effective oversight, it is critical that the transition to the new approach be well managed, that oversight continue throughout the transition, and that Transport Canada understand and mitigate the risks inherent in the transition.

What we found

• As the first civil aviation authority to put in place regulations requiring aviation companies to introduce SMS, Transport Canada developed its own approach. For example, it conducted pilot projects with airlines and small operators and used the results to establish milestones. It also monitored activities and made adjustments to ensure that all regions applied procedures consistently. However, in planning for the transition, the Department did not document risks, such as the impact of the transition process on oversight of air transportation safety, and identify actions to mitigate these risks. Nor did it forecast the overall costs of managing the change.
• Resources have been shifted from traditional oversight activities to SMS activities. However, the Department has not measured the impact of this on the frequency of traditional oversight activities.
• Transport Canada has not yet identified how many inspectors and engineers it needs, with what competencies, during and after the transition. The impact of SMS is being addressed in the reorganization of the Department's Civil Aviation program, now under way. Given that this is not expected to be completed before the end of 2009, Transport Canada could find itself unable to recruit the right mix of skills when it needs them.
• The Department has not developed short- and medium-term performance indicators—those that could signal a need for closer attention or action in a particular area—to measure the impact of its civil aviation activities.

Transport Canada has responded. The Department agrees with all of our recommendations. Its detailed responses follow each recommendation throughout the chapter.

Introduction

Transport Canada's role in civil aviation

3.1 Transport Canada's civil aviation mandate derives primarily from the Aeronautics Act, the Canadian Aviation Regulations, the Transportation of Dangerous Goods Act, 1992, and the standards of the International Civil Aviation Organization (ICAO). Under the Aeronautics Act, the Minister of Transport, Infrastructure and Communities has broad powers to inspect, audit, and enforce regulations related to any aircraft or airport, or any premises used for the design, manufacture, distribution, maintenance, or installation of aeronautical products.

3.2 Transport Canada's mission for civil aviation is "to develop and administer policies and regulations for the safest civil aviation system for Canada and Canadians using a systems approach to managing risks." The Department defines safety as the condition where risks are managed to acceptable levels, and determines this on a case-by-case basis through a risk assessment process. Transport Canada's responsibilities include

• the development of laws, regulations, and policies related to civil aviation safety;
• licensing of aviation personnel;
• certification of aviation companies and products, such as aircraft and engines;
• safety education and promotion; and
• oversight of aviation companies to assess their compliance with safety regulations.

3.3 The Department is involved in all aspects of the aviation industry, including air operators (Canadian and foreign airlines, and smaller operators), maintenance companies, aircraft manufacturers, airports, and air traffic control (Exhibit 3.1).

Exhibit 3.1—Elements of the aviation industry regulated by Transport Canada Civil Aviation

Organizational structure

3.4 Transport Canada carries out its responsibilities for civil aviation through the Civil Aviation Directorate at headquarters in Ottawa and the Civil Aviation branches in the regions. Together these constitute the program known as Transport Canada Civil Aviation (Exhibit 3.2). The Civil Aviation Directorate is part of Transport Canada's Safety and Security Group, which is responsible for developing regulations and national standards; implementing safety and security oversight programs in the aviation, marine, rail, and road modes of transportation; and regulating the transportation of dangerous goods.

Exhibit 3.2—Reporting structure for Transport Canada Civil Aviation

3.5 In each region, the Director of Civil Aviation reports to a Regional Director General for all transportation modes, who in turn reports directly to the Deputy Minister. The regional directors of Civil Aviation have a functional reporting relationship with the Director General, Civil Aviation, whereby headquarters is authorized to issue policies and procedures and to monitor regional compliance with the national program.

3.6 Headquarters is responsible for developing the content, policy, and standards for oversight activities. The regional Civil Aviation branches, along with the National Aircraft Certification Branch and the National Operations Branch at headquarters, carry out the program. The National Operations Branch is responsible for some of the companies operating nationwide, such as Air Canada and WestJet. A total of 1,500 employees work in Transport Canada Civil Aviation at headquarters and in the five Transport Canada regions (Exhibit 3.3).

Exhibit 3.3—Transport Canada's regions

Current priority

3.7 The current priority of Transport Canada Civil Aviation is to introduce a new way of managing oversight of civil aviation, based on the implementation of safety management systems (SMS) by aviation companies (Exhibit 3.4). The International Civil Aviation Organization (ICAO) has stated that by 2009, member countries must establish a safety program requiring aviation companies to implement safety management systems acceptable to the country's regulating authority.

Exhibit 3.4—Transport Canada is adopting a safety management systems approach

3.8 According to ICAO, the rapidly expanding aviation industry and limited resources of oversight authorities make it increasingly difficult to sustain the existing approach to the management of safety. ICAO sees a need to complement the current regulatory approach with safety management systems: "SMS presents the international aviation community with a principled, data-driven approach to determining priorities and allocating the resources required to address safety concerns that hold the greatest risk potential, and towards activities likely to produce the biggest return on resources invested."

3.9 Transport Canada has already introduced new SMS regulations affecting some sectors of the aviation industry. With this new approach, the Department's primary involvement with industry will be at the level of each company's management systems. The oversight role will change from one focused solely on conducting inspections and audits to one of assessing the processes that a company has in place for ensuring safety—although direct inspections and audits may still be carried out if necessary. For example, instead of conducting an inspection to assess whether the tires in the aircraft landing gear are sufficiently inflated, Transport Canada will assess whether a company has the systems in place to ensure that the tires are inflated, following up if necessary. The goal is to make companies more accountable for the management of risks. Transport Canada will still be accountable for safety oversight. The Department maintains that safety management systems will allow more thorough identification and resolution of potential safety problems, making the transportation system safer.

3.10 SMS-based oversight is expected to be fully in place once regulations for all sectors of the aviation industry are in effect, and all companies have been able to implement SMS (expected between 2011 and 2013). Until then, Transport Canada has the task of managing the transition, while continuing its oversight responsibilities.

3.11 While making the transition to SMS-based oversight, Transport Canada has started two other major initiatives for the Civil Aviation program. The first is the implementation of an Integrated Management System (IMS) to coordinate and standardize the management processes, activities, and practices of the Civil Aviation program. A key part of IMS was a gap analysis of practices to identify opportunities for improvement. The second initiative is the reorganization of the Civil Aviation Directorate and regional branches, referred to as the National Organization Transition Implementation Project (NOTIP). The project examines how the organization will be structured to deliver the Civil Aviation program in the future, in accordance with the new Integrated Management System.

Focus of the audit

3.12 The main focus of this audit was Transport Canada's management of the transition to SMS-based oversight of the first sectors to implement SMS—airline operators and associated aircraft maintenance companies. These two sectors comprise 74 companies, including large international and national carriers, such as Air Canada, WestJet, and Air Transat, as well as regional operators and maintenance companies.

3.13 We looked at Transport Canada's transition planning, its work to date with the companies implementing SMS, and its monitoring of compliance with regulations during the transition. We also examined the Department's approach to risk management, human resources planning and training, and performance measurement for safety oversight activities. Our audit was conducted in the midst of the transition. Lessons learned in the early stages can be applied to later implementation of SMS by other categories of air operators (for example, air taxis) and other sectors of the industry (for example, aircraft manufacturers, airports, and air traffic control).

3.14 More details on the audit objective, scope, approach, and criteria are in About the Audit at the end of this chapter.

Observations and Recommendations

Planning for the transition

3.15 In 1999, Transport Canada announced that it would pursue a new approach to oversight of civil aviation safety, based on safety management systems (SMS). New SMS regulations will have an impact on the Canadian aviation industry as well as on Transport Canada itself. For a transition of this scope, we expected the Department to have in place a project plan documenting risks and mitigating actions, expected costs and resource requirements, milestones, roles and responsibilities, and internal communications needs.

Transport Canada is the first civil aviation authority to have regulations for safety management systems

3.16 Transport Canada is the first civil aviation authority in the world to put in place regulations requiring aviation companies to implement safety management systems. Given the lack of precedents in other countries for the new regulatory framework, the Department had to develop its own approach for SMS implementation.

Planning was done for the transition but did not cover some key elements

3.17 Transport Canada developed a timetable that determined the order in which the various sectors of the aviation industry would be required to implement SMS. The Department also prepared several documents that explained the components of SMS and its impact on aviation companies. In addition, the Department conducted pilot projects with airlines and small operators to test how implementation would work. Based on these pilot projects, Transport Canada determined that a phased-in approach to implementation was needed.

3.18 The Department told each region to structure itself appropriately to undertake SMS-related activities and to appoint regional SMS coordinators to be responsible for coordinating these activities. A communications plan was developed, and senior executives visited each region to make presentations on the new approach.

3.19 However, the Department did not document risks, such as the impact of the transition process on oversight of air transportation safety, or prepare mitigating actions. It also did not forecast overall expected costs for the transition. Although a communications plan was prepared, the plan was not evaluated and adjusted regularly over the course of the transition. The Department did not explain how the day to-day work of inspectors would change as SMS-related activities were integrated with traditional oversight activities. It also did not explain the connection between the transition to SMS and the delegation to industry of checks of company pilots, an activity requiring specialized qualifications. Inspectors were told during their SMS training that the transition to SMS involved a shift from specialized and technically trained inspectors to systems auditors and analysts. In 2007, the Department found it necessary to clarify that it would still need inspectors with specialized skills.

3.20 Recommendation. For the introduction of safety management systems in other sectors of the aviation industry, Transport Canada should prepare a project plan that documents risks to the Department, mitigating actions, and expected costs, and provides a timetable for evaluating and adjusting communications efforts.

Transport Canada's response. The Department agrees. The Department intends to enhance the existing safety management systems (SMS) transition plan to include an assessment of the risks and mitigating actions and the expected costs and resource requirements. The communications plan, which is currently an integral part of the existing SMS plan, will be improved with the addition of a timetable for evaluating and adjusting communications efforts.

Implementation to date

3.21 Transport Canada is introducing safety management systems (SMS) by stages (Exhibit 3.5). Regulations have been in place since 31 May 2005 for SMS implementation in the sectors that include airline operators and the companies that perform maintenance on their aircraft. The 74 companies in these sectors have an exemption from the regulations until September 2008, allowing them time to develop and implement their own safety management systems. Companies that applied for an Air Operating certificate or Approved Maintenance Organization certificate on or after 31 May 2005 did not receive an exemption and must already have SMS in place.

Exhibit 3.5—The safety management systems approach is being implemented over several years

3.22 For companies in the first sectors to implement SMS, Transport Canada developed a four-phase approach. Each phase involves putting in place some of the components required for safety management systems within a set time frame. Transport Canada then reviews the components that the company has put in place in a process called an acceptance validation. The Department will only assess the effectiveness of each company's SMS after the four phases are complete. Exhibit 3.6 shows the phases of implementation for the first sectors.

Exhibit 3.6—The implementation of safety management systems for airline operators and related maintenance companies is done in four phases

Validation procedures have been developed and are being followed

3.23 We examined the two phases of SMS implementation completed by September 2007. We expected Transport Canada to have developed appropriate procedures and processes for SMS implementation, and to have applied them consistently to aviation organizations.

3.24 We found that the Department had developed procedures and processes for its inspectors to carry out the SMS validation activities. The Department conducted an assessment of these activities and found that the majority of regional teams indicated that they required further direction to carry out validations. The assessment noted that instructions should include more information in some areas, particularly for planning and preparing for future assessments.

3.25 To assess whether procedures were followed, we examined 21 files documenting validation activities for phase 2 of SMS. The files included representation from all regions and various sizes of air operators and maintenance companies. Although there were minor deviations, we found that validation teams completed the activities according to established procedures, and most files had the required documentation. We noted that in some cases Transport Canada found deficiencies and followed up to ensure that these were corrected. The teams of inspectors were able to conclude that all of the 74 companies had in place the SMS processes required for phase 2.

Efforts are being made to monitor consistency during validations

3.26 To monitor consistency in the application of procedures, SMS regional coordinators in some of the regions were present at all validation activities. In addition, after completing phase 1, managers from all of the regions met and produced a document on lessons learned.

3.27 At the end of phase 2, the Department developed a report noting that despite slight deviations by some inspectors, the SMS validation teams in the regions followed the procedures provided by headquarters to achieve their objectives. The phase 2 report contains recommendations for enhancing the effectiveness and efficiency of validations by improving procedures, processes, guidance, and training. We encourage the Department to develop an action plan to address these recommendations.

Monitoring of compliance during the transition

3.28 At the end of the four phases of SMS implementation, Transport Canada anticipates that it will be able to assess the effectiveness of safety management systems implemented by companies. Until this can be done, we expected Transport Canada to have mechanisms in place for traditional oversight to monitor compliance with regulations during the transition.

Risk analysis methods vary in traditional oversight

3.29 The SMS validation activities that precede the assessments involve considerable effort and the on-site presence of inspectors at companies to assist in the transition. To free up resources for these activities, the Department has supported the shifting of resources from traditional audits and inspections to SMS-related activities. The Department has a Frequency of Inspection Policy Document that outlines in detail the required frequency of traditional oversight tasks. For example, the policy states that certain airline operators must undergo an audit every 6 to 36 months. This policy states that the frequency can be adjusted depending on available resources and on the basis of an assessment of risks. To determine the frequency of traditional oversight tasks during the transition, while SMS activities are also occurring, managers have been asked to use their judgment while referring to risk indicators described in the policy, such as a company's accident record.

3.30 Each region, however, uses different risk management techniques and processes to assess risk. Managers do not always analyze all of the risk indicators listed in the Frequency of Inspection Policy Document and may use other indicators that are not listed. Methods range from formal to informal: Some regions have created their own risk indicator databases, some document the decisions made in the risk analysis process in detail, and others make decisions without documenting details, such as the rationale. The use of different indicators and methods increases the likelihood of reaching different conclusions for similar situations. In addition, differences in the level of documentation could make it more difficult for new inspectors to understand the rationale for decisions made by inspectors who have left the Department. Transport Canada has already identified the use of different risk analysis methods as a deficiency, and it is developing a national program with standard risk indicators to be used across the regions.

3.31 Recommendation. Transport Canada should put in place its national risk indicator program for civil aviation as soon as possible, in order to have a standardized method to assess risk for the allocation of resources. The Department should document all decisions made in the risk analysis process.

Transport Canada's response. The Department agrees. The Department has already commenced work in this area. A working group was formed in May 2007 to review and develop a comprehensive set of risk indicators. The working group has completed its work and will present a final report in April 2008. A plan to implement its recommendations, which will also address this recommendation, will be developed by the end of the 2008–09 fiscal year for implementation in the 2009–10 fiscal year.

Transport Canada has not analyzed the impact of safety management systems on the frequency of traditional oversight activities

3.32 Transport Canada has not established objectives for the frequency of traditional oversight activities during the transition, including the extent to which traditional oversight activities can be adjusted. Because the frequency can be adjusted based on available resources and risk, it is unclear what combination of traditional oversight and validation activity is acceptable and why and how this may change as SMS is implemented.

3.33 Adding to this uncertainty is the lack of overall data on the frequency of traditional oversight activities. The regions report to headquarters on the extent to which they have achieved the target frequency for different tasks. For each task, headquarters looks at significant variances—generally, 25 percent above or below what is planned—and asks the regions to explain these on a case-by-case basis. However, the data is not aggregated to allow for a more comprehensive analysis. Department information systems cannot calculate the extent to which the planned frequency schedules have been met overall for the first sectors implementing SMS. This type of analysis will become even more important when the large group of small air operators and related maintenance organizations begin to implement SMS.

3.34 Recommendation. Transport Canada should establish a standard that defines an acceptable level of activity for oversight of the aviation industry, and it should specify how this will be measured during the transition to SMS and when the transition is complete. The Department should analyze the data to assess the extent to which the standard is achieved.

Transport Canada's response. The Department agrees. A standard has been developed for the oversight activities during the transition to SMS and will be available in early spring 2008.

To validate the effectiveness of the standard, the Department will conduct an analysis of the risk indicator data and information from other data sources to monitor operator compliance with the requirements.

Transport Canada has no national mechanism for monitoring consistency in oversight activities

3.35 We noted that Transport Canada has no national mechanism for monitoring consistency in oversight activities or risk assessments. Only one region has such a mechanism: It reviews audit files to determine whether procedures were followed and sufficiently documented. Other regions should consider implementing similar practices.

3.36 Without a national mechanism in place, Transport Canada cannot determine whether all traditional oversight activities are carried out consistently and according to procedures. To measure the level of consistency in its monitoring activities, the Department is developing a formal process as part of its Integrated Management System.

3.37 Recommendation. Transport Canada should establish a national mechanism to provide the desired level of assurance that policies, procedures, and processes for civil aviation oversight activities, including the assessment of risks, are followed consistently across all regions.

Transport Canada's response. The Department agrees. The Department has developed a national quality assurance (QA) process for implementation beginning in the 2008–09 fiscal year. It includes policies, procedures, and processes for Civil Aviation oversight activities. A risk assessment QA module is under development and will be implemented in the 2009–10 fiscal year.

Human resources planning and training

3.38 Civil aviation inspectors and engineers are given delegated authority from the Minister to provide safety oversight in accordance with Transport Canada's regulatory responsibility. The 890 inspectors and engineers are required to have specialized qualifications, such as a pilot's licence or an aircraft maintenance engineer licence. With the shift to SMS, they have to acquire additional skills to assess an organization's ability to detect hazards to safe operations and mitigate or eliminate them. Effective human resources planning and training are therefore critical for this initiative to succeed. We expected Transport Canada to have a human resources plan for the Civil Aviation program that is integrated with its strategic plan, incorporates both current and future needs, and includes an action plan to meet these needs. As well, we expected Transport Canada to have a sufficient number of inspectors and engineers with the right skills and competencies, in the right place and at the right time, to carry out its mandate.

There is no integrated human resources plan for the Civil Aviation program

3.39 Transport Canada began a reorganization of the Civil Aviation program in 2005, called the National Organization Transition Implementation Project (NOTIP). The project was initiated to review the workforce required for Civil Aviation's future needs in light of SMS implementation, in accordance with the new Civil Aviation Integrated Management Standard. The reorganization, which will address the organization structure at headquarters and in each of the regions, is expected to be completed by December 2009.

3.40 Headquarters and the regions each have their own human resources plans. However, these vary in the extent to which they are aligned with Transport Canada Civil Aviation's strategic plan. The plans do not address issues that cut across regions, such as the potential requirement to move staff from one region to another on a temporary basis to meet workload requirements. Transport Canada currently has no national integrated human resources plan for the Civil Aviation program that identifies its needs for SMS implementation, including the overall resources and competencies required, both during the transition and when it is complete. It is recognized that as the first national regulatory authority to introduce SMS regulations for aviation, Transport Canada has no precedents to follow; however, good human resources planning requires that resources be estimated for any new initiative.

3.41 Human resources planning is particularly critical given that the number of employees has decreased by 8 percent in the past five years (Exhibit 3.7). Departing employees take with them the highly specialized knowledge, skills, and abilities they gained on the job. Hiring, however, has not increased. Some regions submitted estimates showing increased resource requirements when small air operators and related maintenance organizations begin implementing SMS. To address these needs, some regions have put in place measures to temporarily increase staffing levels—for example, retaining retirees on contract or backfilling positions soon to be vacated.

Exhibit 3.7—The number of inspectors and engineers is decreasing

3.42 Transport Canada needs to determine requirements for the short and long term for Civil Aviation as a whole. Since NOTIP is not expected to be completed until December 2009, the delay in hiring presents a risk that Transport Canada will not be able to recruit the right mix of skills when needed. Staffing times and initial mandatory training will increase the time required to meet workforce needs.

3.43 Recommendation. Transport Canada should put in place a national human resources plan for Civil Aviation as soon as possible. This plan should be aligned with the strategic plan, specify the required number of inspectors and engineers and their competencies, and include a recruitment strategy to meet these needs.

Transport Canada's response. The Department agrees. Although regional and headquarters human resources plans already exist, the development of a national human resources plan will depend on the Department's revised Program Activity Architecture (PAA). As part of the implementation of the revised PAA, scheduled to begin in the 2009–10 fiscal year, a national human resources plan will be developed. This plan will be aligned with the strategic plan and specify the required number of inspectors and engineers and their competencies, and also include a recruitment strategy to meet these needs.

Some gaps exist in the training of inspectors and engineers

3.44 The Department provided an initial SMS course to most inspectors and engineers in 2004 and 2005. This course covered the history of SMS, document management, safety oversight, and the use of the SMS assessment guide. Many took the training well before they could apply it. This limited the course's effectiveness; however, briefing sessions were also provided at a later date. No regular recurrent SMS training has been planned. Transport Canada should ensure that ongoing training is provided to keep inspectors and engineers up-to-date on evolving procedures and practices for introducing SMS-based oversight into companies.

3.45 As part of the reorganization of the Civil Aviation program, Transport Canada will be revising competency profiles to add new competencies. Since these have not yet been defined, no training strategy has been developed to address new requirements. Other training needs have emerged, such as interviewing skills. The Department is developing courses to address these needs, but until human resources requirements are known, it will be difficult to develop a comprehensive training strategy.

3.46 Recommendation. Transport Canada should develop a training strategy that is aligned with the human resources plan to be developed for Civil Aviation. The strategy should address required competencies, training needs, courses to meet those needs, and a schedule for recurrent training.

Transport Canada's response. The Department agrees. Competencies for the non-SMS regulatory framework are identified based on the requirements to exercise individual ministerial delegations. A training strategy is in place including initial and recurrent courses.

Employees implementing the SMS regulatory framework are trained and kept current through various mechanisms as implementation continues. Given that no other regulatory authority has implemented an SMS regulatory framework, data was not available prior to the beginning of the first phase of implementation of the SMS framework to permit full identification and integration of training and competency needs. Transport Canada is continuing to monitor its training activities and will revise its strategy to address its future needs. The strategy will be integrated with the national human resources plan discussed in recommendation 3.43 to support the SMS regulatory framework at its end state.

3.47 All inspectors and engineers must undergo initial training as well as recurrent training in risk management training and aviation enforcement training. Both headquarters and the regions have systems for tracking training, but no one system contains an individual's complete training information, or all training information for all inspectors and engineers. We noted that about 15 percent of inspectors and engineers had not completed the required recurrent training; moreover, we noted that another 15 percent had not completed their initial training. These staff, therefore, do not meet the requirements for exercising all their job responsibilities, thus contravening the Department's own training policies. Headquarters is not notified of this training deficiency and does not track action to restrict the authority of individuals who do not meet the job requirements. Headquarters should be informed of the need for any restrictions on staff in carrying out the delegated authority of the Minister.

3.48 Recommendation. Transport Canada should ensure that training requirements are met for all civil aviation inspectors and engineers and that action is taken to restrict authority when required.

Transport Canada's response. The Department agrees. Training requirements are well documented, and the training status of every employee respecting their delegations of authority is documented. This recommendation will be addressed through the Quality Assurance process referenced in recommendation 3.37.

Performance measurement

3.49 The Treasury Board of Canada Secretariat requires departments to develop performance indicators for all expected outcomes in their performance measurement frameworks, and to ensure that the indicators measure both short- and long-term performance against targets by a specified date. The International Civil Aviation Organization (ICAO) recommends the use of safety indicators that are easy to measure and that are linked to the major components of the safety program. We expected Transport Canada to have an effective performance measurement framework for its civil aviation oversight activities, including a set of indicators for measuring the impact of these activities. We also expected the information gained from using the performance indicators to guide management's decision making.

Performance measures for the short and medium term are lacking

3.50 The strategic plan of Transport Canada Civil Aviation sets out two long-term objectives: "continued improvement on the high level of aviation safety in Canada" and "a high level of public confidence in [the country's] Civil Aviation Program." To measure achievement of these objectives, Transport Canada looks at accident rates and findings from surveys on public confidence. According to the Transportation Safety Board of Canada, an independent body that investigates accidents, the Canadian aviation accident rate (calculated as the number of accidents divided by Transport Canada's estimate of hours flown) has been decreasing over the past 10 years. Transport Canada's public opinion surveys found that the public confidence rating for flight safety in Canada ranged between 96 and 98 percent over the period from 2005 to 2007. The surveys were conducted by private suppliers, using telephone interviews with a random respondent sample of 2,516 in 2005, 2,507 in 2006, and 1,016 in 2007.

3.51 However, the Department does not have any short- and medium-term performance indicators to measure the impact of its civil aviation oversight activities. Leading indicators—those that measure conditions and events that precede accidents—are required. These indicators could signal a need for closer attention or action in a particular area. Some work was done on developing these indicators, but this work was set aside while the Department changed its approach to performance measurement. The Department needs short- and medium-term performance indicators to assess the extent to which its civil aviation programs and initiatives, including implementation of SMS, are contributing to achieving the Department's long-term objectives.

3.52 Several possibilities exist for developing leading indicators. ICAO stated that the acceptable level of safety shall be established by the member country's oversight authority, and it has provided examples of indicators and associated targets, such as major aircraft defect incidents, runway incursions, or airspace incidents. Transport Canada stated that it plans to measure performance based on the scores that companies achieve in their initial and subsequent SMS assessments. Other indicators and targets could be developed based on elements that are examined during more frequent oversight activities. Compliance with regulations could also be tracked. Transport Canada acknowledged that it must still address the challenge of tracking and monitoring progress in safety management.

3.53 Recommendation. Transport Canada should develop a set of quantitative and qualitative short- and medium-term performance indicators to assess the extent to which its civil aviation regulatory programs and initiatives, including implementation of SMS, contribute to achieving its long-term objectives.

Transport Canada's response. The Department agrees. The Department is currently revising its Program Activity Architecture (PAA). As part of this process, the Department will be developing performance indicators to assess the Civil Aviation regulatory programs and initiatives.

Through this restructuring process, the new PAA structure, including targets, is required to be in place for the 2009–10 fiscal year.

Databases and systems do not provide an integrated view to facilitate performance measurement

3.54 Transport Canada has noted that data, databases, and trend analysis take on increased importance for companies implementing SMS. This is equally true for the Department itself, as it will have less day-to-day involvement at companies and will increasingly need to analyze trends to assess risk.

3.55 Transport Canada has several databases and systems that contain information related to aviation safety. However, these databases and systems do not provide an integrated view of the safety profile of an aviation company or industry sector. While data can be pulled together to create such a profile, several sources must be consulted. Integrated profiles would help in determining which companies should receive closer oversight. Without easily accessible safety profiles, inspectors need to rely on several sources of data as well as their own experience. A high rate of departures means this knowledge may be lost. The Department has acknowledged that there is a gap in its management of databases and systems, resulting in duplication and underuse in some cases. Integration of the various databases would help to establish profiles of companies and industry sectors and allow for easier analysis.

3.56 Recommendation. Transport Canada should put in place a means to capture all information relevant to oversight of civil aviation safety in an integrated manner. This would allow the Department to develop and track safety profiles for aviation companies and industry sectors and to assess the relative level of risk.

Transport Canada's response. The Department agrees. Safety data is currently available in a variety of databases. Work is ongoing to identify the best application for integrating and analyzing the data available. A capital project was initiated for the purpose of capturing the various aviation intelligence in an integrated fashion.

Conclusion

3.57 Transport Canada is the first national regulatory authority in the world to put in place regulations requiring aviation companies to introduce safety management systems (SMS). The Department developed its own approach to implementation and conducted pilot projects with airlines and small operators. It developed appropriate procedures and processes for SMS implementation, and made efforts to apply them consistently. The first two phases of the four-phase approach for SMS implementation in airline operators and associated aircraft maintenance companies are now complete.

3.58 Despite these accomplishments, we found that Transport Canada's management of the transition to the new approach has had several weaknesses. In planning for the transition, the Department did not document risks or suggest mitigating actions, and it did not forecast overall expected costs for the transition. It also had no mechanisms in place to evaluate the impact of SMS activities on the frequency of traditional oversight activities. The Department has not clarified what combination of traditional oversight and SMS-related activity is acceptable as it proceeds through the transition.

3.59 In addition, the Department has not yet identified human resource requirements for Civil Aviation during the transition and once it is complete, or determined how these requirements will be met. There is no integrated human resources plan for the entire Civil Aviation program that addresses these requirements.

3.60 Finally, the performance measurement framework for Civil Aviation does not contain short- or medium-term indicators for measuring performance. These indicators could be used to signal a need for greater attention or action in a particular area.

3.61 The Department was unable to demonstrate to us that it was managing these areas satisfactorily. It is important that Transport Canada address these weaknesses for the transition in the first 74 companies and in the remaining sectors of the industry—comprising more than 2,000 companies—to be successful.

About the Audit

Objective

The objective of this audit was to determine the extent to which Transport Canada is effectively managing the transition to a safety oversight approach based on safety management systems (SMS).

Scope and approach

This audit focused on Transport Canada's Civil Aviation Directorate at headquarters and the Civil Aviation branches in the Department's five regions. We examined their planning for the transition to SMS and looked at SMS implementation to date for two sectors of the aviation industry: airline operators and companies that perform maintenance on their aircraft. We looked at ongoing oversight during the SMS transition period for these sectors, and we examined risk management techniques and processes used in decisions regarding oversight. Finally, we examined human resources planning and training, as well as performance measurement strategies for safety oversight activities.

The audit did not deal with oversight of other sectors of the aviation industry, such as aircraft manufacturers, airports, and air traffic control. We did not examine the level of air transportation safety in Canada. We also did not examine security—that is, protection from deliberate acts that could cause damage.

Our approach involved analyzing internal documentation and processes. We carried out file reviews at headquarters and regional offices. We conducted interviews with management and employees at headquarters and regional offices, and with several companies that are now implementing SMS. We also observed Transport Canada inspectors conducting oversight activities—both SMS and traditional—at aviation companies.

Criteria

Our audit was based on the following criteria:

• Transport Canada should have in place a project plan that describes the transition to safety management systems, documenting risks and mitigating actions, expected costs and resource requirements, milestones, roles and responsibilities, and internal communications needs. (Source: Treasury Board of Canada Secretariat Project Management Policy)
• The Department should develop appropriate procedures and processes for SMS implementation and should apply them consistently to aviation organizations. (Source: Transport Canada Civil Aviation Directive 31—Safety Management Systems, and Transport Canada Civil Aviation Acceptance Procedures for Phase 2 of the SMS Regulatory Exemption)
• The Department should have safety oversight mechanisms in place during the transition to SMS to monitor compliance with regulations. (Source: International Civil Aviation Organization Safety Management Manual, 2006)
• The Department should apply appropriate risk management techniques and processes consistently in its decision making and should document decisions. (Source: Transport Canada Civil Aviation Directive 30—Risk Management and Decision-Making)
• The Department should have a sufficient number of inspectors with the right skills and competencies, in the right place at the right time, to carry out its mandate. (Source: Canadian Public Service Agency)
• The Department should have a human resources plan for the Civil Aviation program that is integrated with its strategic plan, incorporates both current and future human resources needs, and includes an action plan to meet those needs. (Source: Canadian Public Service Agency Integrated HR and Business Planning Toolkit)
• The Department should have an effective performance measurement framework for its civil aviation oversight activities, including a set of indicators for measuring the impact of these activities. (Source: Transport Canada Civil Aviation Integrated Management System Standard)

Audit work completed

Audit work for this chapter was substantially completed on 30 November 2007.

Audit team

Assistant Auditor General: Mark G. Watters
Principal: Alain Boucher
Lead Director: Sharon Clark
Directors: Christine Boulanger, Esther Becker

Isabelle Marsolais
Marc Bélanger
Jean-Philippe Boucher

For information, please contact Communications at 613-995-3708 or 1-888-761-5953 (toll-free).

Appendix—List of recommendations

The following is a list of recommendations found in Chapter 3. The number in front of the recommendation indicates the paragraph where it appears in the chapter. The numbers in parentheses indicate the paragraphs where the topic is discussed.