This Web page has been archived on the Web.

2008 December Report of the Commissioner of the Environment and Sustainable Development

Chapter 2—Managing Severe Weather Warnings—Environment Canada

Main Points

Introduction

Effects of severe weather on the well-being of Canadians
Special weather statements and severe weather watches and warnings
Environment Canada's weather and environmental services
The process of producing severe weather warnings
Focus of the audit

Observations and Recommendations

Environment Canada's weather monitoring network

Life cycle management has not been implemented
There is no up-to-date strategy to address challenges in managing the monitoring network

Implementation of the NinJo workstation

The Department is implementing a modern forecasting system for its storm prediction centres
Project management has weaknesses

Delivery of severe weather warnings

Environment Canada is consulting with selected user groups
A variety of tools are used to deliver severe weather warnings
There is no national public alerting system

Verification of severe weather warnings

There is no national verification approach or program
Environment Canada is taking steps to determine how well severe weather warnings inform Canadians

Strategic direction

A new results structure is in place
The long-term perspective has eroded

Conclusion

About the Audit

Appendix—List of recommendations

Exhibits:

2.1—Severe weather events in Canada can be costly

2.2—The production of weather forecasts and weather warnings involves several steps

2.3—Canada's radar network plays an important role in forecasting severe weather

Main Points

What we examined

Environment Canada is Canada's primary source of weather information. Its weather services are delivered not through a single unit within the Department but through a variety of departmental activities, overseen by a committee of senior managers—the Weather and Environmental Services Board (WESB). In the 2006–07 fiscal year, spending on weather and environmental services accounted for around $340 million, or about one third of Environment Canada's annual budget.

Environment Canada reports that it issues over 10,000 severe weather warnings, watches, and statements every year. We examined whether the Department's systems and procedures adequately support the delivery of timely and accurate severe weather warnings to Canadians, now and in the future. We looked at Environment Canada's management of its weather observation network, the implementation of a new, advanced forecaster workstation called NinJo, and the delivery and verification of severe weather warnings. We also examined the Department's planning process that supports the delivery of severe weather warnings. We did not examine Environment Canada's public forecasting program or other weather and environmental services such as those related to air quality, climate, or hydrometric services.

Why it's important

Severe weather such as tornadoes, severe thunderstorms, heavy snowfalls, and freezing rain can result in property damage, crop damage, injuries, and loss of life. Severe weather can affect many sectors of the Canadian economy from transportation, to tourism, to agriculture.

The Intergovernmental Panel on Climate Change predicts that Canadians can expect more severe weather events as a result of climate change. Timely and accurate warnings of severe weather can allow Canadians to take appropriate action. For example, a high heat and humidex advisory can trigger media announcements about ways to beat the heat, help ensure that socially isolated individuals and other vulnerable groups are reached, and lead to the opening of public cooling centres.

What we found

  • Although there has been some ad hoc verification in most regions, Environment Canada does not have a national program or approach to verify the timeliness, accuracy, or effectiveness of its severe weather warnings. This type of information would allow the Department to know how good a job it is doing across Canada, whether current performance is reasonable, and where it needs to make improvements to its services. Such information would also assist it in making sound investment decisions.
  • The Department relies on a number of means to issue severe weather warnings to Canadians, including the Internet and the media. Despite its efforts, there is no national system in place that automatically warns the public about severe weather events or other emergencies. Such a system would communicate warnings over all radio and television stations and through mobile devices such as cellular phones. Weatheradio is Environment Canada's only tool that automatically alerts the public of severe weather warnings, but national public surveys indicate that the public's use of Weatheradio is low.
  • Environment Canada has not adequately managed its weather observation networks, including radar and surface stations, to ensure that it can continue to provide the necessary data to issue and verify severe weather warnings. Environment Canada does not have the information on the performance of its assets—for example, trends in repair costs and failure rates during severe weather events—that it needs to make investment decisions over the life cycle of the assets. Investment decisions include whether to repair equipment or replace it.
  • The Department is facing many significant challenges—such as implementing a robust and useful system to verify severe weather warnings, managing its monitoring networks over their life cycle with limited resources, and addressing the risks related to the Department's current strategy of relying on partners. It does not have an up-to-date long-term strategy for meeting those challenges and ensuring that it can continue to deliver timely and accurate severe weather warnings in the future.

The Department has responded. The Department agrees with all of our recommendations. Its detailed responses follow each recommendation throughout the chapter.

Introduction

Effects of severe weather on the well-being of Canadians

2.1 Severe weather, such as heavy snowfalls, freezing rain, severe thunderstorms, and tornadoes, can cause loss of life, injury, and damage to property and businesses. Many sectors of the Canadian economy, such as transportation, tourism, energy, and agriculture, can be significantly affected by severe weather. Those that have to respond to emergencies that result from severe weather, including the police and fire departments, can also be affected.

2.2 Severe weather events can be local and short in duration, such as tornadoes, while other events, such as severe snowstorms, can cover large geographical areas and last for days. These events can be costly (Exhibit 2.1). The Intergovernmental Panel on Climate Change predicts that with climate change, Canadians can expect to experience more severe weather, including intense rain events, heat waves, and storm surges. Warnings can help to minimize damages.

Exhibit 2.1—Severe weather events in Canada can be costly

  • Severe thunderstorms, Southern Ontario (2005): The Insurance Bureau of Canada estimated damage of about $500 million, the greatest insured loss in the province's history.
  • Hurricane Juan, Nova Scotia (2003): This was one of the most powerful and damaging hurricanes to ever affect Canada. The category 2 hurricane had winds reaching up to 174 kilometres an hour and caused 8 deaths.
  • Tornado, Pine Lake, Alberta (2000): This tornado hit at 300 kilometres an hour, resulting in 12 deaths, more than 140 injuries, and $30 million in economic losses.
  • Ice storm, Quebec, Ontario, and New Brunswick (1998): This storm had an estimated cost of $5.4 billion, resulting in the largest insurance payout in Canadian history, with more than $1.6 billion paid by insurers. The storm caused 28 deaths and directly affected more people than any previous weather event in Canada.
  • Floods, Saguenay, Quebec (1996): In less than 36 hours, 290 millimetres of rainfall caused at least 10 deaths; washed out thousands of homes, roads, and bridges; and downed power lines. A total of 15,825 people had to leave their homes.

Sources: Canadian Disaster Database, Public Safety Canada; Environment Canada; Insurance Bureau of Canada

Special weather statements and severe weather watches and warnings

2.3 Environment Canada issues alerts to Canadians about weather conditions that are occurring or expected to occur and that could have an impact on their safety and/or cause damage to property or the environment. Alerts can take the following forms:

  • Special weather statements. These statements are issued for weather events that are unusual, cause general inconvenience or public concern, and cannot be adequately described in a weather forecast.
  • Severe weather watches. These watches provide a "heads-up" that conditions are favourable for a possible summer or winter storm. They may be issued up to 6 hours before summer events and at least 12 to 24 hours before winter events.
  • Severe weather warnings. These warnings alert the public that severe weather is occurring or will occur. They are issued when severe weather is expected to affect land-based communities within 6 to 24 hours.

Hurricane Juan, Prospect, Nova Scotia, September 2003

Hurricane Juan, Prospect, Nova Scotia, September 2003

Source: Environment Canada, Photo: Doug Mercer

2.4 These weather alerts may be issued for a variety of weather conditions, such as hurricanes, tornadoes, severe thunderstorms, heavy rainfall or snowfall, winter storms or blizzards, high winds, blowing snow, freezing rain or drizzle, hail, frost, and temperature-related extremes (for example, high heat and humidity, wind chill, and flash freezes). Environment Canada also issues marine warnings; provincial governments are responsible for issuing flood warnings.

Environment Canada's weather and environmental services

2.5 Environment Canada is Canada's primary source for weather information, with operations that run 24 hours a day, 7 days a week. The Department's mandate for weather services is broadly defined and is derived from the Department of the Environment Act. This act states that the powers, duties, and functions of the Minister of the Environment extend to and include all matters relating to meteorology "over which Parliament has jurisdiction, not by law assigned to any other department, board, or agency of the Government of Canada."

2.6 Environment Canada's weather and environmental products and services include collecting meteorological, hydrological, and climate data; producing five-day public forecasts and marine forecasts; issuing severe weather warnings; and producing seasonal forecasts. In the 2006–07 fiscal year, Environment Canada reported that gross expenditures for its full range of weather and environmental services (for example, weather services, hydrometric services, and air quality monitoring) were around $340 million (about one third of total gross expenditures); these services employed about 2,500 staff. The Department considers severe weather warnings to be the primary focus of weather-related services.

2.7 Environment Canada also provides specialized weather services to the aviation community; meteorological support to national defence; and marine meteorological services, including ice services such as current conditions and forecasts for ice and icebergs. It has cost recovery arrangements with NAV CANADA (NAVCAN), National Defence, and the Canadian Coast Guard related to these weather services. Environment Canada reports that its weather and environmental services as a whole provided the Department with around $55 million in re-spendable revenue in the 2006–07 fiscal year.

2.8 Environment Canada's weather monitoring network forms the foundation for the Department's weather services, with meteorologists using data from this network and from around the world for modelling, producing, and verifying forecasts and severe weather warnings. Weather, hydrological, and climate data is important for many of Environment Canada's programs and the activities of other organizations, such as building and infrastructure design.

2.9 In the mid-1990s, Environment Canada's weather services were delivered through the Atmospheric Environment Service (AES), when the federal government's Program Review significantly reduced the Department's capital funding and its workforce with the closure of 56 weather offices. Following a study on alternative service delivery, in December 1999, the Meteorological Service of Canada (MSC) was established as a service organization within Environment Canada, replacing the AES.

2.10 In March 2003, the MSC began a five-year transition initiative. The transition involved new funding of about $75 million over five years and $5 million a year thereafter. The funding was to be used to consolidate forecast operations into five main storm prediction centres, rationalize the monitoring infrastructure of the MSC, improve products and services, and address important human resource challenges. With the transition funding, it was expected that Canadians would see an improvement in the accuracy and timeliness of forecasts and in the prediction of severe weather events. Environment Canada was to evaluate the transition initiative in 2008.

2.11 In the 2004–05 fiscal year, Environment Canada began a reorganization within the Department. Prior to this time, the MSC was led by an assistant deputy minister, with information technology services managed from within the MSC. As a result of the reorganization, planning and priority setting for weather and environmental services derive from a new results-based management structure that is overseen by the Weather and Environmental Services Board (WESB)—a senior departmental committee consisting of assistant deputy ministers and regional directors general. The Board reports to the Department's Executive Management Council. The Department has centralized, in separate branches, functions that are key to the production and delivery of forecasts and warnings, such as information technology and research and development. A different board (the Departmental Management Services Board) oversees the information technology branch.

2.12 Environment Canada is working on obtaining certification for the weather and environmental services function under ISO (International Standards Organization) 9001. This is expected to result in improved effectiveness and efficiency in the Department's services, including forecast and warning services, and an increased emphasis on quality management. Under ISO 9001, the Department has documented a number of policies, procedures, and guidelines; periodic ISO audits of the Department's adherence to its processes identify non-conformance and areas where corrective action is required.

The process of producing severe weather warnings

2.13 The process of producing weather forecasts and severe weather warnings includes collecting meteorological data, using worldwide data in computerized numerical simulation models to estimate the current state of the atmosphere and predict its future state, and developing forecasts and warnings and delivering them to Canadians (Exhibit 2.2). Canada has been a world leader in producing highly automated regular forecasts. To produce severe weather warnings, however, it must rely more heavily on the direct involvement of meteorologists with real-time meteorological data.

Exhibit 2.2—The production of weather forecasts and weather warnings involves several steps

Four steps in the production of weather forecasts and weather warnings

[text version]

Source: Adapted from Environment Canada documents

2.14 A variety of measures may be used to assess the performance of the various stages of producing weather forecasts and warnings. For monitoring, the measures could include data availability and quality, downtime of assets during severe weather, incidence of repair, financial costs of network assets, and network density. For the production and delivery of severe weather warnings, the measures could include accuracy and timeliness of warnings, warning lead times, false alarms, missed severe weather events, receipt and understanding of warnings, and actions that users may have taken to minimize risks because of the warnings they received.

2.15 In the modern-day world of meteorology, worldwide data on weather is necessary to run computer-generated weather prediction models. Under the coordination of the World Meteorological Organization (WMO), weather data collected by national meteorological services, including satellite data, is shared freely among members of the WMO. Canada belongs to the WMO and thus shares all Environment Canada data and forecasts with other members. Data is also shared within Canada with organizations such as The Weather Network.

2.16 Over the past decade, advances in meteorology and information technology science have had a notable impact on the capacity and effectiveness of weather forecasting. Rapid advances in hardware and software technology have allowed Environment Canada and other weather services around the world using worldwide weather data to improve computer-modelling techniques to the point that they now form the basis for all weather forecasting.

Focus of the audit

2.17 Our audit objective was to determine if Environment Canada's systems and procedures adequately support the delivery of timely and reliable severe weather warnings to Canadians. For the audit scope, we considered severe weather warnings to also include special weather statements and severe weather watches.

2.18 More details on the audit objective, scope, approach, and criteria are in About the Audit at the end of this chapter.

Observations and Recommendations

Environment Canada's weather monitoring network

Life cycle management has not been implemented

2.19 Environment Canada's network of weather observation stations includes radar stations, upper air monitoring stations, marine observation stations, surface weather observation stations, and a network of lightning detection stations. The Department's upper air observations are complemented by observations collected through aircraft-mounted sensors. Environment Canada also operates climate monitoring stations, hydrometric monitoring stations, and air quality monitoring stations. In addition, it collects weather data from other international and Canadian sources, such as provincial governments.

2.20 The Department's surface network consists of automated weather observation stations and a blend of manual and automated climate monitoring stations. Weather observation stations monitor weather variables, such as temperature, wind, and air pressure, on a continuous basis, whereas climate monitoring stations record the measurement of variables, such as minimum and maximum temperature and total precipitation, usually once or twice a day. Forecasters rely on data from weather stations to forecast the weather, including severe weather. Climate data is more important for identifying longer-term trends in climate, such as trends in temperature and precipitation.

Automatic Weather Station, Cape Dorset, Nunavut

Automatic Weather Station, Cape Dorset, Nunavut

Source: Environment Canada

2.21 To ensure a well-functioning network and the collection of high-quality data, and to prevent obsolescence and rust-out of equipment, we expected that Environment Canada was managing its weather monitoring network using a life cycle management approach consistent with its objectives for providing severe weather warnings to Canadians. This type of approach is used to manage the total cost of ownership and performance of assets over their lifetime. It includes assessing data collection requirements and options, investing in network assets based on a combination of financial and non-financial information, operating the assets and regularly monitoring and assessing their condition and performance, and carrying out regular preventive maintenance.

2.22 Environment Canada has made a commitment to manage its monitoring network with a life cycle approach. However, departmental officials believe that allotted funding has been insufficient to implement such an approach. They told us that they have been managing within available funding and are concerned that the monitoring network is "nearing the breaking point." However, they could not support this view due to a lack of performance information. The Department expects the ISO initiative to result in better information in the future.

2.23 Investment in network assets. Life cycle management includes planning for the eventual disposal and replacement of assets. We expected that Environment Canada was replacing and modernizing its network assets on a timely and optimal cost basis, which is important for ensuring the proper functioning of its networks and the continuous flow of high-quality data on weather conditions.

2.24 Environment Canada management reports produced over the last 10 years have noted that the Department's monitoring assets needed urgent, targeted reinvestment. In 2003, as part of its five-year transition initiative, Environment Canada received funding to modernize its network. Among the key reasons for this funding, it cited increasing equipment failures, and inaccurate data creeping into weather forecasting and national climate archives. The Department received about $25 million over five years to invigorate the monitoring capacity of the Meteorological Service of Canada (MSC) by rationalizing its surface weather observation network, enhancing quality assurance and access to data, and enhancing the operation of aircraft-mounted sensors. Rationalizing the network was to include closing and decommissioning lower-priority surface weather observation stations and reallocating resources to maintain the remaining sites to acceptable standards.

2.25 A 2008 Environment Canada evaluation report noted that the Department had not achieved all of the targets related to invigorating its monitoring network. The report states that only 38 percent of the surface weather observation stations identified for closure had been closed, thus preventing the reallocation of resources to the remaining stations. According to the report, closing stations proved to be a sensitive issue at the local level, and land claims negotiations delayed the cleanup and remediation of some sites. The report also cited changes in resource allocation processes and department cutbacks as other factors that affected the rate at which the stations could be closed. The report also brought attention to the fact that the Department could not account for a significant portion of the funds allocated to invigorating the monitoring capacity of the Meteorological Service of Canada.

2.26 Environment Canada's financial data indicates that a large reinvestment/replacement gap may persist. Figures from the 2007–08 fiscal year presented to the Weather and Environmental Services Board show that the capital infrastructure used for monitoring meteorological, hydrological, and climate conditions had a historical value of about $190 million but has depreciated by about 66 percent. Under life cycle management, this information should be integrated with non-financial information to present management with a complete picture of the condition of the Department's monitoring assets. We noted that certain information is not readily available or adequately integrated with this financial profile. This includes information on the performance of monitoring assets relative to data quality and availability; on trends in costs required to maintain existing assets at an acceptable level of performance (for example, unscheduled maintenance and repair costs); and on trends in failure rates (for example, downtime during severe weather events). Given the extent of financial depreciation of the assets, this type of information would show, for example, whether it is becoming more costly to maintain the assets due to unscheduled maintenance and repair, and whether trends indicate an imminent crisis.

2.27 The Department's latest long-term capital plan expired in the 2004–05 fiscal year. We noted that since its expiry, the Department has been working to conform to the Treasury Board's 2007 approach to capital planning under its policy Investment Planning—Assets and Acquired Services, which the Department calls integrated investment planning (IIP). This approach is aimed at ensuring that all investment proposals from across the Department are ranked by priority and risks and fit within the allocated funding levels. At the time of our audit, it was unclear whether this plan would help resolve the investment issues of the monitoring networks. The Department expects to obtain approval of this new capital plan in 2008.

2.28 Network management. We examined operational aspects of three networks that are important for forecasting severe weather: radar, surface, and upper air. Each of these networks is at a different level of implementing a life cycle management approach in its operations.

2.29 Environment Canada considers radar a critical tool for predicting severe weather. The Department reports that in-house development of the existing Doppler radar network began in 1998 and was completed in 2004 for about $40 million in total direct and indirect costs.

Radar Station, Franktown, Ontario

Radar Station, Franktown, Ontario

2.30 When the current radar network (Exhibit 2.3) was completed in 2004, life cycle management expectations were not fully documented. A draft network operations and procedures manual has since been prepared that outlines life cycle management requirements, but not all of the requirements outlined in the manual are in place. For example, the manual requires reports on the performance of the network (such as hours of downtime and percentage of failures due to specific components), as well as reports on maintenance and repairs. Although a national radar information system contains information on individual radar sites, including maintenance logs, the system is not being used to produce these required reports in a consistent or systematic manner.

Exhibit 2.3—Canada's radar network plays an important role in forecasting severe weather

A map of Canada shows the locations of radar sites that make up the national radar network

[text version]

Source: Environment Canada

2.31 Department officials are concerned that the assets of the national radar network are being managed in a reactive manner and that operating and investment decisions are being made based on existing funding levels, not the needs of the network. For the past two fiscal years, management has been asked how to respond to budget reductions, not what is required to properly manage the network with a life cycle approach. In 2007, the Department decided to reduce scheduled radar maintenance visits from six a year to four. This decision was based on the goal of reducing costs, not on an analysis of the network's performance showing that maintenance was greater than required and could be reduced with minimal risks.

2.32 Stable life cycle management of the network, including maintenance, repair, and replacement decisions, is being hindered by several deficiencies. These include a reactive approach to managing the network, the lack of documented life cycle management expectations, and the lack of consistent and systematic tracking and reporting of the network's performance. The absence of proper systematic information hampers good management, particularly when there is high turnover in personnel and loss of corporate memory. We noted that since 2004 there have been four national managers of the radar network.

2.33 Environment Canada manages almost 700 surface weather observation stations and more than 300 reference climate stations. In order to have the information necessary to manage the Department's surface network, a station information system is in place that records data on the network, such as the maintenance history of individual stations. However, the system does not contain data such as failure rates and percentage of failures due to specific components, which would be useful for managing inventory and making informed decisions about preventive maintenance, repair, and replacement. The Department has set standards for the number of preventive maintenance visits required per station per year, but it is currently not able to meet them.

2.34 To ensure a properly functioning network, Environment Canada, using industry standards, estimated in 1998 that 10 percent of the Department's surface weather stations need to be replaced every year. The Department's analysis shows that it has not kept pace with this replacement rate, and as a result there is a growing backlog in the number of stations that need to be replaced—about 50 stations since the 2002–03 fiscal year when they identified about 235 for replacement. The Department also noted that with the use of new electronic components with short life spans at its monitoring stations, the life expectancy of some monitoring equipment may be less than 10 years and, therefore, its replacement rate may be greater than 10 percent per year. It is important that the Department track the performance of its stations over time to ensure that its 10 percent replacement rate is appropriate and that its backlog estimate is accurate.

2.35 The upper air network consists of 31 stations. Twice daily, radiosondes (devices that track variables such as temperature, humidity, wind speeds, and wind direction) are launched at these stations, using helium-filled balloons. The upper air network has implemented several key elements of a life cycle management approach. For example, the performance of the network, including measures such as the number of scheduled flights that occur and height achieved, is tracked and compared with targets on a regular basis. Common problems affecting performance are also identified, tracked, and responded to regularly. Environment Canada recently signed a new long-term contract with an equipment supplier, providing the network with a degree of stability for the supply of parts.

Upper air station and training facility, Stony Plain, Alberta

Upper air station and training facility, Stony Plain, Alberta

Source: Photo courtesy of Lee Suddick, Meteorological Service of Canada

2.36 Recommendation. Environment Canada should document and implement the policies, systems, and procedures necessary to support its commitment to full life cycle management of its monitoring network assets.

The Department's response. Environment Canada agrees with this recommendation and has already taken concrete steps toward this goal. The surface weather, radar, and upper air networks were ISO (International Standards Organization) 9001 certified in 2007, which means that systems and processes are in place to ensure effective management of the networks, as verified by regular and successful internal and external audits. Our ISO certification also requires that performance measures be collected and analyzed to improve effective planning for the maintenance of these assets. The result is a dynamic risked-based life cycle management approach that ensures the health of the networks while respecting resource realities.

Environment Canada will continue to improve its procedures through the Department's new integrated investment planning (IIP) process. The condition of the current assets will first be assessed through the development of an Asset Condition Index (ACI), to be completed before fiscal year 2009–10. Performance measures generated through our ISO processes will provide the basis for ongoing assessment of network and component performance, and will guide ongoing life cycle investment planning through the IIP process.

There is no up-to-date strategy to address challenges in managing the monitoring network

2.37 In addition to the challenge of implementing life cycle management in a context of limited resources, Environment Canada has identified other challenges in managing its monitoring network. It currently has low network density (particularly in the North) and relies extensively on a number of partners for its monitoring data.

2.38 Environment Canada considers the density of its monitoring coverage in Canada to be sparse compared with international standards, especially for monitoring coverage in the North. Increasing interest in development of the North and the federal government's priority to create a greater Canadian presence in the region make the monitoring of weather and environmental conditions in the North more important. On-site monitoring is also essential for forecasting local weather conditions and would be useful for ensuring that forecasters have a complete picture of the weather conditions that may influence the weather in southern Canada. The North is one of the priorities of the Weather and Environmental Services Board, and the Board is considering options for enhanced services in this region.

2.39 Partnerships are important to Environment Canada's monitoring function. The Department relies on a number of organizations for monitoring data, including other federal departments and provincial governments. These relationships are reciprocal, as the monitoring partners rely on, and benefit from, Environment Canada's modelling expertise and forecasting services. Changes in the nature of these relationships can have impacts on the amount and type of data that Environment Canada receives and the expertise and services that it provides. Adequately identifying and managing risks to these relationships is important to ensure that the partnerships succeed.

2.40 For example, NAV CANADA (NAVCAN) is an important partner. Under their current contract ending in 2011, NAVCAN relies on Environment Canada for aviation-forecasting services; for its part, Environment Canada obtains important weather data (that it uses for its aviation and other forecasts) from the some 240 weather observation stations at airports across Canada that it maintains on a contract basis for NAVCAN. Environment Canada also has more than 100 climate monitoring stations located at airports. During our audit, NAVCAN was considering a number of options for its weather observation stations that could change the relationship it has with Environment Canada. Environment Canada officials informed us that they were negotiating a number of issues with NAVCAN about the future of this important partnership.

2.41 As indicated, Environment Canada faces various challenges in managing its monitoring networks, and it has been working on implementing department-wide integrated investment planning (IIP). However, it has not prepared and approved a long-term strategy and capital plan for its monitoring assets, consistent with its corporate strategies and IIP priorities, to deal with these challenges. Such a strategy and plan, and the analysis to support them, would strengthen the Department's ability to assess and communicate the extent of its current and future capacity to meet its challenges and take advantage of opportunities.

2.42 Recommendation. Environment Canada should prepare a fully costed long-term strategy supported by a capital plan for its monitoring networks. The strategy and plan should be consistent with life cycle management and linked to the Department's approved strategies, priorities, and integrated investment planning.

The Department's response. Environment Canada agrees with this recommendation. The Department is developing a fully costed business case that will incorporate a monitoring strategy and long-term capital plan within the Treasury Board's new integrated investment planning (IIP) process that will support Environment Canada's mandate in severe weather and climate monitoring. As outlined in the response to the previous recommendation (paragraph 2.36), the Asset Condition Index (ACI) and ongoing performance measures collected through International Standards Organization (ISO) processes will be fully integrated with and inform this process. Environment Canada will work to identify the ongoing resources to support this strategy in fall/winter 2008–09. From an asset-integrity perspective, investments will be prioritized taking into consideration the condition (as identified by the ACI) and the performance of the asset.

Implementation of the NinJo workstation

The Department is implementing a modern forecasting system for its storm prediction centres

2.43 In 2000, Environment Canada recognized that its meteorological forecasting tools were not standardized across the country, were difficult to maintain, and needed to be replaced. Within this context, the Department began to identify and assess possible options for replacing its forecasting tools.

2.44 We interviewed Environment Canada officials and reviewed documentation to determine if the Department was implementing the NinJo forecaster workstation according to plans and consistent with its objectives. We expected that Environment Canada was following good practices, such as considering user needs, risks, and the strengths and weaknesses of different alternatives, while assessing potential solutions for replacing its forecasting tools. We found that the Department carried out an adequate assessment of various solutions. Its assessment covered forecaster workstations being used by meteorological offices in the United States, the United Kingdom, Germany, and France, as well as the option of building a new system in-house. Based on its assessment, Environment Canada decided to adopt the NinJo workstation.

2.45 NinJo is an advanced computer workstation for forecasters. By manipulating and overlaying various types of data, NinJo allows forecasters to visualize and assess meteorological conditions more efficiently. An international consortium founded in 2000, which consists of the national meteorological services of Germany, the German military, Switzerland, Denmark, and Canada, is developing the workstation. The other members of the consortium have recently implemented NinJo for their meteorologists.

2.46 Environment Canada considers NinJo to be cutting-edge technology that will help forecasters better analyze and forecast severe weather. Given the importance of the storm prediction centres to severe weather forecasting and the need to replace its current forecasting tools, the Department considers the timely and successful implementation of NinJo to be critical to the work of meteorologists at storm prediction centres.

2.47 Environment Canada estimates that NinJo should provide the Department with considerable long-term benefits for the amounts being invested. When Canada joined the consortium in 2003, NinJo was already in development. Canada was asked to pay only a limited amount of the previous development costs, and it agreed to contribute at least two full-time equivalents per year to the development of the workstation in areas where it had recognized expertise. Environment Canada expected Canada's implementation to be more complex, as it estimated that it had a higher degree of automation than its partners to integrate on NinJo.

Project management has weaknesses

2.48 While recognizing that Environment Canada is working within the consortium's project development environment, we expected that the Department would have adequate systems in place to plan for and implement the NinJo workstation in Canada. We interviewed department officials and reviewed documentation to determine if a project plan was prepared that defined the project's Canadian scope and outlined expected outcomes, timelines, and costs. We also examined project documentation to verify that plans were in place to guide implementation of the workstation and that monitoring and reporting were being done.

2.49 We found certain weaknesses in project management that may impede the timely implementation of NinJo in Canada. Although Environment Canada has a draft project charter, it lacks an approved overall project plan. Thus, it is unable to compare actual costs against any original commitments. The Department estimates that since the 2003–04 fiscal year, it has invested close to $11 million in NinJo.

2.50 According to its initial timelines, Environment Canada planned to have NinJo implemented and begin retiring existing software by the end of 2004. In 2006, the Department revised this target to the end of 2007 or early 2008, and it assigned a project manager to the NinJo project, who prepared a plan for deploying the first phase of NinJo into forecast operations. The Department also developed a training plan and online training modules. At the end of our audit, the target for implementation was the fall of 2008, with full retirement of existing software estimated to be three to five years away. In the fall of 2007, an internal NinJo steering committee was established to oversee Environment Canada's involvement in the consortium and implementation of the workstation. Under this guidance, the Department has only recently begun to prepare the portion of a longer-term project plan that covers retirement of existing software, full implementation of NinJo, and the expected costs.

2.51 Recommendation. Environment Canada's NinJo steering committee should ensure that clear timelines and deliverables with assigned accountabilities are established for the successful and timely implementation of the NinJo workstation at Environment Canada. Resource requirements for fully implementing and maintaining NinJo over the longer term should also be clearly identified and tracked on a regular basis.

The Department's response. Environment Canada agrees with this recommendation and has already taken steps toward its implementation. Recognizing that forecaster systems are complex and cannot be bought off the shelf, there has been significant leverage and cost-savings benefits to Canada in working with three other national meteorological services in the design and development of the NinJo workstation. Environment Canada is exercising its project effectiveness through active departmental planning and strong leadership on the International Consortium Steering Committee. Internally, the Department has established its own governance structure to oversee the development and implementation of the project, and has a plan with clear timelines, deliverables, and associated accountabilities that identify December 2008 as the target date for the systems to be operational in the storm prediction centres. Furthermore, the Department has identified and committed to the ongoing resource requirement of 10 FTEs (full-time equivalents) and $300,000 for operations and maintenance for continued development and maintenance of the system. Life cycle management and capital replacement will be addressed through the departmental integrated investment planning (IIP) process. The Department will annually review these requirements and adjust accordingly.

The International Consortium Steering Committee is also committed to developing a long-term NinJo strategy by the fall of 2009. This strategy, which will be discussed and adopted by all consortium partners, will provide long-term direction to Environment Canada's efforts in this area.

Delivery of severe weather warnings

Environment Canada is consulting with selected user groups

2.52 We expected that Environment Canada was making opportunities available to the public and client sectors, such as emergency response officials, to provide feedback on their needs for severe weather warnings, and that such feedback was being used to improve the Department's delivery of severe weather warnings.

2.53 We found that Environment Canada has begun to assess user needs for severe weather warnings. The Department established four national service offices that use a variety of methods, including workshops, meetings, surveys, and case studies, to better understand the needs of key recipients of severe weather warnings. These recipients include, for example, municipal officials responsible for operations and maintenance, as well as users in sectors such as agriculture, energy, marine transportation, and ground search and rescue.

2.54 The Department has also created a National Inquiry Response Team (NIRT) that handles public inquiries on all weather topics. The team categorizes the inquiries it receives and routinely directs them to relevant officials within the Department for their information and/or action. We found that with the exception of the NIRT initiative, the Department could not demonstrate that the feedback from users of severe weather forecasts is being used to improve the delivery of severe weather warnings.

A variety of tools are used to deliver severe weather warnings

2.55 We expected that Environment Canada was considering best practices for the communication tools it uses to deliver severe weather warnings. We found that the Department uses a number of tools to deliver severe weather warnings, including its website (http://www.weatheroffice.gc.ca/), Weatheradio, automated telephone weather information service, and fee-based one-on-one consultation services. The Department also counts on the broadcast media to deliver its severe weather warnings.

2.56 In the past year, Environment Canada has also begun taking advantage of newer technologies by providing forecasts and warnings through the Government of Canada Wireless Portal and through Really Simple Syndication (RSS)—an Internet-based tool for distributing frequently updated information.

2.57 Environment Canada's weatheroffice.gc.ca website is the federal government's most frequently visited website. This website is an example of a pull technology where users request information such as severe weather warnings from the website. Currently, Environment Canada's only push technology, where severe weather warnings are proactively sent to users without their having to request the information, is Weatheradio. According to a 2007 departmental public survey, less than 10 percent of respondents had used Weatheradio.

2.58 We noted that the Department does not have a strategy for delivering severe weather warnings, which would include a mix of push and pull technologies that it wishes to use both now and in the future. Without this type of strategy, it has not set priorities among its tools or made commitments to their longer-term funding.

There is no national public alerting system

2.59 A public alerting system results in severe weather warnings (as well as other warnings such as terrorist threats or environmental emergencies) being broadly communicated over all radio and television stations and sent to mobile devices such as cellular phones. Such a system does not currently exist across Canada. Environment Canada has supported efforts toward mandatory broadcasting of warnings. It is also cooperating with other federal departments as well as provincial and territorial governments and industry stakeholders toward creating a national public alerting system for all types of emergencies. Its role would be to provide severe weather warnings to an eventual system.

2.60 Recommendation. At the same time as it works with the lead federal department and other government departments in the creation of a national public alerting system, Environment Canada should develop a costed strategy to improve the effectiveness of its own "push" technologies in order to increase the reach and effectiveness of its weather warnings.

The Department's response. Environment Canada agrees with this recommendation and recognizes that "push" technology is critical in "short-fused" dangerous situations—such as severe thunderstorms and tornadoes—in ensuring that people receive a warning in time to take action to protect themselves. Integral to the Department business case being completed this fall/winter 2008–09 will be a comprehensive service delivery strategy that will include measures to improve the effectiveness of the existing Weatheradio system, as well as leverage new wireless messaging technologies to issue warnings to Canadians in a timely fashion. Based on the decisions from this business case, the Department would be better positioned to continue to work with other federal departments, the provinces and territories, as well as industry stakeholders on the creation of a national public alerting system. In particular, Environment Canada would work with Public Safety Canada, who has the lead on the creation of a national public alerting system.

Verification of severe weather warnings

2.61 Environment Canada reports that it issues more than 10,000 severe weather warnings across Canada per year. These warnings are considered the most important product among weather services, as they contribute directly to the Department's objective of enabling the public to protect their safety and well-being. The Department has made commitments to improve the timeliness and accuracy of its warnings. Verifying the accuracy and timeliness of these warnings would allow Environment Canada to better understand their quality and to determine whether its performance is improving over time. The Department could also use the results of verification to improve its science and forecasting process and therefore the performance of its warning program. Publicly reporting on the quality of its severe weather warnings would strengthen Environment Canada's accountability and may increase users' confidence in the warnings they receive.

2.62 Verifying severe weather warnings can be challenging for a variety of reasons. For example, warnings may be issued for specific forecast regions lacking surface weather observation stations that could provide verification data. Severe weather events, particularly severe summer storms, can be small-scale and highly localized, appearing and dissipating quickly.

There is no national verification approach or program

2.63 We expected that Environment Canada was verifying the quality of its severe weather warnings. Based on interviews with department officials responsible for forecasts, warnings, and performance measurement; interviews with regional forecasters; and a review of verification reports, we found that there is no national program or approach for verifying severe weather warnings and assessing Environment Canada's overall performance in delivering them. This information is important as it would help Environment Canada know how good a job it is doing across Canada, whether its current performance meets its own and users' expectations, and whether it needs to improve its services and increase investment in areas such as monitoring and delivery.

2.64 Environment Canada has established a national verification program for its public and aviation forecasts and has made some effort to develop a national program for verifying severe weather warnings. The Department produced a draft report in 2006 analyzing the accuracy and timeliness of summer severe weather watches and warnings across Canada from 2001 to 2003. However, the report was never finalized, as resources were redirected from this initiative to other priorities in the Department.

2.65 In three of the four regions we examined, we found some recent verification of severe weather warnings. On an ad hoc basis, the verifications checked a variety of indicators, such as warning lead-time, forecast accuracy, false alarms, and probability of detection of events. While the approaches used were not consistent among regions, these efforts, along with the Department's national program for verifying public forecasts, could be used as the basis for a national program for verifying severe weather warnings.

2.66 Recommendation. Environment Canada should establish and implement a national program for verifying the quality of severe weather warnings throughout the year.

The Department's response. Environment Canada agrees with the spirit of this recommendation. A full comprehensive verification system would be expensive and likely not cost-effective. Instead, the Department is in the process of finalizing the implementation of a Quality Management System, registered to ISO 9001, which provides the building blocks for a holistic, realistic, and affordable approach regarding program and product verification and quality measurement. The Department is committed to implementing an appropriate mix of measures to understand the performance of its warnings, and there are already efforts under way to improve the consistency of the scientific verification of its warnings. However, the cost, complexity, and scientific challenges of developing and implementing such a system warrants analysis in the context of all other available means of performance measurement that may prove more effective and affordable.

This analysis will be undertaken in fall/winter 2008–09 and will result in the development of costed options to implement a comprehensive performance monitoring and measurement system. This system will have the appropriate mix of scientific verification and information gathering, such as the post-event studies conducted through the Warning Preparedness Meteorologist program (as described in our response to the recommendation in paragraph 2.74) and through Public Opinion Research—both representing critical methods to determine the effectiveness of the warning program.

Environment Canada is taking steps to determine how well severe weather warnings inform Canadians

2.67 Ensuring that severe weather warnings are delivered in a timely fashion and sufficiently understood by users to allow them to take appropriate action can be challenging. For example, when a forecaster issues a warning of 100 kilometres-per-hour winds, certain questions arise; for example: Is the warning reaching those it should and do recipients of the warning understand the implications of winds that strong? Should the Department be providing more information on potential impacts, such as the risk of trees losing branches and roofs being damaged?

2.68 To improve its capacity to answer these types of questions, Environment Canada is taking steps to determine how effectively severe weather warnings are informing Canadians. It has conducted national public opinion surveys that capture data on a variety of topics, including public views on whether summer and winter severe weather warnings are clear and provide sufficient notice to prepare for approaching storms.

2.69 Storm surveys. After certain significant storm events, teams of warning preparedness meteorologists and outreach officers have conducted storm surveys. The purpose of these surveys is to confirm the nature of the event (for example, whether it was in fact a tornado as predicted); they may also identify issues such as the effectiveness of communications between Environment Canada and the media and emergency measures organizations. In addition, reports on severe weather events assess issues such as the performance of numeric model predictions and forecasts, together with the warnings issued by storm prediction centres. Although the Department currently conducts storm surveys and case studies on an ad hoc basis, it is considering developing a Forensic Storm Investigation program that would add more rigour and consistency.

2.70 Case study in B.C. In 2007, the Department carried out a case study that focused on how well it informed key people (specifically, emergency measures organizations and municipal officials) and how effectively the local emergency management officials received and understood the severe weather warnings. The study reviewed three severe weather events that occurred in 2007 in the British Columbia interior. It concluded that the storm prediction centre had done little to inform end users of the potential consequences or impacts of the impending storms and that this type of information could have led to more informed risk reduction strategies by local officials.

2.71 The study also concluded that although the timeliness and accuracy of warnings are important, the clarity of the message is also important. If users do not understand the potential impacts and hazards associated with the forecasted severe weather event, they may not take the necessary precautions to protect themselves and minimize risks to persons and property. The Department initiated a similar case study in the spring of 2008, focusing on severe winter storm events in Quebec.

2.72 Warning preparedness meteorologists. Properly informing Canadians of impending severe weather conditions can help minimize the impacts of these events. To enhance this type of information, Environment Canada established 11 positions for warning preparedness meteorologists. They help communities prepare for and respond to severe weather by establishing a rapport and liaising with the media and emergency measures organizations at the provincial and local levels before, during, and after severe weather events. To facilitate access to these meteorologists, Environment Canada established a dedicated telephone line for the media and will have additional meteorologists assist during severe weather events or on-call if there is a high probability of a severe weather event.

2.73 The Department has received a lot of positive feedback on the services provided by warning preparedness meteorologists; at the same time, it has identified instances of some concerns about their accessibility, expressed in the post-storm surveys and case studies that it conducted. The Department has not set clear performance goals for their work, such as level of responsiveness. Thus, despite the numerous activities carried out by these meteorologists, it is difficult for the Weather and Environmental Services Board to know whether the Department's and users' expectations are being met.

2.74 Recommendation. Environment Canada should regularly assess the effectiveness of severe weather warnings from a user's perspective, especially the effectiveness of the methods of delivery to users and how well the warnings are understood by key users and the public.

The Department's response. Environment Canada agrees with this recommendation and has already undertaken measures toward meeting this goal. The Department acknowledges that the full measure of the effectiveness of its warning programs can only be understood with an appropriate mix of monitoring and measurement tools (see our response to the recommendation in paragraph 2.66), including Public Opinion Research and case studies of what action people have taken in response to warnings.

The Warning Preparedness Meteorologist program consists of highly specialized communication and outreach meteorologists who focus on providing emergency prevention support to Emergency Measures Organizations (EMOs) across Canada. This efficient and flexible program delivers effective information and advice during emergency situations. EMOs and the media have indicated on many occasions that they are satisfied with the support they are getting from Environment Canada.

Additionally, Public Opinion Research is used annually as an efficient way to assess the effectiveness of current products, services, and programs in meeting the needs of both specific clients and the public, as well as in contributing to what the requirements may be for the development of different products and services.

Environment Canada is currently conducting a standards review of the criteria used to issue warnings from a user perspective and of how impact statements could be included with the warnings to improve the action taken by citizens in response. Recommendations are expected from this analysis by spring 2009.

Strategic direction

2.75 Given the changes over the last 10 years that have affected the weather services of Environment Canada (see paragraphs 2.9 to 2.12), we interviewed Department officials and reviewed relevant documentation to determine whether Environment Canada's weather and environmental services planning provides strategic direction and supports the current and future delivery of severe weather warnings to Canadians. We expected that Environment Canada had adequately defined its mandate and role for providing severe weather services, had put in place a strategic planning process that enables the allocation of resources consistent with short- and long-term risks and priorities, and had developed statements for measurable results to guide department activities.

A new results structure is in place

2.76 The Weather and Environmental Services Board has defined its mandate as providing meteorological and environmental warnings, forecasts, information, and services enabling Canadians to make effective decisions regarding their health and safety, the economy, and the environment. In support of this mandate, Environment Canada has identified 48 program activity areas ranging from monitoring and forecasting meteorological and environmental conditions to delivering forecasts and warnings to users.

2.77 The descriptions of these activity areas and related expectations for results make it reasonably clear what Environment Canada is attempting to achieve with severe weather warnings. However, the Department has not yet established results-based service level standards that would allow it to better measure its performance related to weather and environmental services, including severe weather warnings, although it has indicated its intention to do so.

The long-term perspective has eroded

2.78 In 1999, the Meteorological Service of Canada produced a five-year business plan. The 2003 transition initiative, ending in 2008, also provided specific strategic direction for the organization. The Department has continued to rely on the relevant principles and general direction of those earlier documents for strategic guidance. However, since 2003, the organization's roles and responsibilities for providing weather services have changed considerably. In addition, a number of new risks and challenges have emerged.

2.79 The scope of Environment Canada's weather and environmental services and the results it wants to achieve for monitoring weather conditions and informing Canadians of severe weather events are broad and ambitious. At the same time, as we note in this chapter, the Department is facing a number of significant challenges to maintain and improve its weather services related to severe weather warnings, including the following:

  • establishing a national system for verifying the effectiveness of its severe weather forecasts;
  • implementing the reorganization of weather-related functions, including the centralization of information technology services and research and development, and establishing clear lines of accountability;
  • delivering weather services with the Department's current capacity—that is, addressing the need for reinvesting in its weather and climate monitoring networks, developing alternatives for delivery tools, and enhancing its ability to provide services to Canada's North;
  • implementing life cycle management of its monitoring networks in a context of limited resources;
  • addressing the risks related to the Department's current strategy of relying on opportunities to partner with others—for example, the potential withdrawal of one of its key partners;
  • implementing its modern workstation for forecasters; and
  • delivering clear severe weather warnings efficiently and effectively using an appropriate mix of push and pull technologies.

2.80 The collection of weather and climate data and the provision of severe weather forecasts are critical to Canada as a member of the international community and as a G-8 economy. The Weather and Environmental Services Board has identified some of its risks and challenges and has set program priorities for the 2008–09 fiscal year. Procedures are also being established to identify short-term priorities for the fiscal year in areas such as capital planning and information technology. In addition, we noted that the Meteorological Service of Canada is taking steps to develop a vision and a renewed strategic direction for Environment Canada's weather and environmental services. Nevertheless, the Department currently has no up-to-date documented strategy to communicate how it will address the longer-term challenges, provide direction for charting a clear course of action, set focused priorities and defined accountabilities under the new organizational arrangements, and identify resource requirements and their potential sources. Nor has it conducted a thorough risk assessment or analyzed the implications for the whole Department and the country of its current actions related to the challenges it faces in this area. If its recent visioning and planning efforts for providing weather services result in an approved and documented strategy, the Department will be able to demonstrate accountability and support its requests for funding while enhancing public confidence in its ability to deliver high-quality severe weather warnings to Canadians in the future.

2.81 Recommendation. Environment Canada should establish and document an updated long-term strategy for its weather and environmental services. The strategy should address identified long-term risks, link to departmental strategies, and reflect verification results related to the usefulness, timeliness, and accuracy of severe weather warnings across Canada. The strategy should also include clear expectations for results and be updated on a regular basis to reflect changing priorities, evolving partnerships, and technological advances.

The Department's response. Environment Canada fully agrees with this recommendation. Long-term strategic plans were developed and implemented in 1999 and 2003. In December 2007, Environment Canada reinitiated a process to renew its vision and strategic direction for weather and environmental service delivery over the next decade by identifying key drivers, challenges, risks, and opportunities. This initiative will lead to a new strategic direction and a business case in fall/winter 2008–09 to support the requirement for new investments.

Conclusion

2.82 Environment Canada's systems and procedures do not adequately support the delivery of timely and accurate severe weather warnings to Canadians. Although it currently delivers a large number of severe weather warnings, it lacks the information necessary to know the condition of its monitoring assets and whether the warnings are complete, timely, or accurate. As a result, Environment Canada is unable to tell Canadians how well they are being served or what improvements are needed to effectively provide this service in the future.

2.83 We examined operational aspects of three networks that are important for forecasting severe weather: radar, surface, and upper air. Although Environment Canada has made a commitment to a life cycle management approach for its monitoring networks, it has not succeeded in implementing this approach.

2.84 The Department's selection of NinJo, a new advanced forecaster workstation that is shared with several European countries and designed to replace the tools currently being used, followed an adequate process. The Department expects that, when implemented, the workstation will provide significant long-term benefits for the amounts being invested.

2.85 In addition to relying on the broadcast media, the Department uses a number of tools to deliver severe weather warnings to Canadians, such as the Internet and Weatheradio—a tool that can be used to automatically alert users of severe weather warnings. It has also begun exploring the use of newer technologies, such as the Government of Canada Wireless Portal. However, it has not defined where it intends to focus its delivery efforts.

2.86 There is no national program or approach for verifying the quality of severe weather warnings and assessing Environment Canada's performance for delivering them. This information is important, as it would help the Department know whether its current performance for delivering severe weather warnings across Canada is reasonable, and whether it needs to improve its services and increase investment levels in areas such as monitoring and delivery.

2.87 Environment Canada faces a number of fundamental risks and challenges in providing weather services, including those related to severe weather, but its planning function does not provide adequate support, as it has not established up-to-date long-term strategic and capital plans for guiding its decisions.

About the Audit

Objective

Our audit objective was to determine if Environment Canada's systems and procedures adequately support the delivery of timely and reliable severe weather warnings to Canadians. This included determining if Environment Canada is

  • using a planning function that provides strategic direction for the current and future delivery of severe weather warnings to Canadians;
  • managing the physical assets of its weather observation network using a life cycle management approach, consistent with its objectives for providing severe weather warnings to Canadians;
  • implementing the NinJo forecaster workstation according to plans and consistent with its objectives;
  • verifying the quality of its severe weather warnings and using this information to improve forecasting of severe weather events; and
  • making opportunities available to users to provide feedback on their needs for severe weather warnings, and considering such feedback and best practices for delivery tools used in all regions of Canada to improve delivery of severe weather warnings.

Scope and approach

We examined Environment Canada's management of its weather observation network, the implementation of a new, advanced forecaster workstation called NinJo, and the delivery and verification of severe weather warnings. We also examined the Department's planning process that supports the current and future delivery of severe weather warnings to Canadians. The focus of our audit was the delivery of severe weather warnings to Canadians; we did not examine Environment Canada's public forecasting program or other weather and environmental services, such as those related to air quality, climate, or hydrometric services.

Our audit approach consisted of reviewing and analyzing key documents, files, and information systems, and interviewing managers and department leads within the weather and environmental services program of Environment Canada (including those responsible for weather monitoring, forecasting, performance measurement, and the delivery of severe weather warnings). We visited four of Environment Canada's five regions (Pacific-Yukon, Prairie-Northern, Ontario, and Quebec), met with forecasters and technicians, and visited two storm prediction centres. We also met with officials from the World Meteorological Organization, the United States National Weather Service, the German Meteorological Service, and MeteoSwiss.

Criteria

Listed below are the criteria that were used to conduct this audit and their sources.

Criteria Sources
Strategic Planning

For strategic planning, we expected that Environment Canada had

  • adequately defined its mandate and role for providing severe weather services,
  • developed measurable results statements to guide departmental activities, and
  • developed a strategic planning process that enables the allocation of resources consistent with risks and priorities.

Treasury Board of Canada Secretariat, Management Accountability Framework (elements/indicators related to governance structure and effective planning function)

Monitoring

For its monitoring network, we expected that Environment Canada had a life cycle approach in place for managing its monitoring network.

  • Treasury Board of Canada Secretariat, Policy Framework for the Management of Assets and Acquired Services, Section 3
  • Treasury Board of Canada Secretariat, Policy on Investment Planning—Assets and Acquired Services, Section 6
  • Treasury Board of Canada Secretariat, Policy on Management of Materiel, Section 6
  • Treasury Board of Canada Secretariat, Policy on Long-term Capital Plans, Section 5
NinJo

For the NinJo workstation, we expected that Environment Canada had selected the workstation based on adequate analysis, and put systems in place to adequately plan for and implement the workstation.

  • Treasury Board of Canada Secretariat, Policy on Project Management (Policy Requirements)
  • Treasury Board of Canada Secretariat, Policy on the Management of Projects, sections 3, 4, and 6
Verification

For the verification of severe weather warnings, we expected that Environment Canada was

  • delivering severe weather warnings in a timely and accurate manner according to its own standards,
  • verifying the quality of its severe weather warnings, and
  • using information from the verification of the quality of severe weather warnings to improve forecasting of severe weather events.
  • World Meteorological Organization, Guidelines on Performance Assessment of Public Weather Services, Section 4
  • Treasury Board of Canada Secretariat, Preparing and Using Results-Based Management and Accountability Frameworks, Section 4 and Results for Canadians: A Management Framework for the Government of Canada
Delivery

For the delivery of severe weather warnings, we expected that Environment Canada was

  • making opportunities available to the public and client sectors to provide feedback on their needs for severe weather warnings, and considering such feedback in plans, reviews, or evaluations to help improve the delivery of severe weather warnings; and
  • considering user needs and best practices for the delivery tools used in all regions of Canada.
  • Treasury Board of Canada Secretariat, Policy on Communications of the Government of Canada (see Policy Statement and Requirements)
  • Treasury Board of Canada Secretariat, Management Accountability Framework (elements/indicators related to external service delivery)

Audit work completed

Audit work for this chapter was substantially completed on 30 May 2008.

Audit team

Principal: Paul Morse
Director: Jim McKenzie

Wilson Ford
Averil James
Marc-André Lafrance
Melissa Miller
Jay Storfer
Mary Anne Strong

For information, please contact Communications at 613-995-3708 or 1-888-761-5953 (toll-free).

Appendix—List of recommendations

The following is a list of recommendations found in Chapter 2. The number in front of the recommendation indicates the paragraph number where it appears in the chapter. The numbers in parentheses indicate the paragraph numbers where the topic is discussed.

Recommendation Response
Environment Canada's weather monitoring network

2.36 Environment Canada should document and implement the policies, systems, and procedures necessary to support its commitment to full life cycle management of its monitoring network assets. (2.19–2.35)

Environment Canada agrees with this recommendation and has already taken concrete steps toward this goal. The surface weather, radar, and upper air networks were ISO (International Standards Organization) 9001 certified in 2007, which means that systems and processes are in place to ensure effective management of the networks, as verified by regular and successful internal and external audits. Our ISO certification also requires that performance measures be collected and analyzed to improve effective planning for the maintenance of these assets. The result is a dynamic risked-based life cycle management approach that ensures the health of the networks while respecting resource realities.

Environment Canada will continue to improve its procedures through the Department's new integrated investment planning (IIP) process. The condition of the current assets will first be assessed through the development of an Asset Condition Index (ACI), to be completed before fiscal year 2009–10. Performance measures generated through our ISO processes will provide the basis for ongoing assessment of network and component performance, and will guide ongoing life cycle investment planning through the IIP process.

2.42 Environment Canada should prepare a fully costed long-term strategy supported by a capital plan for its monitoring networks. The strategy and plan should be consistent with life cycle management and linked to the Department's approved strategies, priorities, and integrated investment planning. (2.37–2.41)

Environment Canada agrees with this recommendation. The Department is developing a fully costed business case that will incorporate a monitoring strategy and long-term capital plan within the Treasury Board's new integrated investment planning (IIP) process that will support Environment Canada's mandate in severe weather and climate monitoring. As outlined in the response to the previous recommendation (paragraph 2.36), the Asset Condition Index (ACI) and ongoing performance measures collected through International Standards Organization (ISO) processes will be fully integrated with and inform this process. Environment Canada will work to identify the ongoing resources to support this strategy in fall/winter 2008–09. From an asset-integrity perspective, investments will be prioritized taking into consideration the condition (as identified by the ACI) and the performance of the asset.

Implementation of the NinJo workstation

2.51 Environment Canada's NinJo steering committee should ensure that clear timelines and deliverables with assigned accountabilities are established for the successful and timely implementation of the NinJo workstation at Environment Canada. Resource requirements for fully implementing and maintaining NinJo over the longer term should also be clearly identified and tracked on a regular basis. (2.43–2.50)

Environment Canada agrees with this recommendation and has already taken steps toward its implementation. Recognizing that forecaster systems are complex and cannot be bought off the shelf, there has been significant leverage and cost-savings benefits to Canada in working with three other national meteorological services in the design and development of the NinJo workstation. Environment Canada is exercising its project effectiveness through active departmental planning and strong leadership on the International Consortium Steering Committee. Internally, the Department has established its own governance structure to oversee the development and implementation of the project, and has a plan with clear timelines, deliverables, and associated accountabilities that identify December 2008 as the target date for the systems to be operational in the storm prediction centres. Furthermore, the Department has identified and committed to the ongoing resource requirement of 10 FTEs (full-time equivalents) and $300,000 for operations and maintenance for continued development and maintenance of the system. Life cycle management and capital replacement will be addressed through the departmental integrated investment planning (IIP) process. The Department will annually review these requirements and adjust accordingly.

The International Consortium Steering Committee is also committed to developing a long-term NinJo strategy by the fall of 2009. This strategy, which will be discussed and adopted by all consortium partners, will provide long-term direction to Environment Canada's efforts in this area.

Delivery of severe weather warnings

2.60 At the same time as it works with the lead federal department and other government departments in the creation of a national public alerting system, Environment Canada should develop a costed strategy to improve the effectiveness of its own "push" technologies in order to increase the reach and effectiveness of its weather warnings. (2.52–2.59)

Environment Canada agrees with this recommendation and recognizes that "push" technology is critical in "short-fused" dangerous situations—such as severe thunderstorms and tornadoes—in ensuring that people receive a warning in time to take action to protect themselves. Integral to the Department business case being completed this fall/winter 2008–09 will be a comprehensive service delivery strategy that will include measures to improve the effectiveness of the existing Weatheradio system, as well as leverage new wireless messaging technologies to issue warnings to Canadians in a timely fashion. Based on the decisions from this business case, the Department would be better positioned to continue to work with other federal departments, the provinces and territories, as well as with industry stakeholders on the creation of a national public alerting system. In particular, Environment Canada would work with Public Safety Canada, who has the lead on the creation of a national public alerting system.

Verification of severe weather warnings

2.66 Environment Canada should establish and implement a national program for verifying the quality of severe weather warnings throughout the year. (2.61–2.65)

Environment Canada agrees with the spirit of this recommendation. A full comprehensive verification system would be expensive and likely not cost-effective. Instead, the Department is in the process of finalizing the implementation of a Quality Management System, registered to ISO 9001, which provides the building blocks for a holistic, realistic, and affordable approach regarding program and product verification and quality measurement. The Department is committed to implementing an appropriate mix of measures to understand the performance of its warnings, and there are already efforts under way to improve the consistency of the scientific verification of its warnings. However, the cost, complexity, and scientific challenges of developing and implementing such a system warrants analysis in the context of all other available means of performance measurement that may prove more effective and affordable.

This analysis will be undertaken in fall/winter 2008–09 and will result in the development of costed options to implement a comprehensive performance monitoring and measurement system. This system will have the appropriate mix of scientific verification and information gathering, such as the post-event studies conducted through the Warning Preparedness Meteorologist program (as described in our response to the recommendation in paragraph 2.74) and through Public Opinion Research—both representing critical methods to determine the effectiveness of the warning program.

2.74 Environment Canada should regularly assess the effectiveness of severe weather warnings from a user's perspective, especially the effectiveness of the methods of delivery to users and how well the warnings are understood by key users and the public. (2.67–2.73)

Environment Canada agrees with this recommendation and has already undertaken measures toward meeting this goal. The Department acknowledges that the full measure of the effectiveness of its warning programs can only be understood with an appropriate mix of monitoring and measurement tools (see our response to the recommendation in paragraph 2.66), including Public Opinion Research and case studies of what action people have taken in response to warnings.

The Warning Preparedness Meteorologist program consists of highly specialized communication and outreach meteorologists who focus on providing emergency prevention support to Emergency Measures Organizations (EMOs) across Canada. This efficient and flexible program delivers effective information and advice during emergency situations. EMOs and the media have indicated on many occasions that they are satisfied with the support they are getting from Environment Canada.

Additionally, Public Opinion Research is used annually as an efficient way to assess the effectiveness of current products, services, and programs in meeting the needs of both specific clients and the public, as well as in contributing to what the requirements may be for the development of different products and services.

Environment Canada is currently conducting a standards review of the criteria used to issue warnings from a user perspective and of how impact statements could be included with the warnings to improve the action taken by citizens in response. Recommendations are expected from this analysis by spring 2009.

Strategic direction

2.81 Environment Canada should establish and document an updated long-term strategy for its weather and environmental services. The strategy should address identified long-term risks, link to departmental strategies, and reflect verification results related to the usefulness, timeliness, and accuracy of severe weather warnings across Canada. The strategy should also include clear expectations for results and be updated on a regular basis to reflect changing priorities, evolving partnerships, and technological advances. (2.75–2.80)

Environment Canada fully agrees with this recommendation. Long-term strategic plans were developed and implemented in 1999 and 2003. In December 2007, Environment Canada reinitiated a process to renew its vision and strategic direction for weather and environmental service delivery over the next decade by identifying key drivers, challenges, risks, and opportunities. This initiative will lead to a new strategic direction and a business case in fall/winter 2008–09 to support the requirement for new investments.

 


Definition:

Push and pull technologies—Delivery tools that broadcast or send messages and information, such as severe weather warnings, to users.

Push technologies send (push) information directly to users without their having to request it. An example is Environment Canada's Weatheradio, where the user purchases a special receiver that can be programmed to alert them when a warning is issued for their area.

Pull technologies rely on users to request information. Examples are Environment Canada's weatheroffice.gc.ca website and its automated telephone weather information service. (Return)

 

PDF Versions

To access the Portable Document Format (PDF) version you must have a PDF reader installed. If you do not already have such a reader, there are numerous PDF readers available for free download or for purchase on the Internet: