Office of the Auditor General of CanadaOAG reports published in the past are available through Publications.gc.ca.
2006 November Report of the Auditor General of Canada Chapter 3—Large Information Technology Projects
2006 November Report of the Auditor General of Canada
Chapter 3—Large Information Technology Projects
What we examined
The many large information technology projects now under way across the federal government are no longer only about introducing new computer hardware, software, or systems. They are meant to help departments change the way they do business—for example, by introducing new processes and modernizing work practices.
We examined a sample of seven large IT projects from four perspectives:
- Governance. Did the processes used by the government to approve and manage large IT projects increase the project's likelihood of success?
- Business case. Did the department or agency proposing the project clearly define the business needs it expected the project to meet?
- Organizational capacity. Did the department have people with the needed skills and experience to manage a large IT project and did the organization have the ability to use all of a system's capabilities to improve the way it does business?
- Project management. Did the department follow accepted best practices in managing the project?
We also attempted to examine the role the Treasury Board Secretariat played in the challenge and oversight of these large information technology projects. However, we could not audit this role because we were denied access to most of the information and analysis it collects and prepares. The government has effectively imposed a limitation on the scope of the Auditor General's examination.
Why it's important
Few departments today can deliver their business without the support of their IT. Governments must deliver successful IT projects if they are to provide their services to the public economically, efficiently, and effectively. Large IT projects are becoming more complex than ever and often involve a growing number of players across government.
In the last three years, the federal government has approved funding of $8.7 billion dollars for new business projects with significant use of IT. Despite their importance to departments, large IT projects have had a history of overspending, delays, performance shortfalls, and abandonment after major investments. Understanding why some projects have succeeded while others have failed would better position the government to plan and manage IT projects for success, maximizing the benefits of its investment in information technology by giving departments effective tools for delivering services.
What we found
- Only two of the seven projects we looked at: the 2006 Census Online and My Account, My Business Account projects met all of our audit criteria for well-managed projects.
- Overall, the government has made limited progress since our last audit of IT projects in 1997. Although since 1998 the Treasury Board Secretariat has established a framework of best practices for managing IT projects, many of the problems we cited in past reports have persisted.
- The quality of governance varied widely from project to project. In four of the seven projects we found that governance responsibilities were not carried out adequately because key issues that impacted project performance were either not reported or not resolved.
- Five of the seven projects we looked at were allowed to proceed with a business case that was incomplete or out-of-date or contained information that could not be supported.
- Four of the projects were undertaken by departments that lacked the appropriate skills and experience to manage the projects or the capacity to use the system to improve the way they deliver their programs.
- Depending on the project, the quality of project management ranged from good to seriously flawed. In two cases, poor project management led to long delays and large cost overruns.
The government has responded. The Treasury Board Secretariat, on behalf of the government and the federal organizations we audited, agrees with all of our recommendations. In its responses, the Secretariat describes the actions they will take to address the recommendations.
The nature of large information technology projects
3.1 In the last three years, the Treasury Board has approved and the government has undertaken 88 information technology (IT) projects, with costs that totalled $7.1 billion. In addition to these Treasury Board-approved investments, we noted that the government has invested $1.2 billion in the Canada Health Infoway, and the Canada Revenue Agency has invested $431 million in its own information systems. Over the last three years, the government's total investment has been about $8.7 billion.
3.2 IT projects are critical to improving the quality and efficiency of public services. This is commonly known as IT-enabled business change. Nearly all IT projects today have some elements of business change associated with them that go beyond introducing new computer hardware, software, or systems. These elements include introducing new processes, modernizing working practices, and offering new services to the customer. New customer services often involve replacing the manual processing of high-volume routine claims or operations with automated or electronic processing (for example, submitting records of employment online).
3.3 During the last six to nine years, fundamental changes occurred in large IT projects. The federal government has recognized that there are complex IT issues that cross departmental boundaries. That recognition has increasingly resulted in horizontal initiatives (for example, Government On-Line and Secure Channel). The efficient use of resources in delivering these types of IT projects depends on integrated decision making across federal organizations. However, when horizontal management is inadequate, the government is less likely to achieve the business transformations that are expected from these projects.
3.4 Compared with other disciplines, the management of IT projects is relatively new, and organizations are still finding their way with it. Therefore, any organization that invests in IT to support its business must be cautious. Large IT projects are inherently complex, expensive, and risky, and they usually involve long planning and development times.
3.5 Over the last 10 years, other governments and the private sector have studied the challenges facing large IT projects and the reasons why so many have failed. In our 1995 Report, Chapter 12, Systems under Development, we quoted the Standish Group's 1994 research study. The study noted that "a staggering 31.1 percent of projects will be cancelled before they ever get completed," and "52.7 percent of projects will cost 189 percent of their original estimates." In its 2000 report, the Standish Group noted that a significant trend had emerged in IT projects: most new projects fit within its "Recipe for Success," which limits the size of projects to six months and six people. By 2000, the average cost of a project had fallen to less than half the average cost of a 1994 project.
3.6 In 2004, the Standish survey indicated that the success rate for projects had improved somewhat since 1994. It observed that
This year's results show that 29 percent of all projects succeeded (delivered on time, on budget, with required features and functions); 53 percent are challenged (late, over budget and/or with less than the required features and functions); and 18 percent have failed (cancelled prior to completion or delivered and never used).
Focus of the audit
3.7 Our audit of the development of large IT projects had three objectives:
- Determine whether the process for approving and managing projects was sufficiently rigorous.
- Determine whether the business cases we examined clearly identified the expected results of the projects and how they would help to meet the business needs of the government and the sponsoring department at a reasonable cost.
- Determine whether the departments and agencies that sponsored the IT projects in our sample are managing large IT projects according to the criteria for governance, business cases, organizational capacity, and project management.
3.8 During this audit, we attempted to determine whether the Treasury Board Secretariat has adequately fulfilled its challenge and oversight responsibilities for the large IT projects in our sample. After consulting the Privy Council Office, the Treasury Board Secretariat has denied us access to most of the information and analysis it collects and prepares, citing Cabinet confidence of a type that cannot be disclosed to us. We met with the Privy Council Office in an attempt to resolve the issue but were unsuccessful. As a result, we are unable to conclude whether the Treasury Board Secretariat has carried out a proper challenge and oversight role for these projects.
3.9 The Auditor General's reporting obligations are set out in section 7 of the Auditor General Act. Among other things, we are required to report to Parliament when we observe that the government has not shown due regard for economy when it makes significant expenditures. Under section 13 of the Act, we have a right to the information and explanations we need to examine these significant expenditures. The extent to which the Treasury Board Secretariat has fulfilled its role in analyzing significant expenditures is an important element of the Auditor General's examination.
3.10 Since we have been denied access to most of the documents we need to complete this examination, we cannot complete this audit and provide Parliament with the assurance that the government has in fact shown due regard for economy in this case. Furthermore, we are required to report to Parliament when we do not receive all of the information and explanations we require. By denying us access to the required information, the government has effectively imposed a limitation on the scope of the Auditor General's examination that may diminish Parliament's fundamental role of holding the government to account. This could be viewed as an infringement of House of Commons privileges. However, only the House itself can determine whether such a breach has occurred.
3.11 The seven sample projects we audited are large IT projects that involve large amounts of money and offer service improvements to large sections of the community. The selected projects were not intended to be a representative sample, but to provide an insight into government management of large IT projects.
3.12 More details on the audit objectives, scope, approach, and criteria are in About the Audit at the end of this chapter.
Observations and Recommendations
Reviews of previous recommendations
Previous audit findings persist
3.13 The Office of the Auditor General last examined systems under development in three government-wide audits done in 1995, 1996, and 1997.
3.14 In 1995, we found
- inadequate analysis of underlying business issues,
- inconsistent support from management and project ownership,
- a lack of experienced resources on project teams,
- inconsistent involvement and acceptance on the part of users, and
- a lack of effective monitoring of systems under development.
3.15 In 1996, we examined another set of large IT projects. We reported that our findings reinforced our previous recommendation that strong governance—consistent support and effective sponsorship of IT projects by senior management—is critical to a project's success and the achievement of intended results.
3.16 We noted that government departments often set their projects up for failure by assigning part-time project leaders, with no training or experience in managing large IT projects.
3.17 In its May 1996 report, the Public Accounts Committee directed the government to respond to our 1995 recommendations by
- allocating more resources to monitor large information technology projects;
- submitting an annual report to the Committee on the status of departments' investments in systems that are under development;
- having the Treasury Board Secretariat take steps to ensure that there is complete ownership and accountability for large IT projects; and
- whenever possible, dividing projects into smaller, more manageable components.
3.18 The Treasury Board Secretariat responded to our recommendations and the Committee's report by creating the overall guidance document, Enhanced Management Framework (EMF). In 1998, the Secretariat created An Enhanced Framework for the Management of Information Technology Projects—Part II to help departments apply the EMF.
3.19 Since 1998, the Secretariat has produced little additional guidance on the management of large IT systems. The Chief Information Officer branch of the Treasury Board Secretariat informed us that it is undertaking a complete review of the EMF and has recently piloted a new methodology for measuring and reporting the outcomes of a few large IT projects.
3.20 Some departments and agencies have had third-party reviews of their projects. We noted that several other governments—notably of the United Kingdom, the United States, and Australia—have adopted new methodologies for monitoring large IT projects. These standard methodologies require independent third parties to assess their largest IT investments at specific intervals during the life of a project. In assessing projects, reviewers apply rigorous oversight that allows them to give advice directly to the executive branch of government.
3.21 For example, in the United Kingdom, the government established the Office of Government Commerce (OGC) to help its departments to develop and deploy successful IT projects. The OGC developed a framework of independent "gateway" reviews, for early detection of any problems that could threaten the success of projects. Since 2001, the OGC's periodic reports have made large IT projects more visible and transparent, making it easier for the government's central agencies to focus on problem areas and, where required, marshal high-level support from management.
3.22 In our previous audits we made recommendations about strengthening governance, business case analyses, project management, and assessments of organizational capacity. The EMF was developed to address these recommendations. Our current audit found that many problems, which our previous reports called attention to, persist because departments and agencies are not following the EMF. We are concerned that although research clearly indicates that small IT projects are more likely to succeed than large ones, departments and agencies are again undertaking large IT projects. Because the portfolio of large-scale departmental IT projects is growing, we believe that a strong governance and management framework is critical if the government is to avoid the past mistakes.
Role of the Treasury Board Secretariat
3.23 We examined IT governance at two levels. Our audit examined how IT governance is exercised at the Treasury Board Secretariat level and at the project level. At both levels, the governance framework includes the administrative structures, the processes, and the people that are involved in various aspects of information technology. If the governance framework is sound with clearly defined roles, responsibilities, and accountabilities, it is more likely that the government will be able to use information technology more successfully.
3.24 More information about project level governance can be found under Governance: Departments and agencies need to improve internal quality reviews (see paragraph 3.45).
3.25 At the Treasury Board Secretariat level, the governance framework is designed to ensure that IT projects fit with the government's priorities, help departments improve their operations or quality of service, offer value for money, and make good business sense for the government. The Treasury Board Secretariat's main role is to review the risks associated with departmental project proposals.
3.26 The Treasury Board's Project Approval Policy states that departments must seek approval before starting certain projects that exceed established departmental project authorities. For projects that involve IT investments, established departmental project authorities range from $1 million (most departments have this level of authority) to $30 million (only National Defence has this level of authority). The minimum project authority for each department is based on the size of the department and its project history. When a department or agency is planning a project that exceeds its established project authority, it must seek the Treasury Board's approval, even if the department or agency plans to use funds that are already in its budget.
3.27 For the types of projects we examined, departments and agencies must seek the Treasury Board ministers' approval for two phases in a project: "preliminary project approval" and "effective project approval." For preliminary project approval, the department or agency presents the complete business proposal, as understood at a preliminary level, with indicative cost estimates. It also seeks the necessary authority to conduct further analyses (and any necessary pilot projects) to add substantive estimates to the proposal.
3.28 Once the preliminary project approval phase is complete and if the original proposal is still sound, the department seeks effective project approval. At this stage, project estimates are considered substantive and the department is expected to manage the project within the time given, using only approved funding and staying within any other limits that Treasury Board ministers impose.
3.29 Treasury Board ministers may set specific conditions, when they approve a proposal, which a department or agency must meet. The Treasury Board may impose conditions related to on/off ramps. For projects that involve significant investments in complex IT, the Secretariat may continue to oversee the projects' progress. For example, the Secretariat may require regular progress or performance reports, or it may sit on advisory and steering committees that provide governance for the project.
3.30 The Secretariat informed us that before it presents them to the Treasury Board ministers for their consideration, it reviews submissions from departments and agencies to
- ensure that the proposals comply with all applicable policies,
- assess the business value of the proposal, and
- determine whether projects have the essential project management elements to be successful.
3.31 The Secretariat's Chief Information Officer branch, its Procurement and Project Management Policy directorate, and its Financial Systems Acceptance Authority directorate have the specialized knowledge needed to scrutinize proposed IT projects. The Secretariat told us that the program sectors lead the review of the submissions, take advice from other relevant areas in the Secretariat, and combine that advice with their own review when they make recommendations to Treasury Board ministers, for their consideration of the submission.
Denial of access to information
3.32 The Secretariat describes its role as one of selective oversight of departments based on an assessment of risks and the Secretariat's available resources and capacity to oversee and monitor projects.
3.33 We asked to see the analysis that the Secretariat used to support the advice it gave the Treasury Board ministers for five IT projects in our sample. The Secretariat refused us access to most of the documents, citing Cabinet confidence of a type that cannot be disclosed to us (referred to as Cabinet confidence in the rest of the chapter). The Treasury Board Secretariat consulted officials in the Privy Council Office who agreed with its decision, which led the Secretariat to confirm that they would not provide this information.
3.34 We requested the analysis, or evidence of project document review that the Secretariat used to determine whether and how well the project complied with the Enhanced Management Framework and related government policies and guidelines. The analysis we requested related to the business case, options analysis, project charter, project plan, and risk management plan.
3.35 We need this analysis to determine whether the Treasury Board Secretariat has
- given adequate consideration to the governance structure, sound management practices, and specific, achievable deliverables; and
- assessed the organization's readiness to accept the business transformation that the project will require and its capability and commitment to deliver the project successfully.
3.36 Without access to this information, we cannot assess the rigour or quality of the Treasury Board Secretariat's analysis of the projects that we reviewed. In fact, we are unable to audit the Secretariat's important challenge and oversight roles relating to the spending of public funds, particularly funds for the significant IT projects.
Analysis by the Expenditure Management Sector
3.37 Projects that are considered high-risk investments or that require access to reserve funds are subject to special expenditure reviews.
3.38 In 2004, the Expenditure Management Sector of the Treasury Board Secretariat designated a group of staff to provide advice on budgetary pressures and on requests for access to funds from the Management Reserve (a source of discretionary funding), and to review and challenge Secretariat expenditure issues and proposals. The designated group's role is to perform a thorough and comprehensive—"due diligence"—review of priority investment proposals to assess whether they will result in a positive return to the Crown. The members of the designated group have government and private sector experience in determining financial implications of decisions.
3.39 The Expenditure Management Sector did a special expenditure review of the Secure Channel project's proposal for multi-year funding for operations. Its analysis of the information from the project's proponents, and benchmarks and comparative analyses from industry experts, led the designated group to conclude that the Secure Channel project did not have a comprehensive business case. They then concluded that the Secure Channel project did not meet the economic requirements to receive funding from the Management Reserve and that it should receive short term funding while it transitions to a new economic model. Public Works and Government Services Canada (PWGSC), the department responsible for the project, requested funding for one year and approval in principle for funding for an additional four and half years. PWGSC also made a commitment to examine issues raised by the Expenditure Management Sector.
3.40 The recommendations of the Expenditure Management Sector considered the risks and the benefits of the project and were based on a rigorous due diligence review. For example, when it reviewed the business case, the Expenditure Management Sector requested a substantive analysis of measurable results, an options analysis, and an assessment of demand. The Expenditure Management Sector tested the substantive evidence offered by the project's proponents and performed its own assessment of benefits, options, and capacity. These requirements are found in the Treasury Board Project Approval policy and the Enhanced Management Framework (EMF).
3.41 The Expenditure Management Sector analysis only represents a part of the advice provided to Treasury Board ministers. We were unable to determine whether the Secretariat considered the Expenditure Management Sector's advice and recommendations when it examined the PWGSC submission, before presenting it to the Treasury Board for approval, because we were denied access to the analyses performed by the Chief Information Officer branch and other sectors of the Treasury Board Secretariat.
Assessment of seven projects
3.42 We reviewed a sample of seven projects and assessed them against four key criteria that are associated with the successful development of IT projects.
3.43 The sample projects that we audited are large IT projects that involve large sums of money and propose service improvements to large sections of the community (Exhibit 3.1).
3.44 Our observations and recommendations are organized under four key criteria: governance, business case, organizational capacity, and project management. We refer to specific projects to illustrate our findings.
Governance: Departments and agencies need to improve internal quality reviews
3.45 At the project level, governance focuses on delivery of business change at an affordable cost with an acceptable level of risk. Departments and agencies that have demonstrated their readiness to use the government's Enhanced Management Framework (EMF) to govern their large IT projects may require less monitoring by the Treasury Board Secretariat.
3.46 According to the Project Management Policy, the department or agency that sponsors a project (and the required IT investment) is responsible for controlling the costs and risks and for ensuring that the project benefits the organization and meets a business need. It is expected to have the capacity to manage and deliver the project and to demonstrate that it is making a sound investment decision—based on objective, unbiased information. A sound governance framework at the project level is critical to fulfilling these responsibilities.
3.47 Departments and agencies have spending limits for IT projects. Unless the Treasury Board approves extra funding, the spending limits for departments and agencies range from $1 million for most departments to $30 million for National Defence. We outline some representative examples of project governance in the following paragraphs.
3.48 Integrated Revenue Collections. The Integrated Revenue Collections (IRC) project is a new system that the Canada Revenue Agency will use to prioritize and allocate collections work according to level of risk.
3.49 The project started in 2001 and progressed very slowly until 2005. For example, the business case that was prepared in July 2001 was not updated until October 2005. The updated business case was approved in September 2006. The governance framework for the project began with quarterly oversight by a bilateral committee of the IT and collections branch staff in 2002 and there was also periodic progress monitoring by the IRC steering committee. By March 2005, the Agency had spent $13 million on this project. We found that it is only since 2005 that the Agency has started to put in place a more rigorous governance framework, a framework that we would have expected to find from the beginning of the project.
3.50 We found that, in 2005, the Revenue Collections branch of the Agency had taken the following measures to improve its governance of the project:
- reorganizing its Technology Solutions directorate,
- establishing a project management office,
- dividing the project into more manageable components,
- clearly defining what needs to be accomplished, and
- assigning clearer responsibilities for carrying out the work.
3.51 Early in 2006, the Canada Revenue Agency took steps to further improve its governance and control of its projects. It established the Resource Investment Management Committee to
- oversee the management and progress of major projects,
- oversee the allocation and control of the Agency's financial resources, and
- establish budget priorities and requirements according to the Corporate Business Plan.
3.52 In addition, the Agency's Finance and Administration branch drafted a project management policy that will give the Agency guidance on managing approved projects effectively, using a clearly established project governance framework.
3.53 All of these measures have brought the governance of the project closer to our expectations.
3.54 Secure Channel. The Secure Channel is a complex, innovative, multi-department project that provides Canadians and businesses with secure and private access to online government services. As a key component of the Government On-Line initiative, this common, secure infrastructure ensures that Canadians can use a single window to conduct online transactions safely and effectively, according to federal privacy standards. We did not audit the privacy and security aspects of the Secure Channel.
3.55 During the planning, design, and field trial period (from 1999 to 2003), the Treasury Board Secretariat's Information Management/Information Technology Management Board (IMB) had final responsibility for oversight, strategic direction, and decision making for the Secure Channel project. The IMB comprised representatives from the technology and program sectors of the larger departments, and it was charged with providing a government-wide strategic direction for all significant IT investments. A key part of its mandate was to ensure that potential Secure Channel users had as much input into its design as possible.
3.56 In December 2003, PWGSC took over the project management responsibility for the Secure Channel project from the Treasury Board Secretariat. The IMB, which the Treasury Board Secretariat chairs, retained the program governance responsibility.
3.57 The governance structure of the Secure Channel has not resolved a number of issues. For example, at the time of our audit, departments and agencies had not agreed on how to sustain this project or what benefits it will provide. For the Secure Channel to succeed as a strategic government-wide investment, a complete and approved program budget is needed. The budget needs to reflect the full life-cycle costs and financial and non-financial benefits. At the time of our audit, this program budget had not been completed.
3.58 Another unresolved issue is the "take-up" by departments and agencies. Currently, take-up of the Secure Channel is far below projected expectations, and it is not clear whether the future demand for the Secure Channel will increase. For example, in November 2005, the Treasury Board Secretariat asked the major user departments and agencies to confirm their forecasted demand analyses. In January 2006, the Canada Revenue Agency responded and stated that while some of its estimates were reasonable and based on available information, others were "best guesses of take-up rates for applications." Some of the Agency's applications are under development. The Agency advised the Secretariat that it would not necessarily endorse any funding model that the Secretariat or the PWGSC may eventually propose.
3.59 Departments and agencies with major service delivery requirements need to identify and commit to
- key outcomes,
- methods to measure those outcomes and accountability for them,
- an expected delivery schedule, and
- a monitoring process.
3.60 We found that, for four of the seven projects in our sample, the governance framework was deficient. Even though the governance structure was defined for all projects, we found that not all responsibilities were carried out, and not all key business issues were reported and resolved.
3.61 Recommendation. Government departments and agencies should improve their internal quality reviews of IT projects. Senior departmental executives should review the key decision documents that are produced to support the IT project and ensure that the analysis is thorough and supportable before signing off on the submission.
The government's response. The government agrees that improvement is needed in many departments and agencies. The government undertakes many significant business initiatives that are supported by information technology with varying degrees of complexity. By approaching these IT-enabled projects from a business perspective, rather than simply a technology perspective, departments and agencies will place appropriate attention on planning the project, on managing the necessary transitions, and on involving all key stakeholders.
Business cases: Five of the seven assessed business cases were inadequate
3.62 The business case is the foundation of every sound investment decision. For IT projects, the business case explains the rationale for the project and the project results that are needed to meet an organization's business needs.
3.63 The Treasury Board Secretariat defines a business case as "a detailed investment proposal [that] provides an analysis of all the costs, benefits and risks associated with a proposed investment and offers reasonable alternatives." Treasury Board Secretariat guidelines further note that
The business case is reviewed and re-validated at each scheduled project review and whenever there is a significant change to the project or the business function. If the business case changes during the course of an IT project, the project needs to be re-approved through the departmental planning and approval process.
3.64 According to the Treasury Board Secretariat guidance on project management, the organization must define what constitutes success in its business case, and must use that definition to measure the project's success later on. A business case would indicate how the project aligns with the organization's broader strategy, and would include
- a definition of the project's scope;
- an options analysis;
- the expected outcomes;
- the cost (in both time and resources);
- a risk analysis; and
- criteria for measuring success.
3.65 Departmental management and ministers need the information from a sound business case to make informed investment decisions. They want to invest in projects that will yield the most benefits for the lowest cost and will take the shortest time to complete. The information that a department or agency presents in its business case must be objective, unbiased, and verifiable.
3.66 The Treasury Board Secretariat is required to evaluate departments' and agencies' compliance with all applicable policies and the benefits of information technology investments that they recorded in their business case analyses.
3.67 Except for the 2006 Census Online project and the My Account, My Business Account project, we found that the business cases for our sample of seven projects were incomplete, were out-of-date, or contained information that could not be supported.
3.68 Secure Channel. The Secure Channel project business case contained some information that was either not complete or could not be supported. As a result, this project did not comply with the Treasury Board's policy for approving projects and the guidelines for business cases.
3.69 For example, the business case did not identify a source of funds to sustain the ongoing operating and maintenance costs of the Secure Channel, as required in the Project Approval Policy. On several occasions, project deadlines were extended and short-term funds were advanced, even though the project's business case did not show the expected benefits of the new technology, or how the investment costs would be recovered.
3.70 During the Secure Channel project, the Treasury Board made five requests for a long-term sustainable business model that indicate the source of funds for the operation of the Secure Channel. In the summer of 2005, Public Works and Government Services Canada (PWGSC) prepared a business case, which the Treasury Board Secretariat endorsed. However, we found that the business case did not address the sustainability concerns that we raised in our November 2003 Report. At that time, we recommended that
The government should address the key risks and challenges it faces by finalizing a comprehensive business case for the Secure Channel project, addressing its long-term financing, establishing mechanisms to encourage the adoption of Secure Channel by departments and agencies, businesses, and Canadian citizens . . . .
3.71 In March 2006, the Treasury Board Secretariat's Expenditure Management Sector summarized its analyses and agreed with our findings that the business case was not comprehensive. By 31 March 2006, the project had received funding 11 times, for a total of $596 million (including $196 million for operations).
3.72 The project began as a "proof of concept," designed to show that the technology could be applied in a government setting. In September 2003, the project received effective project approval to start to use the technology. By May 2004, PWGSC was supposed to
- request the remainder of the funds needed to use the technology;
- clearly identify the source of funds, which could be charged back to departments; and
- produce a business plan that clearly showed how much the system would cost to operate and where the funds would come from, or submit a plan to wind up the project.
3.73 PWGSC did not produce a business plan or a plan to wind up the project. Instead, it provided the Secretariat with the estimated cost of an exit strategy. However, in March 2004, PWGSC received approval to use the funds earmarked for an exit strategy to sustain operations for another six months—including to continue to work on the business plan and to identify a long-term source of funds to sustain Secure Channel operations.
3.74 In April 2005, PWGSC asked the Treasury Board for $126 million to fund operations until 31 March 2006. PWGSC also requested approval in principle for $509 million to fund operations for the next 4.5 years. At that time, PWGSC made a commitment to send an updated request before December 2005, for approval to sign a contract with the Secure Channel service provider and for final approval of the funds. This updated request was actually made in June 2006. When PWGSC made the request in April 2005, it indicated that three departments that use the technology may receive $5.2 billion of indirect benefits—notably, HRSDC and PWGSC could each realize benefits of about $2.5 billion over five years. We asked to see the analysis that supported these projected benefits. However, because we were not given this analysis, we do not know how these amounts were determined.
3.75 The Secure Channel started operations in September 2004. Because PWGSC did not receive firm commitments from departments before the system went into production, it has operated at less than 50 percent of its capacity, for a total cost of $196 million, from its starting date to 31 March 2006.
3.76 The Expenditure Management Information System (EMIS). The Expenditure Management Information System (EMIS) was designed to support all aspects of government expenditure management by providing information on the cost of meeting the government's priorities and achieving particular results. The Treasury Board Secretariat sponsored the EMIS project and although the project did have a business case, we found that it was incomplete. Without an effective business case, it is difficult to measure whether the EMIS project successfully met its specific objectives and timelines.
3.77 From the beginning, the business goals of the EMIS project were not clear and were not understood by all parties.
3.78 The business case for the project had no comprehensive project plan; the business goals were vague, immeasurable, and not "time limited." As a result, it has been increasingly difficult for the EMIS project team to determine whether they are achieving their overall objectives.
3.79 An options analysis is a key component of any business case, and it is the foundation for deciding what direction the project will take. Because the most recent EMIS business case, in 2004, does not include an options analysis, such as whether to buy a system or build one in-house, we could not determine the reason for the project's direction. Since the project team members had not properly assessed their options, it is not clear if the one they chose—to build the system—was the most appropriate, effective, and efficient. Nevertheless, this project, which is currently valued at $53.7 million, is still proceeding but in a different direction.
3.80 A new business case is being developed for the Treasury Board to consider in the fall of 2006.
3.81 The Global Case Management System (GCMS). In June 2000, Citizenship and Immigration Canada (CIC) presented a business case to the Treasury Board Secretariat for a Global Case Management System (GCMS). CIC proposed that it would develop the new system in-house with external help as needed. The purpose of this project was to replace 14 existing systems with one integrated case-management system that would better meet domestic and international operational needs.
3.82 In March 2001, the project was approved with a total budget of $194.8 million, conditional upon CIC's considering an off-the-shelf package instead of developing a system in-house.
3.83 The overall objective of the project was to develop an integrated case management system. However, the business case did not define measurable objectives and benefits. CIC had to consider purchasing an off-the-shelf system, which involved modifying its business case to focus on different business needs and new technologies. With the modified business case, CIC obtained effective project approval in January 2002.
3.84 CIC had to struggle with a number of problems, including changes in the project's scope, difficulties accessing funds, and a lack of people with the required skills. In 2003 and 2005, CIC approached the Treasury Board to request budget increases and more time.
3.85 We found that CIC received budget increases without submitting a revised business case to the Treasury Board. CIC included specific issues in its requests. However, the original business case was not revisited to provide a clearer picture of the project or to redefine the objectives and outcomes as the project evolved. The cost of the GCMS project is now estimated to be $242.8 million—the original estimate was $194.8 million—and the system is scheduled to be completed by August 2007.
3.86 My Account and 2006 Census Online. My Account and 2006 Census Online had sound business cases. The 2006 Census Online project gave Canadians the option of completing their 2006 census questionnaires electronically and using the internet to submit them to Statistics Canada securely and confidentially. My Account is a series of projects that will give Canadians and Canadian businesses a personalized, secure, and private online service for conducting transactions with the Canada Revenue Agency, the provinces, and federal programs.
3.87 We found that these two projects, with sound business cases (either as originally submitted or after revisions) proved feasible, met a defined business need, and made sense from a value-for-money perspective.
3.88 AgConnex. AgConnex was originally designed to give farmers a single window to access the various income support programs offered by Agriculture and Agri-food Canada. However, in the summer of 2002, the launch of the joint federal/provincial Agriculture Policy Framework significantly changed the scope and cost of the project, which was originally estimated at $60 million. By the summer of 2003, the estimated cost of the project had increased to over $177 million. In 2003, the Department prepared a revised business case after it had spent about $14 million. Six months later, the Department decided not to invest further in this project and it was discontinued because it was considered high risk. The Department followed up by conducting a business alignment exercise, at an additional cost of $8.5 million, which resulted in several smaller, lower risk projects. We did not audit these projects.
3.89 Recommendation. Before seeking effective project approval, departments and agencies should prepare a business case that includes, at a minimum, precise and measurable objectives; a full analysis of options, benefits, costs, and risks; and an implementation plan.
The government's response. The government agrees and will continue to improve the calibre and breadth of departmental business cases.
3.90 Recommendation. The Treasury Board Secretariat should improve the requirements for sound business cases prepared by departments and agencies to ensure that they include, at a minimum, precise and measurable objectives; a full analysis of options, benefits, costs, and risks; and an implementation plan.
The government's response. The government agrees. The Treasury Board Secretariat has undertaken a comprehensive policy suite renewal that includes the project management policy. In this context, the Secretariat has been working with departments and agencies to address several matters related to the management of projects, including governance; risk management controls; capacity; and integration with investment planning and procurement management.
As part of this policy suite renewal, the Secretariat has been working with departments and agencies on improving the Enhanced Management Framework (which provides the supporting project management methods, processes, guidelines and tools for IT-enabled business projects) and looking at ways to share best practices. For example, significant progress has been made in the integration of outcomes management into the management of IT-enabled business projects. When fully implemented, this will enable departments and agencies to provide greater clarity around the business problem that is to be addressed, to continuously align project activities to the desired outcomes, to monitor outcomes even after the project is concluded, and to measure results. The Secretariat has completed a number of pilot projects related to outcomes management and is working towards ensuring greater understanding and use by departments.
Organizational capacity: Organizational capacity was assessed for only two of the seven sample projects
3.91 In this chapter, organizational capacity refers to the technical and managerial ability to deliver an IT project. It also refers to the ability of the entire organization to improve the way it does business by using all of a system's capabilities.
3.92 According to the Treasury Board's Project Management Policy and the Project Risk Assessment and Management Guidelines, departments and agencies need to objectively assess and manage the risks associated with their project. In our view, this includes their ability to use the system to improve the way they deliver their programs. We found that only two of our seven sample projects—2006 Census Online and My Account, My Business Account—were rigorously assessed in their Project Profile and Risk Assessment, as required by the Project Management Policy. We found that organizations proceeded with projects without having appropriate expertise to manage them.
3.93 Expenditure Management Information System (EMIS). The Treasury Board Secretariat sponsored the EMIS project, and we found it lacked the organizational capacity for the new system. The Secretariat never assessed whether it had enough people with the appropriate skills and experience to complete the project. Specifically, the Secretariat never determined the resources it needed to meet the objectives of the EMIS project. In addition, we found no evidence of a formal mechanism to align human resources planning with technology planning. In November 2005, the Treasury Board Secretariat reported that 33 percent of its funded positions associated with the EMIS project were vacant.
3.94 High turnover in management positions has also been a major problem. Since 2001, the EMIS project has had six project managers and seven senior responsible officers.
3.95 Clearly, poor organizational capacity—not filling vacant positions with experienced and qualified people—prevented the Secretariat from achieving its goals. The Secretariat has recently made significant improvements to strengthen its governance and project management practices. However, at the time of our audit, there were not enough experienced and qualified people working on the project for it to progress as originally planned.
3.96 2006 Census Online. Statistics Canada thoroughly and successfully assessed the Census 2006 Online project and built its capacity, and that of its partners, to complete the project.
3.97 Business and IT project leaders had been managing similar Statistics Canada projects for several years. The knowledge and experience of the staff was preserved during the project because there was little turnover on the project team. Their experience allowed them to prepare contingency plans for problems that could have caused Statistics Canada to miss a non-negotiable deadline: Census Day.
3.98 Recommendation. At the start of a project, departments and agencies should clearly demonstrate that their organization is ready to accept the business transformation that comes with the project and has the capability and commitment to successfully deliver the project. Specifically, departments and agencies should analyze their track record in
- completing projects of similar size and complexity;
- completing skills appraisals and plans to address shortfalls;
- planning for business preparation, transition, and operational phases; and
- considering key stakeholder buy-in and commitment.
The government's response. The government agrees and will continue to work with departments in this area. The government recognizes the need for a collaborative environment where project management practitioners and executives from across government can contribute to improving the understanding and application of mature project management disciplines. The Treasury Board Secretariat will continue to work with departments and agencies to ensure that a sound community of practice is established.
3.99 Recommendation. Before recommending that the Treasury Board approve an IT project, the Treasury Board Secretariat should
- ensure that the departments and agencies submit their analyses of organizational capacity, and
- validate the content and approach of these assessments.
The government's response. The government agrees but this is a joint responsibility of the Treasury Board Secretariat and departments and agencies. The Secretariat has been actively engaged in the development of a standardized capacity assessment tool that is based on industry standards to make it easier for departments and agencies to assess their organizational capacity and for the government to validate those assessments. Where appropriate, Secretariat officials will request completed assessments from departments and conduct high-level reviews to understand the approach used by departments and to assess high-level completeness. Departmental and agency executives remain responsible for conducting the detailed analyses and creating the appropriate action plans to ensure that their organizations have the necessary skills and environment to successfully deliver their business projects.
Project management: Four projects were well managed
3.100 An organization uses project management to control and coordinate its activities, resources, time, and costs.
3.101 The objective of developing sound project management is to ensure that the project produces successful outcomes, within a specified time. A key component of project management is risk management, which involves assessing the potential impact of risks, continuously monitoring the risks, and dealing with potential risks before they arise. Risks must be identified and controlled for all IT projects if the projects are to succeed.
3.102 In this audit, we looked for evidence that projects had employed good project management practices and that they could anticipate and control risk. We found that the quality of project management varied widely from project to project—from good to seriously flawed. Projects that had weak project management practices have experienced long delays and large cost overruns.
3.103 The 2006 Census Online. By having sound project management and an effective governance framework, Statistics Canada showed good project management for the 2006 Census Online project, which met its requirements on time and within budget. It had specific project "gates" (decision points) and deliverables. The organization continuously reported on progress and addressed key issues, such as managing risks, managing changes as they arose, and financial reporting.
3.104 The 2006 Census Online project clearly defined roles and responsibilities for meeting its objectives. The management structure allowed close collaboration among partners and was effective for this large, complex, and multidisciplinary initiative that had a fixed delivery date and budget.
3.105 Using a thorough risk management strategy, the 2006 Census Online project was able to assess and deal with the various challenges and risks that might have prevented it from achieving the project's objectives.
3.106 Expenditure Management Information System (EMIS). The EMIS project, which began in 2000, is an example of ineffective project management. Before January 2006, the EMIS project had no effective project plan—critical to ensure that it would be able to meet its goals. It did not have a performance measurement framework with clear reporting timelines, nor did it have a detailed project plan with specific project milestones and deliverables.
3.107 We found that risk management for the project was weak and incomplete. Although a risk management plan was developed in March 2004, it was not used effectively. The plan identified risks and their severity, but we did not see evidence that those risks were mitigated.
3.108 Since January 2006, the EMIS project team has made significant changes to strengthen the project's practices for managing risks and managing the project as a whole.
3.109 We acknowledge that sound project management does not necessarily mean that a project will be delivered within budget and on time. However, good project management is essential to delivering a system that works.
3.110 Four projects that we audited—the 2006 Census Online, Global Case Management, My Account, and Secure Channel—were well managed.
3.111 Recommendation. Departments and agencies should
- comply with the Treasury Board Secretariat project management guidance,
- closely manage the project risks,
- monitor key success factors, and
- regularly report to management any significant internal or external event that may prevent the expected project benefits or result in cost overruns or in the project not being delivered on time.
The government's response. The government agrees. It is clear that departments are accountable for bringing forward well thought out proposals that are supported by sound business cases and achieved through well-managed projects. A number of departments are using third-party reviews to monitor their IT-enabled business projects. The Treasury Board Secretariat will undertake additional work to establish a more consistent approach to employing these reviews as part of the ongoing oversight of such projects across the Government of Canada. The government is committed to clarifying the policy instruments relating to project management and to creating the supporting environment required for success, including an indicator in the Management Accountability Framework to assess how well departments are managing their projects.
3.112 Exhibit 3.2 summarizes our criteria, the projects we reviewed, and how well each project rated against the criteria.
3.113 Overall, the government has made limited progress since our last audit of IT projects in 1997. Although, in 1998, the Treasury Board Secretariat established a framework of best practices for managing IT projects, many of the problems we cited in past reports have persisted.
3.114 With the exception of the 2006 Census Online project and the My Account, My Business Account project, we found that the proponents of large IT projects had not adequately explained the project's rationale or the results they expected to achieve in their business cases. In addition, they had not adequately assessed their capacity to take on high-risk IT projects that have long life spans and require large amounts of public funds to build and maintain.
3.115 Most projects we looked at suffered from the same shortage of experienced and qualified people and inadequate analysis of key business issues as before. The lack of progress is worrisome, because large IT projects are becoming more and more complex and often involve a growing number of players across government.
3.116 Although we found that four of the seven projects were well managed, the quality of project management varied widely. Those with weak project management practices experienced long delays and large cost overruns.
3.117 One of the objectives of this audit was to determine whether the Treasury Board Secretariat has adequately fulfilled its challenge and oversight responsibilities as part of its role in the governance of large IT projects. Since we were denied access to most of the evidence of the Secretariat's review of project documents such as the business case, options analysis, project plan, and risk management plan for our sample of IT projects, we are unable to determine whether it followed a complete and rigorous process when it reviewed these IT projects.
About the Audit
In considering the development of large IT projects, we had three objectives.
We wanted to determine
- whether the process for reviewing and approving projects was sufficiently rigorous;
- whether the business cases and the project charters for the projects that we examined had clearly identified the expected results, and how they would contribute to meeting the business needs of the government and the sponsoring department; and
- whether the projects were managed effectively.
Scope and approach
Our audit focused on two levels:
- Treasury Board Secretariat level. The Treasury Board Secretariat's review and oversight of large projects, more specifically, large IT projects.
- Project level. Seven large IT projects in the following departments: Public Works and Government Services Canada, Citizenship and Immigration Canada, Statistics Canada, the Canada Revenue Agency, Agriculture and Agri-Food Canada, and the Treasury Board Secretariat.
We assessed the process that departments and agencies use to approve and manage projects, the soundness of their business cases, and their organizational capacity to carry them out.
We could not audit the review and oversight role played by the Treasury Board Secretariat because it has denied us access to most of the information and analysis it collects and prepares, citing Cabinet confidence. The government has effectively imposed a limitation on the scope of the Auditor General's examination.
Our audit assessed the performance of the Treasury Board Secretariat. It also examined how well the departments and agencies that sponsor the seven IT projects in our sample meet the following criteria:
- Governance. Clear accountabilities and a governance structure have been established to ensure that projects are completed successfully.
- Business case. Large IT projects are aligned with and support departmental and government business directions and priorities.
- Organizational capacity. Project management teams are skilled and experienced and have the leadership, capabilities, and commitment to deliver their projects successfully.
- Project management. IT project management is appropriate and based on risk management. It can be used to deal with changes to planned deliverables and the project scope; to identify and resolve project issues that arise during the project life cycle; and to identify, report, and manage project risks.
Audit work completed
Audit work for this chapter was substantially completed on 30 June 2006.
Assistant Auditor General: Douglas G. Timmins
Principal: Richard Brisebois
Directors: Greg Boyd, Tony Brigandi, Guy Dumas, Maria Wisniowski
Marie-Claude La Salle
José Alpizar Fallas (International Fellow)
For information, please contact Communications at 613-995-3708 or
Horizontal initiative—According to the Treasury Board Secretariat, an initiative in which partners from two or more organizations have established a formal funding agreement (for example, a Memorandum to Cabinet, Treasury Board submission, or federal-provincial agreement) to work toward the achievement of shared outcomes. (Return)
On/off ramps—Future decision points are set to determine if the project can continue. (Return)
Expenditure Management Sector—Primarily responsible for managing the reserves assigned to Treasury Board to support priority investments or provide interim funding for urgent program requirements. (Return)