2011 June Status Report of the Auditor General of Canada

Chapter 2—Large Information Technology Projects

Main Points

Introduction

What we found in previous audits
Events since our last audit
Focus of the audit

Observations and Recommendations

Treasury Board of Canada Secretariat’s roles and responsibilities

Central monitoring of large IT projects has been recently enhanced
The Secretariat’s review mechanisms contribute to the success of large IT projects
The Secretariat has improved its policy framework for the overall direction of large IT projects
The Secretariat implemented an action plan, improved its guidance, and improved reporting

Progress in key management areas

Governance structures remain problematic for two of four projects we examined
Most business cases do not clearly identify and measure benefits
Most of the large IT projects we audited have taken steps to address capacity issues
Most of the large IT projects we audited have taken steps to address project risk management issues

Conclusion

About the Audit

Appendix—List of recommendations

Exhibits:

2.1—Progress in addressing our recommendation on business cases

2.2—Progress in addressing our recommendation on organizational capacity

2.3—The size and scope of certain projects have changed significantly

2.4—Progress in addressing our recommendation on improving project governance and accountabilities

2.5—Progress in addressing our recommendation on business cases

2.6—Progress in addressing our recommendation on organizational capacity

2.7—Progress in addressing our recommendation on project risk management and reporting

2.8—Overall progress against our recommendations for the projects

Main Points

What we examined

Large information technology (IT) projects involve more than introducing new hardware and software and systems. These projects can introduce new processes and practices—new ways of doing business—that also need to be successfully implemented before organizations can take advantage of potential efficiencies and savings. When successful, these projects can change the way that departments carry out their work and improve services to Canadians. Because large IT projects are complex and costly, they usually involve long planning and development times and require significant investments (on average, over three years and more than $70 million).

In 2006, we looked at seven large IT projects and found that only two of them met all of our audit criteria for well-managed projects. We found that five of the seven projects had proceeded even though their business cases were incomplete or out of date or contained information that could not be supported. In addition, four of the projects were undertaken by departments that lacked the required skills and experience to manage them. Although by 2006 the Treasury Board of Canada Secretariat had established a framework of best practices for managing IT projects, many of the problems identified some nine years earlier had persisted. In 2006, we also found that limited progress had been made since our previous audit in 1997.

For this status report, we examined the progress made since 2006 by the four departments that had not met all of our criteria in 2006. We also selected a new project, approved by the Treasury Board since 2006, in order to assess the government’s progress in the way it approves and manages large IT projects.

In our 2006 audit, we were denied access to information by the Secretariat, which prevented us from completing our review of its challenge role in support of oversight of large IT projects by Treasury Board ministers. In this audit, we were able to review information that demonstrated the role played by the Secretariat.

Audit work for this chapter was substantially completed on 29 October 2010.

Why it’s important

The federal government relies on information technology systems to provide many programs and services to Canadians. Large IT projects are inherently complex, expensive, and risky. Since 2002, the federal government has approved funding of $7.5 billion for new business projects making significant use of information technology. In the four audits we performed since 1995, the projects audited have a history of cost overruns and delays, and of not delivering what had been planned originally.

What we found

  • Overall, the government has made unsatisfactory progress on its commitments in response to our 2006 recommendations. Although some improvements have been made, progress has been unsatisfactory in the important areas of governance and project business cases. Only two of the five projects we looked at, the Temporary Resident Biometrics Project and Integrated Revenue Collections systems, met most of our criteria for well-managed projects.
  • In order to increase the likelihood of success, departments have significantly reduced the scope of the projects and considerably extended their timelines. In the area of project governance, the Expenditure Management Information System and Global Case Management System have had important deliverables deferred without full analysis of the impacts and costs of not completing these projects. In three of the five projects examined, the project business cases did not identify measurable benefits or the benefits have not been measured. For example, the 2007 business case for the Secure Channel no longer quantifies the financial benefits.
  • In three of the four projects we examined, departments adequately assessed their capacity to manage the projects and demonstrated that they were ready to accept the business transformation that came with them; in three of the four projects, departments have done an adequate job of managing projects risks.
  • The Secretariat has met the commitments it made in response to our recommendations we addressed to it. Since 2007, it has completed a policy suite renewal, which has led to new policies, standards, and guidance that will directly impact information technology projects by the end of the five-year implementation period. The Secretariat has been actively challenging departments in their preparation of their IT project submissions; recently (September 2010) it also submitted its first semi-annual report to Treasury Board ministers summarizing the health of large IT projects across government. That report provided a snapshot of 12 ongoing projects with a total value of $2.4 billion. However, it is too early to assess whether these measures will have a significant impact on the management of large IT projects by departments.

The organizations have responded. The organizations agree with all of the recommendations. Their detailed responses follow the recommendations throughout the report.

Introduction

2.1 Information Technology (IT) projects are critical to maintaining and improving the quality and efficiency of the federal public service. These projects extend beyond the technical aspects of replacing hardware and software. For example, new high-speed electronic services may replace high-volume manual transactions, resulting in the modernization of current work practices and transforming the way the departments or agencies do business. These projects could also involve replacing aging applications and infrastructure that are becoming increasingly expensive to operate and may pose certain risks to the delivery of services to Canadians.

What we found in previous audits

2.2 The Office of the Auditor General has examined the management of large IT projects (referred to in our previous reports as “systems under development”) four times since 1995. In 1995, 1996, and 1997, we reported that IT systems under development were characterized by

  • inadequate analysis of underlying business issues,
  • inconsistent support from senior management,
  • a lack of experienced resources on project teams,
  • unrealistic time frames,
  • inconsistent involvement and acceptance on the part of users, and
  • a lack of effective monitoring.

2.3 In Chapter 3 of our 2006 November Report, we revisited large IT projects, given the significant investments that the federal government was making in this area. We found there had been limited progress since our 1997 audit and that, although the Treasury Board of Canada Secretariat had established a framework of best practices for managing IT projects, many of the problems that we cited in past reports were still evident. We also found that, only two of the seven projects we reviewed met all of our audit criteria for well-managed projects.

Events since our last audit

2.4 At the time of our 2006 audit, a large IT project was defined as a business project that incorporated a significant IT component, with an estimated value that exceeded the project approval limit of a department or agency, and therefore required Treasury Board approval to proceed. Since 2006, new criteria for setting departmental approval limits for projects, including IT projects, were introduced. These criteria assist in ranking a department’s overall capacity to manage projects and the given IT project’s risk and complexity.

2.5 In late 2006, the Treasury Board approved the new Policy Framework for the Management of Assets and Acquired Services, supported by policies approved in 2007. These policies and their supporting standards are designed to ensure that the government’s investments in large capital projects, including IT-enabled projects, are managed more effectively and economically. Departments have until April 2012 to fully implement these policies and standards.

2.6 From July 2006 to 2010, the Treasury Board approved 45 business projects with a significant information technology component for a total value of over $3.2 billion. This compares with the 75 projects approved for a total value of $3.4 billion from 2002 to June 2006. During the same period, the Canada Revenue Agency approved 35 projects with a significant information technology component for a total value of $456 million, compared with 23 projects for a total value of $431 million from 2002 to 2006.

Focus of the audit

2.7 We examined the Secretariat’s role in challenging large IT projects and supporting Treasury Board ministers’ oversight, as well as its provision of guidance to departments on effective management and oversight of their IT projects.

2.8 We also assessed progress since 2006 of the four projects that had not met all of our audit criteria and were not cancelled:

  • Expenditure Management Information System, at the Treasury Board of Canada Secretariat
  • Global Case Management System, at Citizenship and Immigration Canada
  • Integrated Revenue Collections, at the Canada Revenue Agency
  • Secure Channel, at Public Works and Government Services Canada

2.9 We reviewed one additional project, the Temporary Resident Biometrics Project, at Citizenship and Immigration Canada. Since the Secure Channel project was completed by 2006, during our current audit, we looked at only the progress made in measuring and reporting on defined benefits presented in its business case.

2.10 We also reviewed actions taken in relation to the government’s commitments in response to the recommendations the Parliamentary Standing Committee on Public Accounts made in its 7th and 4th reports to the House of Commons on 25 February 2008 that were relevant to issues within our audit scope.

2.11 More details about the audit objectives, scope, approach, and criteria can be found in About the Audit at the end of this chapter.

Observations and Recommendations

Treasury Board of Canada Secretariat’s roles and responsibilities

2.12 This section reports on new audit work that deals first with the Treasury Board of Canada Secretariat’s challenge role in support of Treasury Board ministers’ oversight of large IT projects. Second, it reports on the Secretariat’s guidance to departments on effective management and oversight of large IT projects. Finally, we reviewed the actions taken in relation to the government’s commitments in response to recommendations made in 2006 that related directly to the Secretariat’s leadership role, and in response to recommendations made by the Public Accounts Committee in its 7th report to the House of Commons in 2008.

During our 2006 audit, the Office of the Auditor General did not have full access to information related to the Secretariat’s challenge role in support of Treasury Board ministers’ oversight responsibilities, so we could not complete that part of our audit. During this audit, we were able to review information that demonstrated the role played by the Secretariat and have included our findings here along with our examination of progress against past recommendations.

2.13 The Secretariat makes recommendations and provides advice to the Treasury Board on policies, directives, regulations, and program expenditure proposals with respect to the management of the government’s resources. Its other roles include providing leadership to help departments and agencies to both adopt best management practices and improve their performance.

2.14 We examined the Secretariat’s governance mechanisms to determine whether it has an effective framework for the government-wide portfolio of large IT projects under development.

Central monitoring of large IT projects has been recently enhanced

2.15 We examined whether the Secretariat, and more specifically its Chief Information Officer Branch (CIOB), has instituted measures for monitoring and reporting on large IT projects.

2.16 Since December 2009, the CIOB has periodically received executive project dashboards for 12 large IT-enabled business projects with a total value of $2.4 billion. These 12 IT projects are a subset of the total IT project portfolio; they were selected based on the following criteria: significant complexity and risk, significant change in performance, and an estimated budget that exceeded $100 million. The dashboards provide a periodic (usually monthly) snapshot of the overall health of these projects.

2.17 The Secretariat committed to report, on a semi-annual basis, to Treasury Board on the overall health of large IT projects. The IT-enabled Project Portfolio Status Report is the primary governance mechanism for monitoring and advising the Treasury Board on the health of large IT projects across government. The CIOB presented this report to the Treasury Board for the first time in September 2010. It indicated that eight projects totalling $1.9 billion were on track, one project needed some course correction, and three projects totalling $463 million needed significant course correction. During this audit, we examined two of the twelve projects in the report.

2.18 The Assistant Deputy Minister Executive Project Oversight Committee, which was established in July 2009, is chaired by the Chief Information Officer (CIO). The committee provides guidance on the development of the semi-annual report on IT-enabled project oversight and advises the CIO, who in turn uses the report to provide advice and recommendations on selected key IT initiatives to the Treasury Board.

2.19 These governance mechanisms that the Secretariat has recently implemented are a significant improvement in monitoring the health of large IT projects in which the government invests billions of dollars. However, it is too early to assess their impact on the success of these projects.

The Secretariat’s review mechanisms contribute to the success of large IT projects

2.20 We examined the systems and practices that the Secretariat has designed to increase the likelihood of success for large IT projects at the outset, during the Treasury Board submission and approval process. This relates to our 2006 recommendation for improved requirements for the business cases prepared by departments and agencies.

2.21 A key practice to enhance the likelihood of success is to review large IT project submissions from departments before presenting them to the Treasury Board for approval. A submission must include

  • links to new and existing government programs;
  • clear links to Government of Canada IT strategic directions;
  • clear objectives, expected results, and business outcomes; and
  • details of the options considered.

2.22 We found that the Secretariat looks for evidence that departments are using effective project management and oversight practices that are suitable for large IT projects. We also noted that the Secretariat has provided comprehensive guidance to departments in its Business Case Guide and Outcome Management Guide.

2.23 We found that the CIOB reviews IT project submissions to determine whether the “conditions for success” are apparent in the management framework that the department will use for the proposed project. CIOB officials probe and challenge the project proposal to determine whether there is

  • a clear business justification;
  • a clear and reasonable scope;
  • a suitable governance framework; and
  • reasonable estimates, decision points, and independent reviews.

2.24 In the submissions we reviewed, we found documentation of the Secretariat’s challenges in the form of emails and meeting notes but no formal procedures on how it performs its review and challenge function. The Secretariat informed us that the challenge process is largely done orally and the extent of the review depends on the nature of the proposal, and that various policies apply on a case-by-case basis.

2.25 Based on industry-leading practices, the CIOB has created additional tools and guides to assist the departments in improving IT project management. Although it is not mandatory for departments to use these tools, the CIOB uses them in its review of project submissions. The Secretariat’s program sectors seek advice on IT issues from the CIOB, and both the CIOB and the program sectors communicate with departments to offer them advice on the proposed management framework of their IT projects. The CIOB has implemented an independent review program that uses a gating framework designed to enable departments to improve their investment outcome. It strongly encourages departments that are undertaking complex, high-risk IT projects to take advantage of this program early in the planning process. The departments use these independent reviews to gather objective evidence to validate their options and costs, assess the feasibility of their project plans and schedule, and assess the capacity of the organization to take on the project.

2.26 Secretariat officials told us that its challenge function is limited to a review of compliance with government policy because the deputy minister of a department is responsible for ensuring the completeness and accuracy of the submission. The deputy minister in turn relies on the recommendation of the senior financial officer to proceed with a project submission to the Treasury Board. Since December 2007, the senior financial officer must attest to the deputy minister that, in his or her professional opinion, the information in the submission is fairly presented, proper analysis has been carried out, and due diligence has been exercised.

2.27 We observed that, since the clarification of the senior financial officer’s responsibility, the Treasury Board submissions and supporting business cases for the projects that we reviewed have more conservative plans and reduced scope than before. We also noted that the departments have undertaken independent reviews of their project plans and organizational capacity to assist them in performing their due diligence before submitting their project proposals to Treasury Board.

2.28 Based on these findings, the Secretariat and its CIOB have made satisfactory progress against our recommendation for improved requirements for business cases (Exhibit 2.1).

Exhibit 2.1—Progress in addressing our recommendation on business cases

Recommendation Progress
The Treasury Board Secretariat should improve the requirements for sound business cases prepared by departments and agencies to ensure that they include, at a minimum, precise and measurable objectives; a full analysis of options, benefits, costs, and risks; and an implementation plan.

(Recommendation 3.90 of the 2006 November Report of the Auditor General of Canada, Chapter 3, Large Information Technology Projects)

Satisfactory

 

Satisfactory—Progress is satisfactory, given the significance and complexity of the issue, and the time that has elapsed since the recommendation was made.

Unsatisfactory—Progress is unsatisfactory, given the significance and complexity of the issue, and the time that has elapsed since the recommendation was made.

The Secretariat has improved its policy framework for the overall direction of large IT projects

2.29 We recommended in 2006 that the Secretariat ensure that departments demonstrate their capacity to undertake large IT projects.

2.30 In 2006, Treasury Board approved the Policy Framework for the Management of Assets and Acquired Services, which sets overall direction for large projects. The framework is made up of policies and standards that seek to ensure that government projects provide value for money and are managed effectively.

2.31 The Secretariat is responsible for assessing whether departmental deputy ministers comply with the framework’s policies and standards and for advising the Treasury Board on the need for additional mechanisms if needed. Full compliance with these policies and standards is not required until 2012, five years after they came into effect.

2.32 The Policy on Investment Planning, which is part of the new framework, requires departments to submit an investment plan covering a minimum five-year period. The Policy on the Management of Projects, which is also part of the new framework, requires departments to submit an Organizational Project Management Capacity Assessment. The result of the capacity self-assessment is one of the new measures used to arrive at the project approval limit for the department. The Secretariat has committed to completing a high-level review of the completeness of the department’s self-assessment and does not examine the department’s project management capacity for each of its capital projects, which Treasury Board policy indicates is the role of deputy ministers. At the time of this audit, Public Works and Government Services Canada was the only department among those we examined that had completed the self-assessment of its organizational project management capacity and submitted it to the Secretariat for review. However, progress in addressing our recommendation is satisfactory given that a tool is in place and some departments have started to complete these self-assessments (Exhibit 2.2).

Exhibit 2.2—Progress in addressing our recommendation on organizational capacity

Recommendation Progress
Before recommending that the Treasury Board approve an IT project, the Treasury Board Secretariat should
  • ensure that the departments and agencies submit their analyses of organizational capacity, and
  • validate the content and approach of these assessments.

(Recommendation 3.99 of the 2006 November Report of the Auditor General of Canada, Chapter 3, Large Information Technology Projects)

Satisfactory
Satisfactory—Progress is satisfactory, given the significance and complexity of the issue, and the time that has elapsed since the recommendation was made.

Unsatisfactory—Progress is unsatisfactory, given the significance and complexity of the issue, and the time that has elapsed since the recommendation was made.

The Secretariat implemented an action plan, improved its guidance, and improved reporting

2.33 In its 7th report to the House of Commons in 2008, the Public Accounts Committee (PAC) made recommendations based on the results of our 2006 chapter on large IT projects. The government agreed to act on these recommendations.

2.34 PAC recommended that the Secretariat develop an action plan to address our recommendations. The Secretariat submitted its action plan to PAC and has implemented it, including the completion of the Policy Framework for the Management of Assets and Acquired Services and the new Policy on the Management of Projects. It has also made significant improvements to its guidance.

2.35 PAC also recommended that the Secretariat direct departments and agencies to provide financial and performance information in the reports they submit to Parliament on IT projects that are expected to cost over $10 million, including

  • original and current estimated total costs,
  • costs incurred to date,
  • the expected completion date, and
  • the intended outcomes.

2.36 In response, departments have included two tables in their 2009–10 departmental performance reports, entitled Status Report on Major Crown/Transformational Projects and Status Report on Projects Operating with Specific Treasury Board Approval.

Progress in key management areas

2.37 This section presents our findings of the progress the government made against the commitments it made to address recommendations in our 2006 Report to Parliament. These commitments related to four key management areas—governance, business cases, organizational capacity, and project risk management. We assessed the progress made since 2006 of four large IT projects: Expenditure Management Information System, Global Case Management System, Integrated Revenue Collections, and Secure Channel. We also reviewed an additional project, the Temporary Resident Biometrics Project, which was implemented since our 2006 audit. Since the Secure Channel project was completed in 2004, we examined only whether defined benefits noted in its business case have been achieved and measured. The project history in Exhibit 2.3 shows that departments have significantly reduced the size and scope of certain projects. For all the projects we reviewed, budgets have increased by approximately $649 million since their inception, and time frames have also increased significantly.

Exhibit 2.3—The size and scope of certain projects have changed significantly

Department or Agency and Project Description Initial 2006 2010
Schedule Budget Schedule Budget1 Schedule Budget1 Change in Scope/Deliverables
Treasury Board of Canada Secretariat

Expenditure Management and Information System: An integrated financial system designed to support all aspects of government expenditure management.

2000–Ongoing $16.2M 2000–Ongoing $53.7M 2000–2007 $52.8M $50.5M (actual) Originally 4 phases. 2 delivered in 2007, and 2 phases deferred indefinitely.
Citizenship and Immigration Canada

Global Case Management System: A multi-year program to replace Citizenship and Immigration Canada's outdated business systems with an integrated system that will improve all of its client operations in Canada and abroad.

2000–2005 $194.8M 2000–2007 $242.8M 2000–2011 $387M Scope significantly reduced in 2008, with the remainder of project deferred until after March 2011.
Canada Revenue Agency

Integrated Revenue Collections: A new system and approach consisting of a series of solutions and tools used to prioritize and allocate collections workloads according to a risk-based approach.

2001–2004 $2.5M 2001–2012 $147M 2001–2014 $144M Scope clarified and increased significantly.
Public Works and Government Services Canada and Treasury Board of Canada Secretariat

Secure Channel: A common, secure infrastructure that ensures Canadians can use the Government On-Line single window to conduct online transactions safely and effectively.

1999–2002 $96M 1999–2004 $377M
(actual)
N/A N/A N/A—Implemented in September 2004.
Citizenship and Immigration Canada

Temporary Resident Biometrics Project: A project designed to capture fingerprint, photo, and identity information and business processes, to prevent dangerous and inadmissible people from entering Canada.

N/A N/A N/A N/A 2008–2013 $180M Scope reduced to 25% of target population, based on risk.
1 Revised costs reflect changes in project’s scope
Governance structures remain problematic for two of four projects we examined

2.38 Governance at the project level focuses on delivery of business change at an affordable cost with an acceptable level of risk. Treasury Board policies governing investment planning and projects stipulate that the sponsoring department or agency of an IT investment project is responsible for establishing and controlling the costs and risks and for ensuring that the project benefits the organization and meets a defined business need. The sponsoring department is responsible for ensuring the appropriate capacity will be in place to manage and deliver the proposed project and to show that it is making a sound investment decision—based on objective, unbiased information. A sound governance framework at the project level is critical to fulfilling these responsibilities.

2.39 In 2006, we examined whether the processes departments used to approve and manage seven selected large IT projects increased their likelihood of success. We found that four of the projects had deficient governance frameworks. Even though the governance structure was defined for all projects, we found that not all responsibilities were carried out and not all key business issues were reported and resolved.

2.40 As previously noted, the Treasury Board of Canada Secretariat has put in place new Treasury Board policies and tools to assist departments in approaching IT projects from a business perspective rather than simply a technology perspective. This business perspective would place appropriate attention on planning the project, managing the necessary transition, and involving all key stakeholders. We looked at whether departments have implemented this approach for their projects.

2.41 Expenditure Management Information System (EMIS). In 2000, the Secretariat initiated EMIS to replace five legacy systems that were originally created in the mid-1980s to prepare the annual budgetary estimates for Parliament’s approval. As we reported in 2006, the project’s scope was expanded significantly to include the building of a new comprehensive system that would provide an overall view of the government’s financial performance. By 2006, the newly defined project had four phases and its budgeted expenditures had gone from $16.2 million to $53.7 million. In 2007, the Secretariat scrapped the two components it had built between 2000 and 2006 and started over, at a cost of $34.5 million, because the components were found to be unreliable. Since 2007, the Secretariat has completed the first two phases of the project and has deferred indefinitely the last two phases.

2.42 In 2006, we reported that this project had no clear governance and accountability structure in place. As well, there was no monitoring of key performance indicators or status of deliverables. In our current audit, we found that, since 2006, most of the governance weaknesses have been addressed. A clear governance structure was in place, including regular dashboard reporting to a senior steering committee that included financial updates and information on risk and progress. However, we found little evidence that senior management had analyzed the risks and impact of deferring the last two phases indefinitely. Without building the remaining functionalities, the Secretariat will be unable to achieve the overall project’s objectives.

2.43 In its 4th report to the House of Commons in 2008, the Public Accounts Committee (PAC) recommended that the Secretariat provide an update on the implementation of EMIS and that it appoint a blue-ribbon panel to examine financial information systems with a goal of improving spending decisions. The government agreed to have the Secretariat report to PAC on the status of EMIS in July 2008. However, when it did, it did not report the full extent of delays in completing the second phase or the possibility of deferring the remaining two phases indefinitely. In lieu of appointing a blue-ribbon panel, the government indicated that the Secretariat would engage external advisers to provide technical and strategic advice on EMIS. We found that the technical advice, related to the subsequent phase of EMIS, was provided and that, more recently, an advisory committee was created to provide strategic advice.

2.44 Recommendation. The Treasury Board of Canada Secretariat should analyze the risks and impact of deferring the last two phases of the Expenditure Management Information System (EMIS) indefinitely and take the necessary action consistent with the Secretariat’s priorities and future objectives for the project.

The Secretariat’s response. Agreed. The Secretariat will complete the documentation of the decision to cancel the last two phases originally planned for the EMIS project by April 2011. The risks and benefits of the decision have been assessed, and the decision is consistent with the Secretariat’s priorities and objectives.

2.45 Global Case Management System (GCMS). In 2000, Citizenship and Immigration Canada (CIC) initiated the GCMS with the business objective of improving security and efficiency in the delivery of immigration and citizenship activities. The GCMS was to replace 14 legacy and aging systems that did not work well together and placed CIC in a position of serious risk. GCMS is based on one integrated case-management system that provides an end-to-end view of the client continuum from first contact to final disposition. The project’s original cost was $195 million, and it was to be completed by 2005.

2.46 In 2006, we found that the project’s governance structure was not well established. This was further confirmed by an independent review that was conducted in the same year, which noted that the governance mechanism relied on the efforts of individuals and that the project management office lacked the authority to challenge the state of deliverables. It recommended that a suitable governance process be put in place for both CIC and the Canada Border Services Agency (CBSA) to use GCMS as a shared resource.

2.47 CIC has made progress in addressing weaknesses in the GCMS governance. For example, it has established clear lines of accountability by naming the Assistant Deputy Minister, Corporate Services, as project leader; it has appointed an experienced business director to represent the interests of CIC and CBSA; and a senior interdepartmental committee exercises project oversight through, among other things, the review of executive project dashboards and project summaries, and the monitoring of major issues and risks.

2.48 Since our 2006 audit, we found that CIC significantly improved its ability to pursue the delivery of GCMS, as seen in the current phase of the project, Release 2, which focuses on immigration processing. This phase has received close scrutiny by senior CIC committees and has been reporting regularly on its progress. As well, the deployments to date have been successful. Current project dashboards show that the current phase will be delivered on time and slightly below budget.

2.49 The GCMS business objectives that were first identified in 2000 were reconfirmed in several key documents throughout the life of the project. However, the current phase is set to end on 31 March 2011, and a significant amount of work still remains. CIC has assessed that about 25 to 31 percent of GCMS functionalities will still not be developed. The GCMS work that will remain is mainly related to extending the GCMS capability for domestic enforcement and immigration services at ports of entry and inland offices. As a result, many of the legacy systems that were to be replaced will continue to be used after project closing. For example, the Field Operations Support System (FOSS), a 30-year-old system critical to the National Immigration Program, which is considered high risk due in part to integration issues, will not be decommissioned until the 2015–16 fiscal year.

2.50 Two independent reviews, in 2007 and 2009, recommended that CIC continue to focus on future requirements and that it start planning subsequent phases of GCMS before completing the current phase. As well, in its 2009–2012 IMTB (Information Management and Technologies Branch) Integrated Business and Resource Plan, CIC stated that it would produce a business case and funding proposal for the remaining work. Although discussions have occurred at senior levels, they have not clearly addressed the issue of the outstanding work and its impact on the business objectives. As well, some documents have been initiated but they are still in draft form, have not been approved, and include figures that are not supported with analysis.

2.51 Recommendation. Citizenship and Immigration Canada should prepare a plan to address the costs and risks related to the remaining work to be done after completion of Release 2 of the Global Case Management System (GCMS) in order to achieve the project’s overall business objectives.

The Department’s response. Agreed. Over time, the implementation of the GCMS project has been adapted to meet the changing business and operational needs of Citizenship and Immigration Canada (CIC). The overall project plan was amended in 2008 to focus on overseas immigration processing. GCMS Release 2 will be successfully delivered worldwide by March 2011, fulfilling the scope of the Major Crown Project. CIC is preparing a five-year Roadmap to achieve its overall business needs, which have evolved since the original business objectives of GCMS were established. As we move forward, we will continue to build on the GCMS platform and address any risks related to the changes to the original scope. The first year of this Roadmap will be funded through the CIC Business Planning and Investment Planning process, and the next deployment is scheduled for summer 2011. As business priorities shift and change quickly, the yearly process of approval will be followed for GCMS over the coming years. A portion of immigration inland processing will be implemented in the Case Processing Centre Mississauga and CIC Etobicoke in March, with minimal IT changes to the system.

2.52 Integrated Revenue Collections (IRC). In 2001, the Canada Revenue Agency (CRA) initiated IRC to help prioritize and allocate collections work according to level of risk. This project had an original budget of $2.5 million with an expected completion date of 2004; however, it did not progress for many years. By fall 2005, senior management decided to put the project on hold while a new business case was developed. By that time, the project had spent $13 million and had few deliverables.

2.53 In 2006, we reported that several measures had been taken to improve the project’s governance. For example, the CRA established a new senior committee, the Resource Investment Management Committee (RIMC), in early 2006, to oversee the management and progress of major IT projects, including IRC. This committee is chaired by the Commissioner, and senior executives representing internal stakeholders from other sectors of the Agency are members. Its role is to provide assurance that all resource and investment decisions support and are fully integrated with Agency direction and strategies.

2.54 Since 2006, RIMC has monitored IT projects such as IRC through progress reports and other similar presentations. It also requires project managers to develop a full business case and project sponsors to identify the expected benefits and, after implementation, confirm that they have realized these benefits. Other governance improvements include establishing a project management office and dividing the IRC project into more manageable components.

2.55 Temporary Resident Biometrics Project. Led by Citizenship and Immigration Canada (CIC), the biometrics project aims to enhance security by capturing photographs and fingerprints of temporary resident applicants to ensure individuals posing safety and security threats do not gain access to Canada. The project is dependent on multiple stakeholders, including the Canada Border Services Agency (CBSA), which will verify travellers’ identities against previous captured biometrics, such as fingerprints, at selected ports of entry. The Royal Canadian Mounted Police (RCMP) will cross-reference fingerprints against criminal records databases as part of its Real-Time Identification (RTID) Project.

2.56 The project is governed by interdepartmental committees composed of senior managers from CIC, the RCMP, and CBSA, as well as representatives from the Secretariat and other departments. These committees meet regularly and provide a forum for members to raise and address issues. Our review of committees’ minutes and planning documentation reveals that the biometrics project is a high-risk project that has been in “red status” (meaning that significant course correction may be required) since April 2009, when projects began to maintain dashboards.

2.57 The biometrics project has been in red status for various reasons. First, the project received only a portion of required funding under Budget 2008: $180 million instead of the required $208 million. Second, the project’s revised scope, designed to enable delivery for $180 million, was still being finalized in December 2010. Third, the project is approximately a year behind in receiving approval and issuing a request for proposal. We found that CIC has mechanisms in place to alert senior management and the Secretariat to these problems in a timely manner, and that delivery partners have used this information in their decision to continue with the project.

2.58 In summary. We found that overall progress on the departments’ efforts to place more attention on planning large IT projects, managing the necessary transition, and involving all key stakeholders has been unsatisfactory; two of the four projects, the Global Case Management System and the Expenditure Management Information System, have significant governance issues (Exhibit 2.4).

Exhibit 2.4—Progress in addressing our recommendation on improving project governance and accountabilities

Recommendation Progress
Government departments and agencies should improve their internal quality reviews of IT projects. Senior departmental executives should review the key decision documents that are produced to support the IT project and ensure that the analysis is thorough and supportable before signing off on the submission.

(Recommendation 3.61 of the 2006 November Report of the Auditor General of Canada, Chapter 3, Large Information Technology Projects)

Satisfactory
(Biometrics, IRC)

Unsatisfactory
(GCMS, EMIS)

Satisfactory—Progress is satisfactory, given the significance and complexity of the issue, and the time that has elapsed since the recommendation was made.

Unsatisfactory—Progress is unsatisfactory, given the significance and complexity of the issue, and the time that has elapsed since the recommendation was made.

Most business cases do not clearly identify and measure benefits

2.59 A business case is the foundation for sound investment decisions. For IT projects, the business case explains the rationale for the project and the results that the project is expected to produce to meet an organization’s business needs. It also provides an analysis of all the costs, benefits, and risks associated with the proposed investment. Furthermore, the business case must define what constitutes success, which will be used to measure the project’s success later on.

2.60 In 2006, we examined seven projects to determine whether departments had clearly defined the business needs they expected to fulfill. We reported that, for five of the projects, the business cases were incomplete, out of date, or contained information that could not be supported. In this audit, we examined whether the business cases have improved for four of those projects, as well as for the new Temporary Resident Biometrics Project.

2.61 Expenditure Management Information System (EMIS). In 2006, we reported several deficiencies in the Expenditure Management Information System’s original 2004 business case, notably, that it contained incomplete or unclear business objectives and timelines and did not include an options analysis.

2.62 In 2006, the Secretariat approved a new business case that reduced significantly the scope of this project. The new business case contained a detailed options analysis and it set out clear business goals and measurable performance objectives and planned results, which were measured against actual outcomes. Key stakeholders were involved in the decisions made on the reduced scope, and it was completed on time and within budget.

2.63 In July 2008, the government agreed to provide the Public Accounts Committee (PAC), by the end of 2008, with a close-out report summarizing lessons learned from the original EMIS project; it said in its response that these were taken into account in the planning and delivery of the newly scoped project. The report would indicate how the Secretariat would apply lessons learned, including the deficiencies in the business case that were identified in our 2006 audit. Although the Secretariat has completed this report, it has not provided it to PAC. In addition, the government agreed that it would make project close-out reports, as well as their filing with CIOB, mandatory for all large IT projects. We noted that the Secretariat advised PAC in March 2010 that its project evaluation guidelines indicate that close-out reports are to be prepared within three months of completion of a project. However, the use of these evaluation guidelines is not mandatory.

2.64 Global Case Management System (GCMS). In 2006, we reported that, since project initiation in 2000, Citizenship and Immigration Canada (CIC) struggled with a number of issues, including changes in project scope, difficulties accessing funds, a lack of people with the required skills, and delays in procurement. We also found that the original business case did not present a detailed analysis of options and did not quantify the expected business outcomes and benefits. During that time period, the budgeted cost escalated from an original budget of $195 million to $243 million; however, there was no update of the original business case to reflect these significant changes. As a result, the implementation of the citizenship module used most of the original budget of $195 million and took as much time as the whole project had been expected to take.

2.65 CIC issued a revised business case in 2007, which addressed scope and capacity issues and, more specifically, focused on the immigration line of business. It made the case for another budget increase of $144 million for a total of $387 million and a significant reduction of scope. We found that the business case included an improved options analysis; however, it reiterated almost the same expected business benefits as those identified in the original business case in 2000. Again, it did not quantify them to make them ultimately measurable. In January 2008, the Secretariat, in its challenge role, suggested to CIC that it needed to better define the project business benefits and provide measures. However, CIC has not acted on this advice and has not prepared a benefit measurement plan. As a result, it will be difficult for it to provide performance information that shows whether the GCMS will achieve the intended business benefits.

2.66 In 2009, CIC updated the business case again to reconfirm the relevance of the direction taken in 2007. Although it also mentioned the business benefits at a high level, it did not quantify them or define measures. In addition, the significant cost avoidance from the replacement of the 14 disparate legacy systems by one integrated system mentioned in the original business case was no longer mentioned after 2007.

2.67 Integrated Revenue Collections (IRC). In 2006, we reported that IRC was one of several projects in which the business case was incomplete and out of date.

2.68 The Canada Revenue Agency prepared a revised business case in 2007 and recommended it for approval by the Resource and Investment Management Committee (RIMC). We found this business case to be complete, with the exception of the section on benefits realization, which required more measurement detail. The main benefit identified in the business case was an estimated $700 million in annual savings from program effectiveness, improved accounts receivable, and revenue collection productivity gains. Of this total, $200 million in annual benefits was specifically identified as the result of the implementation of the first phase of the project, due to be completed by June 2012. However, the business case lacked details on how these results will be achieved and contained no information on the remaining $500 million.

2.69 In late 2009, a benefits measurement plan for Phase 1 was prepared and presented to RIMC for approval. The RIMC did not approve the plan at that time, as the plan did not outline the benefits resulting from the implementation of the first phase of the IRC project. A revised version has since been prepared and was to be presented to RIMC in December 2010. This revised plan contains more details on how the $200 million in annual gains will be achieved, including benchmark information and data collection. It also provides a timeline of when the project will start to realize the benefits. However, this timeline still lacks details on when this target will be achieved and how progress will be measured. In addition, as a result of a change in investment decisions for the second phase of IRC, the remaining $500 million in annual benefits is no longer applicable and any future benefits are to be defined at each additional phase.

2.70 Secure Channel. The Secure Channel is a complex, innovative, multi-departmental project that provides Canadians and businesses with secure and private access to online government services. This common secure infrastructure ensures that Canadians can use a single window to conduct online transactions safely and effectively. In 1999, the Secretariat initiated the Secure Channel and was responsible for its oversight, strategic direction, and decision making. Public Works and Government Services Canada (PWGSC) was responsible for project management and also became responsible for operations in 2004. The project’s development had a total cost of $377 million, of which approximately $285 million can be attributed to the development of the authentication (epass) service, one of five main services that the Secure Channel provides. The project’s 2005 business case identified benefits and related targets to be achieved over a five-year period, and included projections on the average Internet transaction volumes, and significant cost savings and cost avoidance. As well, the business case identified program administration benefits estimated at $5.2 billion (unaudited), which could not be achieved without this secure infrastructure.

2.71 Since our 2006 audit, the Secretariat has made the use of all the Secure Channel Services mandatory, by amending the Treasury Board Common Services Policy. It did this to ensure that federal departments would have to use Secure Channel services and thus these services could be funded on a cost-recovery basis. The cost of operating the Secure Channel from October 2004 to March 2010 was $483 million. It is estimated that another $134 million will be spent between April 2010 and December 2012. This will bring the total expenditure to $617 million, of which the epass service component is $367 million.

2.72 Human Resources and Skills Development Canada (HRSDC) and Canada Revenue Agency (CRA) programs accounted for over 90 percent of the epass service annual volumes. Because of this, PWGSC required that they make binding commitments on the number of projected transactions and used these to calculate the amount that each department would be required to pay on a yearly basis. These costs were binding and payable whether or not HRSDC or CRA achieved their projected transaction volume levels.

2.73 The HRSDC My Service Canada Account (MSCA) started using the epass service in November 2006. Soon after, it started to encounter performance and system availability issues during periods of high transaction volumes. As a result, HRSDC stopped using the epass service for its MSCA in February 2007. The MSCA generates, on average, 12 million transactions annually. PWGSC indicated that it had addressed these issues by August 2007 and formally advised HRSDC in February 2008. HRSDC then reintegrated the MSCA in May 2008 when service disruption to Canadians would be kept to a minimum.

2.74 In its initial letter of intent in November 2006, CRA indicated its concern that the cost of the epass service was substantially more than its costs to operate and maintain its own in-house solution that met its security and operational requirements. Furthermore, when determining its service volume commitments in 2006, CRA decided to exclude electronically filed tax returns, which generate, on average, 12 million transactions annually, as including them was not cost effective. The Agency has recently built an in-house service that provides it with a different level of security and privacy that meets its requirements. CRA stopped using the epass service as of October 2010.

2.75 In early 2007, the Secretariat approved a revised Secure Channel business case. This business case focused on the technical business requirements, such as security and privacy, timely availability, a single window, and interoperability between departments, programs, and services. The business case also indicated that there would be operational benefits related to departmental take-up and transaction volumes, as well as financial benefits consisting of cost savings and cost avoidance.

2.76 The 2007 business case does not identify specific measures, timelines, and targets for the technical requirements or operational and financial benefits. This is in sharp contrast to the previous business case in 2005, which identified the business benefits with specific targets and measures. Finally, there is no benefit measurement plan and no indication of who would be responsible for measuring benefits. Given the experience with the two largest user departments on its most significant service offering, it would have been important to clearly indicate in the new business case a target take-up rate or the level of financial benefits expected to be achieved.

2.77 The Secretariat’s Chief Information Officer Branch (CIOB) is currently considering a new strategy for the provision of authentication services to departments and agencies after the current epass service contract ends in December 2012. As part of this review, the CIOB has acknowledged that the current epass single architecture, with the bundling of security services, resulted in a solution set that was perceived to be excessive and too costly for most applications and that the take-up by departments was not achieved as originally estimated. The CIOB has developed a new business case that outlines options and cost estimates for a more flexible authentication solution for departments and agencies.

2.78 In its 7th report to the House of Commons in 2008, the Public Accounts Committee recommended that PWGSC conduct a cost-benefit analysis of the continued use of Secure Channel by 31 December 2008, and that PWGSC also provide meaningful results-based information in its annual departmental performance report (DPR). In June 2009, the government agreed to both recommendations and provided the Committee with its SC/EPASS Evolution Analysis Report, which gives an update on progress with regards to cost recovery of all Secure Channel services. PWGSC has also provided on an annual basis a number of performance measures relating to the usage of the Secure Channel epass service in its DPR.

2.79 We reviewed these performance measures and found they included the number of programs and departments using the service, transaction volumes, per transaction cost, and capacity use, as well as the number of epasses issued to businesses and Canadian individuals. However, these measures lacked context. There were no targets against which this information can be compared to assess performance. For example, the last report stated that 8.2 million epasses have been issued to date. This was represented as showing that Secure Channel has achieved full capacity. We note that there is no discussion of how this compares with the architectured capacity of 30 million epasses. In addition, the actual number of transactions between April 2007 and December 2009 of 47.9 million was not compared with the minimum transaction volume commitment of 93 million made for that period. As well, this information was not compared against the annual estimated capacity of 45 million transactions per year, which an independent review in 2005 had stated was needed for the epass service to be cost effective. It is difficult to properly assess the performance of the Secure Channel service without meaningful comparisons against plans and targets.

2.80 Temporary Resident Biometrics Project. Citizenship and Immigration Canada estimated the initial cost for implementing the biometrics project, including the CBSA and RCMP components and interfaces, at $208 million. However, we note that the 2008 Budget did not provide most of the funding for the CBSA component and approved funding of only $180 million. Because of this funding shortfall, CBSA indicated that it would not be able to complete some of its development activities and would have difficulty supporting ongoing operations of fingerprint enrolment and systematic verification at all ports of entry. The biometrics project’s 2009 business case took into account the reduced funding; however, it did not clearly articulate what functionalities would be reduced and what benefits would not be realized.

2.81 In spring 2010, CIC made a strategic change to its service delivery model, based on lessons learned from other jurisdictions. This changed the project costs significantly as it was going from a user-pay model to one where the Government of Canada enters into contracts with third-party service providers and pays for the service. While this approach enhanced oversight and control, it also shifted the financial burden from applicants to the Government of Canada.

2.82 Because of the revised funding method and change in strategic direction, CIC developed a new business case in September 2010, proposing a reduced-scope option for the project. Under this option, CBSA will be conducting only discretionary fingerprint verification at selected ports of entry. This reduced scope option was still being finalized in December 2010.

2.83 We found that the September 2010 business case contains well-defined objectives and an options analysis, and represents an effort on the part of CIC to develop a viable and focused scope for the project where deliverables can be achieved. The project team has also begun to develop a framework for monitoring performance. While the framework does not contain performance targets or timelines, it does include key performance indicators, data sources, and mechanisms for reporting performance.

2.84 In summary. The Secretariat has met its commitments and developed and published business guidance to be used by departments in preparing their business cases (paragraph 2.22 and Exhibit 2.1). However, we found that three of the five projects we examined in this audit—GCMS, IRC, and Secure Channel—continue to have deficiencies regarding identification and measurement of benefits (Exhibit 2.5).

Exhibit 2.5—Progress in addressing our recommendation on business cases

Recommendation Progress
Before seeking effective project approval, departments and agencies should prepare a business case that includes, at a minimum, precise and measurable objectives; a full analysis of options, benefits, costs, and risks; and an implementation plan.

(see Recommendation 3.89 of the 2006 November Report of the Auditor General of Canada, Chapter 3, Large Information Technology Projects)

Satisfactory
(EMIS, Biometrics)

Unsatisfactory
(GCMS, IRC, SC)

Satisfactory—Progress is satisfactory, given the significance and complexity of the issue, and the time that has elapsed since the recommendation was made.

Unsatisfactory—Progress is unsatisfactory, given the significance and complexity of the issue, and the time that has elapsed since the recommendation was made.

2.85 Recommendation. When departments and agencies develop a business case, they should include a benefit measurement plan that

  • contains specific quantifiable benefits including clear benchmarks,
  • assigns responsibilities for delivering identified benefits, and
  • includes a process for measuring and reporting benefits.

The Canada Revenue Agency’s (CRA’s) response. Agreed. The Agency believes that since the end of the audit (31 October 2010) it has addressed the recommendation in the following manner:

As noted by the Auditor General, a benefits measurement plan for Integrated Revenue Collections (IRC) Phase 1 had been completed by the CRA in 2009. The benefits measurement plan for IRC Phase 1 was further refined in 2010 to reflect annual gains to be achieved, benchmark information, and a data collection strategy, including when benefits would start to be realized, and who would be responsible for the delivery of the identified benefits. The 2010 revised plan was recently approved and work is ongoing in respect of measuring and reporting on the benefits. In fact, the approval process in place at the Agency requires that a benefits realization plan be prepared as part of business cases for large IT-enabled projects, which is an established best practice.

Citizenship and Immigration Canada’s (CIC’s) response. Agreed. Future projects will leverage the CIC Performance Measurement Framework and Operational Baselining Strategy in order to quantify, measure, and report on benefits.

Public Works and Government Services Canada’s (PWGSC’s) response. Agreed. PWGSC is committed to adhering to applicable Treasury Board policies, as well as Treasury Board of Canada Secretariat Chief Information Officer Branch guidelines and standards on the development of business cases. In cases where PWGSC is responsible for the implementation of large, horizontal IT projects with government-wide or multi-departmental impacts, the Department will continue to work cooperatively with the Secretariat and client departments to develop business cases that clearly assign accountabilities for benefit measurement plans and for tracking and reporting of benefits where applicable and justified.

Most of the large IT projects we audited have taken steps to address capacity issues

2.86 Organizational capacity refers to the technical and managerial ability to deliver an IT project. It also refers to the ability of the entire organization to improve the way it does business by using all of a system’s capabilities.

2.87 In 2006, we examined whether, for the seven projects we selected, departments and agencies had clearly demonstrated that their organization was ready to accept the business transformation that came with the project and whether they had the capability and commitment to successfully deliver the project. We reported that four of the departments lacked the appropriate skills and experience to manage the projects or the capacity to use the system to improve the way they deliver their programs.

2.88 The Secretariat committed to develop and implement an Organizational Project Management Capacity Assessment Tool and a “Project Complexity and Risk Assessment Tool” to assist departments to better understand their capacity to manage projects and to better ascertain the level of risk and complexity of proposed projects (paragraph 2.32). We examined whether the projects included in this audit took steps to improve their organizational capacity with, among other things, the use of these tools.

2.89 Expenditure Management Information System (EMIS). EMIS was designed to support all aspects of government expenditure management by providing information on the cost of meeting the government’s priorities and achieving particular results. The favoured delivery approach was through an in-house system development.

2.90 After spending over $34 million on this project since its inception in 2000, the Treasury Board of Canada Secretariat realized that the project had to be re-scoped to accommodate capacity issues and to adopt a more widely applied technology. There was also a need to substantially reduce the risks associated with human resource issues, both within the Secretariat and from professional services contracts. We found that business stakeholders were consulted; however, the Secretariat did not prepare an organizational capacity assessment prior to revising the project deliverables after 2006.

2.91 Global Case Management System (GCMS). In 2006, we reported that the GCMS’s organizational capacity was deficient. The project had experienced untimely shortages of key resources, and Citizenship and Immigration Canada had no previous experience in developing and implementing a project of a similar size and complexity. Since then, CIC has assessed its capacity to deliver such a project through independent reviews and internal audits. Also, it has produced plans for business preparation, transition, and operational phases. Moreover, instead of attempting to deliver the project as a whole in one implementation, the Department adopted a phased approach to implementing GCMS. This approach has provided the organization with greater capacity to implement and manage the change.

2.92 Integrated Revenue Collections (IRC). Since our 2006 audit, the Canada Revenue Agency has had to address certain organizational capacity issues. In 2006, the Agency prepared initial staff resource plans and staffing concerns were discussed at senior committees. As a result, CRA included in the 2007 business case a new human resource strategy concentrating on project challenges related to capacity and availability of expert resources. This helped CRA address the resource shortfall previously identified.

2.93 Temporary Resident Biometrics Project. In June 2009, an independent review of the biometrics project recommended that the project team strengthen its project management expertise. The project team hired two directors to address this gap. As well, in response to weaknesses identified in the Organizational Project Management Capacity Assessment completed in October 2009, the project team took steps to define and group the discrete elements for the project, to better describe deliverables and identify tasks required to complete them. In August and September 2010, the project team took further steps to determine whether Citizenship and Immigration Canada had the right mix and appropriate number of people to ensure effective implementation. The Department has not yet approved the project’s human resource management plan.

2.94 In summary. We found that for three of the four projects included in this portion of the audit, progress on improving organizational capacity was satisfactory. EMIS did not assess organizational capacity as required (Exhibit 2.6).

Exhibit 2.6—Progress in addressing our recommendation on organizational capacity

Recommendation Progress
At the start of a project, departments and agencies should clearly demonstrate that their organization is ready to accept the business transformation that comes with the project and has the capability and commitment to successfully deliver the project. Specifically, departments and agencies should analyze their track record in
  • completing projects of similar size and complexity;
  • completing skills appraisals and plans to address shortfalls;
  • planning for business preparation, transition, and operational phases; and
  • considering key stakeholder buy-in and commitment.

(Recommendation 3.98 of the 2006 November Report of the Auditor General of Canada, Chapter 3, Large Information Technology Projects)

Satisfactory
(GCMS, IRC, Biometrics)

Unsatisfactory
(EMIS)

Satisfactory—Progress is satisfactory, given the significance and complexity of the issue, and the time that has elapsed since the recommendation was made.

Unsatisfactory—Progress is unsatisfactory, given the significance and complexity of the issue, and the time that has elapsed since the recommendation was made.

Most of the large IT projects we audited have taken steps to address project risk management issues

2.95 Sound project management ensures that projects produce successful outcomes, within a specified time. A key component of project management is effective risk management, which involves assessing the potential impact of risks, continuously monitoring the risks, and dealing with potential risks before they arise.

2.96 In 2006, we found that the quality of project risk management varied widely from project to project—from good to seriously flawed. Although we found that four of the seven projects we audited were well managed, the quality of project risk management varied widely. Those with weak project management practices experienced long delays and large cost overruns.

2.97 The Secretariat developed a Project Complexity and Risk Assessment Tool that includes an assessment of how departments manage project risks. None of the projects we examined in this audit were subjected to this tool because departments have until April 2012 to implement it.

2.98 Expenditure Management Information System (EMIS). EMIS has successfully completed two of its four phases on time and within budget. We observed that project risks had been adequately managed. For example, the project’s 2006 Business Case Risk Management Plan indicated preliminary identified risks, including, for each risk, probability of occurrence, impact, severity, and risk response (risk mitigation).

2.99 Risk information, such as executive project dashboards, was regularly reported to the EMIS Steering Committee. All risks related to the first two phases of the project were addressed.

2.100 Global Case Management System (GCMS). For the current phase of GCMS, ending on 31 March 2011, Citizenship and Immigration Canada has a risk management approach in place, consisting mainly of a risk register capturing major existing, anticipated, and emerging risks that are assigned to a specific owner and analyzed by likelihood and impact.

2.101 The risk register contains action plans and status updates and is reviewed on a regular basis by the Senior Review Board. However, the risks related to the work that remains after this phase is complete have not been documented (paragraph 2.49).

2.102 Integrated Revenue Collections (IRC). We reviewed the risks identified in the risk management framework document for the IRC project. We selected specific risks initially recorded in both the 2006 and 2007 business cases, and assessed how these were being managed. We found that the Canada Revenue Agency has implemented a formal assurance process for assessing the adequacy of its risk management. In addition, regular progress reports that include summary information on the status of current project risks are submitted to the Resource Investment Management Committee for review and feedback.

2.103 Temporary Resident Biometrics Project. Given that the biometrics project is complex and high risk, mechanisms for identifying, assigning ownership of, addressing, and communicating risks to partners are essential. We found that the biometrics project has mechanisms for identifying and assigning ownership of risks, including a risk register and an issue-tracking log, and that it communicates these risks through its governance structure.

2.104 However, we found weaknesses in the project’s mechanisms for addressing risks and analyzing their impacts in a comprehensive manner. For example, we found that, while delivery partners were first made aware of the project’s key risk—delays in the RCMP Real-Time Identification (RTID) project—in April 2009, the project team did not develop a contingency plan until November 2010. We found that this contingency plan provides a general overview of costs and options, but does not provide specific details on how the project team would implement each option. This gap is problematic, because the timely implementation of the RTID project, which was on hold in May 2010, is critical to the successful implementation of the biometrics project. Without a functioning RTID, Citizenship and Immigration Canada will not be in a position to assess criminality to ensure applicants posing a risk to the safety and security of Canada are denied entry.

2.105 We also found that the project team had to redevelop key planning documents due to the funding shortfall and changes to scope, and is now experiencing delays of about a year as a result. However, the project team has not analyzed the impacts of these delays in a comprehensive manner. Given these developments, the project team’s ability to deliver on the March 2013 implementation date is at risk.

2.106 In summary. We found that overall progress on the implementation of risk management for the projects we reviewed has been satisfactory, although progress is unsatisfactory for the Temporary Resident Biometrics Project (Exhibit 2.7).

Exhibit 2.7—Progress in addressing our recommendation on project risk management and reporting

Recommendation Progress
Departments and agencies should
  • comply with the Treasury Board Secretariat project management guidance,
  • closely manage the project risks,
  • monitor key success factors, and
  • regularly report to management any significant internal or external event that may prevent the expected project benefits or result in cost overruns or in the project not being delivered on time.

(Recommendation 3.111 of the 2006 November Report of the Auditor General of Canada, Chapter 3, Large Information Technology Projects)

Satisfactory
(GCMS, IRC, EMIS)

Unsatisfactory
(Biometrics)

Satisfactory—Progress is satisfactory, given the significance and complexity of the issue, and the time that has elapsed since the recommendation was made.

Unsatisfactory—Progress is unsatisfactory, given the significance and complexity of the issue, and the time that has elapsed since the recommendation was made.

Conclusion

2.107 Overall, the government’s progress on its commitments to our 2006 recommendations is unsatisfactory (Exhibit 2.8). In the area of project governance, two of the four projects we examined (the Expenditure Management Information System and the Global Case Management System) have had important deliverables deferred without full analysis of the impact, costs, and risks of not completing these projects. In the area of business cases, three of the five projects we examined (the Global Case Management System, Integrated Revenue Collections, and Secure Channel) have not quantified benefits to be achieved or measured them.

Exhibit 2.8—Overall progress against our recommendations for the projects

Project Governance Business Cases Organizational Capacity Project Risk Management
Expenditure Management Information System Unsatisfactory Satisfactory Unsatisfactory Satisfactory
Global Case Management System Unsatisfactory Unsatisfactory Satisfactory Satisfactory
Integrated Revenue Collections Satisfactory Unsatisfactory Satisfactory Satisfactory
Secure Channel N/A Unsatisfactory N/A N/A
Temporary Resident Biometrics Project Satisfactory Satisfactory Satisfactory Unsatisfactory
Overall Rating by Management Area Unsatisfactory Unsatisfactory Satisfactory Satisfactory
Satisfactory—Progress is satisfactory, given the significance and complexity of the issue, and the time that has elapsed since the recommendation was made.

Unsatisfactory—Progress is unsatisfactory, given the significance and complexity of the issue, and the time that has elapsed since the recommendation was made.

2.108 As illustrated in Exhibit 2.8, progress has been satisfactory in the areas of organizational capacity and project risk management. We found that three of the four projects we audited have adequately assessed their technical and managerial ability to deliver the project. We also found that three of the four projects we audited have properly assessed and monitored the potential impacts of risks.

2.109 The Treasury Board of Canada Secretariat’s work in reviewing and challenging large IT projects before they are submitted to Treasury Board for approval has improved their likelihood of success. The Secretariat’s Chief Information Officer Branch has also established several monitoring mechanisms to assess the health of the government’s portfolio of large IT projects since our last audit. However, it is too early to assess the results of the monitoring activity because the Chief Information Officer has only recently (September 2010) provided the assessment to the Treasury Board ministers for their consideration.

The Secretariat has responded. While the Treasury Board of Canada Secretariat accepts the findings of the audit, the Secretariat is of the view that the overall characterization of the government’s progress on the management of large IT projects as “unsatisfactory” is not warranted. On balance, the Secretariat would find that overall progress is indeed “satisfactory,” given the actions taken by the Secretariat related to its policy leadership role and the findings related to the individual IT projects reviewed. The difference of views on the overall rating rests on the weight the Secretariat would accord to the actions presented in this report, which the OAG has found as largely “satisfactory.”

About the Audit

All of the audit work in this chapter was conducted in accordance with the standards for assurance engagements set by The Canadian Institute of Chartered Accountants. While the Office adopts these standards as the minimum requirement for our audits, we also draw upon the standards and practices of other disciplines.

Objectives

The objective of our audit was to determine whether the selected departments and agencies have made satisfactory progress in implementing the recommendations related to the November 2006 Report, Chapter 3, Large Information Technology Projects.

Scope and approach

We examined the Treasury Board of Canada Secretariat’s role in challenging large IT projects and supporting Treasury Board ministers’ oversight, as well as its provision of guidance to departments on effective management and oversight of their IT projects.

We also examined four projects that did not meet all of our audit criteria in the 2006 chapter and we selected one additional project that was approved since our last audit. The projects selected are with the following entities: Canada Revenue Agency, Citizenship and Immigration Canada, Public Works and Government Services Canada, and the Treasury Board of Canada Secretariat.

We also reviewed actions taken in relation to the government’s commitments in response to the recommendations the Parliamentary Standing Committee on Public Accounts made in its 7th and 4th reports to the House of Commons on 25 February 2008 that were relevant to issues within our audit scope.

Criteria

To determine whether the Treasury Board of Canada Secretariat, in its role, is challenging large IT projects and supporting Treasury Board ministers’ oversight, we used the following criteria:
Criteria Sources

The Secretariat has an adequate governance framework for the government-wide portfolio of large IT investments under development.

Policy on Management of Information Technology, section 6.1.4, Treasury Board, 2007

The Secretariat has set a government-wide strategic direction for the management of large IT projects.

  • Policy on Management of Information Technology, section 8, Treasury Board, 2007
  • Directive on Management of Information Technology, section 8.1.2, Treasury Board, 2009

The Secretariat has adequate systems and practices in place to support the successful delivery of large IT projects

Policy on the Management of Projects, sections 1.3 and 5.1, Treasury Board, 2009

The Secretariat has adequately challenged and monitored departmental compliance with its policies, directives, and standards regarding investment plans and project submissions for large IT projects.

Project Management Policy (Monitoring), Treasury Board, 2007

The Secretariat and, more specifically, the Chief Information Officer Branch has in place measures to monitor and report on large IT projects.

  • Project Management Policy (Monitoring), Treasury Board, 2007
  • IT Project Review and Oversight, Treasury Board of Canada Secretariat Chief Information Officer Branch
To determine whether the selected entities have adequate project governance and oversight mechanisms in place for their IT projects, we used the following criteria:
Criteria Sources

The selected entities have adequate large IT project governance and oversight mechanisms in place that are documented and maintained.

  • Policy on the Management of Projects, section 6.1.1, Treasury Board, 2009
  • COBIT PO10.3 Project Management Approach

The selected entities have adequately defined the key roles and responsibilities of the key personnel responsible for the selected large IT projects.

  • COBIT PO10.3 Project Management Approach
  • Val IT IM5 Develop the detailed candidate project business case

The selected entities have in place adequate procedures for monitoring and reporting on the selected large IT projects.

  • Policy on the Management of Projects, section 6.2, Treasury Board, 2009
  • COBIT PO10.13 Project Performance Measurement, Reporting and Monitoring

The selected entities have adequate management plans in place that define, measure, and monitor the realization of benefits for the large IT projects selected.

  • Policy on the Management of Projects, section 6.2, Treasury Board, 2009
  • Val IT IM4 Develop full life-cycle costs and benefits

The selected entities have included the key stakeholders of the large IT projects in their communication plan.

COBIT PO6.5 Communication of IT Objectives and Direction

To determine whether the selected entities have a business case that includes at a minimum: definition of project scope, option analysis, deliverables, investment cost, risk analysis, definition of success criteria, and the alignment of the IT project with the overall organization strategy for large IT projects, we used the following criteria:
Criteria Sources

The selected entities have adequately defined the business goals for their large IT project and how the project will deliver on them.

Business Case Guide, section 1.1.2, Business Need, Treasury Board of Canada Secretariat

The selected entities have identified and measured the critical success factors for each large IT project.

Business Case Guide, section 1.1.4, Business Outcomes, Treasury Board of Canada Secretariat

The selected entities have involved the key stakeholders for their large IT projects in the decision-making process.

Business Case Guide, sections 1.4.2: Stakeholder Analysis, and 2.2, List of Possible Options, Treasury Board of Canada Secretariat

The selected entities have prepared an adequate options analysis case for their large IT project.

Business Case Guide, section 2, Preliminary Options Analysis, Treasury Board of Canada Secretariat

To determine whether the selected entities have a skilled and experienced project management team with leadership, capability, and commitment to deliver the IT project with success, we used the following criteria:
Criteria Sources

Each selected entity has done an adequate assessment of its capacity to deliver its large IT project.

Standard for Organizational Project Management Capacity, Treasury Board of Canada Secretariat

Each selected entity has adequate human resources available to implement its large IT project.

  • Standard for Organizational Project Management Capacity, Treasury Board of Canada Secretariat
  • COBIT PO10.8 Project Resources
To determine whether the selected entities have project management practices that are based on risk management, we used the following criteria:
Criteria Sources

Each selected entity has an adequate process to identify, assess, and assign responsibility for and monitor current and emerging risks for their large IT projects.

  • Integrated Risk Management Framework, Element 3: Practising Integrated Risk Management, Treasury Board of Canada Secretariat
  • COBIT PO10.9 Project Risk Management

Each selected entity has kept their risk management plans up to date for their large IT projects.

COBIT PO10.9 Project Risk Management

Each selected entity has in place an adequate process for managing and monitoring progress and changes to requirements that is based on a risk analysis.

Integrated Risk Management Framework, Element 3: Practising Integrated Risk Management, Treasury Board of Canada Secretariat

Each selected entity has an adequate project re-justification at each major checkpoint (stage) for their large IT project that takes into account risk-based changes in the business strategy, scope, and objectives.

COBIT PO10.6 Project Phase Initiation

Each selected entity has a third-party assurance process for assessing the adequacy of risk management for their large IT project.

COBIT ME4.7 Independent Assurance

Management reviewed and accepted the suitability of the criteria used in the audit.

Period covered by the audit

The audit covered the period between November 2006 and October 2010. However, some documents reviewed go back to 2000. Audit work for this chapter was substantially completed on 29 October 2010.

Audit team

Assistant Auditor General: Nancy Y. Cheng
Principal: Richard Brisebois
Directors: Bernard Battistin (Lead), Tony Brigandi, Greg Boyd, Marie-Claude La Salle

Bridget O’Grady
Jessica L. Perkins
William Xu

For information, please contact Communications at 613-995-3708 or 1-888-761-5953 (toll-free).

Appendix—List of recommendations

The following is a list of recommendations found in Chapter 2. The number in front of the recommendation indicates the paragraph where it appears in the chapter. The numbers in parentheses indicate the paragraphs where the topic is discussed.

Recommendation Response
Progress in key management areas

2.44 The Treasury Board of Canada Secretariat should analyze the risks and impact of deferring the last two phases of the Expenditure Management Information System (EMIS) indefinitely and take the necessary action consistent with the Secretariat’s priorities and future objectives for the project. (2.38–2.43)

The Secretariat’s response. Agreed. The Secretariat will complete the documentation of the decision to cancel the last two phases originally planned for the EMIS project by April 2011. The risks and benefits of the decision have been assessed, and the decision is consistent with the Secretariat’s priorities and objectives.

2.51 Citizenship and Immigration Canada should prepare a plan to address the costs and risks related to the remaining work to be done after completion of Release 2 of the Global Case Management System (GCMS) in order to achieve the project’s overall business objectives. (2.45–2.50)

The Department’s response. Agreed. Over time, the implementation of the GCMS project has been adapted to meet the changing business and operational needs of Citizenship and Immigration Canada (CIC). The overall project plan was amended in 2008 to focus on overseas immigration processing. GCMS Release 2 will be successfully delivered worldwide by March 2011, fulfilling the scope of the Major Crown Project. CIC is preparing a five-year Roadmap to achieve its overall business needs, which have evolved since the original business objectives of GCMS were established. As we move forward, we will continue to build on the GCMS platform and address any risks related to the changes to the original scope. The first year of this Roadmap will be funded through the CIC Business Planning and Investment Planning process, and the next deployment is scheduled for summer 2011. As business priorities shift and change quickly, the yearly process of approval will be followed for GCMS over the coming years. A portion of immigration inland processing will be implemented in the Case Processing Centre Mississauga and CIC Etobicoke in March, with minimal IT changes to the system.

2.85 When departments and agencies develop a business case, they should include a benefit measurement plan that

  • contains specific quantifiable benefits including clear benchmarks,
  • assigns responsibilities for delivering identified benefits, and
  • includes a process for measuring and reporting benefits. (2.59–2.84)

The Canada Revenue Agency’s (CRA’s) response. Agreed. The Agency believes that since the end of the audit (31 October 2010) it has addressed the recommendation in the following manner:

As noted by the Auditor General, a benefits measurement plan for Integrated Revenue Collections (IRC) Phase 1 had been completed by the CRA in 2009. The benefits measurement plan for IRC Phase 1 was further refined in 2010 to reflect annual gains to be achieved, benchmark information, and a data collection strategy, including when benefits would start to be realized, and who would be responsible for the delivery of the identified benefits. The 2010 revised plan was recently approved and work is ongoing in respect of measuring and reporting on the benefits. In fact, the approval process in place at the Agency requires that a benefits realization plan be prepared as part of business cases for large IT-enabled projects, which is an established best practice.

Citizenship and Immigration Canada’s (CIC’s) response. Agreed. Future projects will leverage the CIC Performance Measurement Framework and Operational Baselining Strategy in order to quantify, measure, and report on benefits.

Public Works and Government Services Canada’s (PWGSC’s) response. Agreed. PWGSC is committed to adhering to applicable Treasury Board policies, as well as Treasury Board of Canada Secretariat Chief Information Officer Branch guidelines and standards on the development of business cases. In cases where PWGSC is responsible for the implementation of large, horizontal IT projects with government-wide or multi-departmental impacts, the Department will continue to work cooperatively with the Secretariat and client departments to develop business cases that clearly assign accountabilities for benefit measurement plans and for tracking and reporting of benefits where applicable and justified.

 


Definition:

Executive project dashboard—A business management tool used to visually represent the health and status of a project or portfolio of projects. Dashboards report on the core project metrics affecting the project’s overall health and benefits realization, including cost, schedule, scope, risks, and issues. Their purpose is to draw attention to project areas that might require corrective action. Executive project dashboards are essential for ensuring consistency in reporting on the management of IT-enabled business projects to senior management, primarily at the deputy minister and assistant deputy minister levels. (Return)

 

PDF Versions

To access the Portable Document Format (PDF) version you must have a PDF reader installed. If you do not already have such a reader, there are numerous PDF readers available for free download or for purchase on the Internet: