This Web page has been archived on the Web.

1997 December Report of the Auditor General of Canada

Main Points

23.2 Compared with contract projects, in-house projects were better aligned with the business objectives of the departments and were broken down into more manageable components. In addition, senior management was more involved, communications were more effective and the expertise and experience of the staff were better used.

23.3 Two of the three in-house systems that directly serve Canadians need to reflect a better understanding of who the clients are and what their needs are.

23.4 The disciplines of planning, oversight and quality assurance reduce the risks associated with the business and technical complexity of both contract and in-house projects. However, the three projects that we examined had some deficiencies in these areas.

23.5 A major concern we have reported before is the lack of recording and tracking of actual project costs. Without such tracking, projects can take longer, cost more and deliver less. Although one project was able to provide us with fairly complete estimates, none of the projects could provide precise actual costs. We estimate that each of the three projects we examined will cost the taxpayers between $50 million and $100 million.

Introduction

Focus of the audit

23.7 This year, we focussed on information technology systems managed and developed by departmental staff, with assistance, when needed, provided by individual consultants. We refer to these development projects as "in-house" projects. Although we could not establish firm costs for the three projects we reviewed this year, the estimated cost of each is in the $50 million to $100 million range. In previous years, our focus was on projects that were larger in size, often costing in the hundreds of millions of dollars, with most of their development contracted out in large fixed-price contracts (referred to hereafter as "contract projects"). In this chapter, we also review the status of the eight other projects that we examined in previous years and draw some general conclusions on the major differences and similarities in risk profiles between in-house and contract projects.

23.8 Further details on the scope and approach of this year's audit are included at the end of the chapter in the section About the Audit.

Recent systems under development events

23.10 Departments were directed by Treasury Board ministers in June 1996 to apply the Framework to existing projects, as applicable, and future information technology projects, and so attest when seeking Treasury Board project approvals. The Treasury Board Secretariat has advised us that actions are under way in all departments to implement the Framework. In addition, two departments have declared their intention to improve their software development processes up to level 2 of the Software Engineering Institute's Capability Maturity Model ( Exhibit 23.1 ). The model has five levels of increasingly effective management of software development processes, with level 5 being the optimal level. Additionally, in response to a direct request by the Public Accounts Committee, Transport Canada has declared that it would fully apply the Framework if it were to develop another major system.

23.11 We recommended in our November 1996 Report that departments prepare action plans for the implementation of the Framework. The Treasury Board Secretariat will assess the implementation of the Framework within departments against a series of four target plateaus beginning in March 1998 and continuing to 2002. The Chief Information Officer Branch of the Secretariat reports that it will conduct a review of the progress of the 20 largest departments toward the March 1998 plateau. Departments will be expected as part of this exercise to develop an improvement plan that focusses on the achievement of the goals at the next three plateaus.

Implications of increasing business and technical complexity

23.13 Business complexity in government has increased in recent years because of massive budget cuts, downsizing (in some cases, without a corresponding reduction of program requirements), program changes, new legislation, mergers of departments and reorganizations. Because of the increasing need to process information rapidly and to meet the pressing information needs of both managers and clients, business processes have to be re-engineered to provide real-time and just-in-time information. Furthermore, as we reported in Chapter 12 of our October 1997 Report, Information Technology: Preparedness for the Year 2000, the pending situation, referred to as the Year 2000 problem, has added new business imperatives that put all other information technology projects at risk in the struggle for available resources.

23.14 The 1990s have seen an escalation in technological sophistication to address more complex business needs that go beyond the simple automation of manual processes. New systems are based on the integration of new business processes and the supporting technologies, such as client-server, on-line transactions, networks, the Internet and interfaces to many legacy systems. Finding and developing the technical and business talent to address this sophistication is a major risk factor in itself.

23.15 To meet these challenging requirements, information technology must become part of the business processes themselves. The projects that have the best chance of succeeding are the ones where the development teams can integrate business and technical expertise to create new ways of delivering products and services to clients. The challenge for development teams is to combine these two sets of expertise effectively.

Observations

Our Assessment of the Projects Reviewed

• closely aligning the objectives of the new system with Revenue Canada's overall objectives;
• obtaining ongoing involvement and support of senior management;
• extensively involving intended users of the planned system in development activities;
• conducting regular independent risk assessments;
• using standards consistently; and
• using overall management discipline.

23.17 At the time of the audit of the Self-Serve Systems group of projects at Human Resources Development Canada, the projects, as a group, were assessed at high risk of not meeting their goals. However, the Department mitigated this risk by scoping the projects so that financial risk was reduced greatly and so that projects could stand alone to reduce interdependencies between them and protect them from delays in other projects. Risk was also mitigated by piloting projects in the field. The projects had good alignment of their objectives with the strategic goals of the Department, had senior management support and involvement and made good use of in-house expertise. In line with best practices, there were also efforts to establish a better quality assurance function. However, in some or all of the self-serve projects, we noted deficiencies with the planning, standards, documentation, task management and management of change requests.

23.18 At Industry Canada, we examined Phases II and III of Strategis. As Strategis moves forward, we are concerned that the project may experience high risks of not meeting its schedule, cost and service objectives. With the expansion of scope and complexity of Strategis, the focus and alignment that was firmly established for Phase I is becoming diffuse, the management of Strategis is not sufficiently integrated with the business side of Industry Canada and the current project management structure may not be adequate to achieve best results. There are efforts under way to make Phase I and some elements of Phase II part of normal publishing operations. Senior management has asserted that Strategis is more than just a web site but an initiative to transform the Department by providing access to government information resources, business transactions and new means of consultation. However, the development of a large web publishing site is a relatively new activity for government and industry. The Department has not taken this opportunity to apply project management best practices to an initiative of this type.

23.19 In summary, we observed that, in terms of best practices, in-house projects:

• were well aligned with departmental objectives and received strong support from senior management;
• were broken down into manageable components;
• made good use of in-house resources; and
• had good communications between developers and internal clients.

• apply a planning discipline that includes the precise definition of tasks over reasonable time frames;
• consult more systematically with intended clients;
• manage change requests; and
• implement a discipline relative to standards and documentation.

Planning Projects

• Knowledge / Involvement of Clients
• Integrated Teams
• Business Case
• Project Charter
• Overall and Detailed Project Plans
• Quality Assurance Plan

Knowledge and involvement of clients is essential

23.22 At Revenue Canada, there were extensive consultations to set up the new Standardized Accounting and Corporate Tax Redesign project. The Department has found it necessary to continue these discussions even while the development of the system is well under way. Since the clients believed that the system would have a great impact, Revenue Canada responded by continuing to be open and straightforward about the costs and benefits. An important innovation is the establishment of a steering committee with provincial partner representation and decision-making authority to co-ordinate ongoing consultations on electronic filing of corporate tax returns and its implementation.

23.23 At Human Resources Development Canada, there was evidence of efforts to consolidate previous information on clients and obtain some additional information from some client groups or associations. There are also efforts to continue the consultation in the context of specific pilot projects. The Department has provided mechanisms to gather client comments through its kiosks. However, in the case of client groups who do not use kiosks, feedback is obtained by staff and manually input into an information gathering system.

23.24 At Industry Canada, the intensive consolidation of useful client data at the start of the project, as well as continuing discussions with business associations, has provided a means for keeping in touch with client needs. The Strategis web site also permits users to make comments via electronic mail. Industry Canada staff are able to view these comments at any time and respond to them. However, obtaining feedback from all intended clients and not only from those who have reached the site and care enough to respond is a challenge the Department continues to face. At the time of the audit, to identify the continually changing clientele and their needs, the Department relied mostly on attendance by Strategis staff at industry association meetings, gathering of client input at the many demonstrations of the web site and measuring activity on the Strategis web site.

Fully integrated development teams are necessary for buy-in and support of projects

23.26 At Industry Canada, the projects are definitely business-driven but the same level of integration of the project teams does not exist. For example, business sectors are quite autonomous and, in many cases, have their own IT staff to prepare their applications and databases for the web site. Our observation is that this makes it difficult for Strategis to be more than just a web site because it still lacks the full buy-in and support of the whole Department for centrally co-ordinated standards of quality that would help Strategis to be "an initiative of transformation" for Canadian businesses and for the Department itself.

23.27 At Human Resources Development Canada, business units drive all IT activity but systems development follows a more traditional model where there is an organizational separation between the developers and their clients. Requirements are handed over to IT staff and a working system is handed back later. Business people and IT staff do not work in integrated teams but rely instead on meetings and other consultations throughout.

Investment proposals and business cases need to be complete and up-to-date

23.29 Over the years, the Treasury Board Secretariat has published guidance on the preparation of business cases and cost/benefit analyses. Most recently, the requirement was reiterated in the Treasury Board's Enhanced Framework of May 1996, and supported by a new guide, Creating and Using a Business Case for IT Projects, first issued in May 1995 and reissued in November 1996. The guide sets out in detail what constitutes a business case and deals with how to identify costs and benefits, prepare plans for the realization of these benefits, assess risks, analyze and present alternatives, and conduct reviews to confirm and update the business cases over the life of IT projects.

23.30 A critical requirement for business cases is that they be kept up-to-date. An update can be the basis for what the Enhanced Framework refers to as a "gate", where a "go/no-go" decision is made on the next stage of the plan.

23.31 To help departments identify areas for improvement, we took advantage of this year's audit to do a review of business cases prepared since the issuance of the new Treasury Board's guidance on business cases. Although the three projects we looked at had prepared business cases and, in some manner, had updated them, we noted that many of the elements of the government direction were not being followed. The following matters would have needed to be addressed in order to bring these business cases in line with the new Treasury Board guidelines:

• All costs were not included in the business cases. In all cases, some categories of costs were not considered, for example, money invested prior to the creation of the business cases. In two cases, costs were estimated for development only and not for ongoing maintenance. This factor can have a significant impact on the investment decision and the potential return on investment to the government.
• In all cases, risk assessments - the forecast of potential problems in the successful development and implementation of the proposed systems - were not included with the business cases presented to senior management. At Revenue Canada, risks assessments were done shortly after the business cases were prepared.
• In all three cases, the analysis of proposed investments did not consider measures of financial return, such as return on investment or other investment criteria suggested by the Treasury Board Secretariat's guide on business cases.
• The three business case reports were not complete as they did not present the information noted above; nor did they describe what alternatives to the suggested strategy were considered.

Project charters bring discipline to projects

23.34 Revenue Canada prepared charters for each major component of the SA/T2 project. Human Resources Development Canada has also adopted the practice but, at the time of the audit, had applied it to only 2 of the 15 projects that make up Self-Serve Systems.

23.35 Compared with contract projects, in-house projects tend to lack the same level of project management discipline as is specified in contracts. We see project charters as an important means of bringing elements of that discipline into projects managed in-house. Project charters effectively become internal contracts that guide project teams.

Project plans set detailed management and supervisory agendas

23.37 In reviewing the planning of in-house projects this year, we found that the best practice was at Revenue Canada. It developed a top-down overview plan and a series of detailed plans over reasonable time horizons that identified all key work packages or project segments. These work packages were further broken down into task level plans that covered periods of one to two weeks. Key interdependencies at all levels and critical path items were identified.

23.38 To validate this top-down plan, Revenue Canada followed with a bottom-up planning exercise that fully involved all people responsible for the tasks. The Department did a complete review of all plans at the task level to verify all the details and rebuild the plans. As part of the integrated teams, people from the business units were full participants in this validation exercise.

23.39 Task completion criteria. To be completely effective at the task level, project plans must identify completion events precisely enough to give a clear "yes/no" indication of satisfactory completion. This implies that quality standards have been set for each particular task. We would suggest that a task be considered not complete until it has passed its quality gate, at which point it would be considered fully complete. None of the projects examined this year defined their tasks to this level of detail.

23.40 Revision of plans. All elements of planning need to be considered living documents that must be re-examined and revised periodically as necessary. During these revisions, all parties have to be consulted, risks reassessed and planning assumptions re-established. With each major revision of the plan, the business case also has to be reviewed and gating decisions made.

23.41 The purpose of planning is to set the management and supervisory agendas in as much detail as possible to permit effective oversight and monitoring of task execution until the project is completed successfully.

Project Tracking and Oversight

Breakdown of tasks identifies deliverables and resources

23.43 The very magnitude and complexity of technology and business projects today make it necessary for management to exercise proper oversight to achieve the intended results. The key outcome of oversight is the timely application of corrective action when behaviour or results deviate from plan. The in-house projects we looked at were careful to break down their system requirements into more manageable components. While this approach adds another level of complexity to the project and requires more oversight in the form of co-ordinating, reporting and communicating, it is still preferable to the mega-project approach. However, except for the SA/T2 project at Revenue Canada, none of the in-house projects reviewed had broken down their work packages into tasks small and detailed enough to serve as meaningful tools to monitor actual progress and take corrective action.

Objective assessment measures must complement subjective ones

23.45 None of the managers on the projects we reviewed had the information needed to determine the actual project task costs. In all three projects, human resource costs, which are always the largest component of actual project costs, are known only at the corporate level and cannot be broken down to project tasks. Two objectives of project oversight are to determine how much productive time has been devoted to specific project tasks and how much work has actually been performed against plan. This allows managers to predict how the rest of the plan will likely proceed and to take corrective action where appropriate.

23.46 Managers usually need several methods to measure performance and project status. As described earlier, in setting up the plan, there must be a very concentrated effort to make the tasks short (one to two weeks) and defined such that a tangible, measurable output is produced. Measuring the actual time spent on a project task gives the manager an estimate of progress made relative to the original estimate. A less reliable and even softer measure is subjective judgment, although this can be firmed up by peer reviews and quality control "walkthroughs".

23.47 At Revenue Canada, there is no time measurement but management has compensated for this deficiency by defining tasks of a short duration with tangible outputs and scheduling peer-level task reviews. Management claims these reviews are thorough and timely enough to quickly expose the exact status of the tasks and to allow management to take corrective action.

23.48 At both Human Resources Development Canada and Industry Canada, progress is measured with subjective assessments and management reviews of these assessments. There is no time tracking or peer review to provide another more objective view of progress. In our opinion, while it is important to have the assessment and review of experienced managers, this in itself is inadequate for oversight and tracking. By the time management finds out that a time estimate is incorrect, the deliverable date may have passed and, by the time a recovery plan is created, the whole project may be close to being over budget. This potential risk can be mitigated if project components are small enough, task durations are very short and management acts quickly and decisively to recover from an incorrect time estimate or a missed deadline.

23.49 Both Human Resources Development Canada and Industry Canada use a development technique where a series of gradually improved prototypes allow the business line people to validate requirements and to monitor certain tangible outputs of the process. Although we would commend this as a useful and effective technique for developing systems rapidly and keeping business people involved throughout the development process, we do not see it as taking the place of proper management oversight and tracking. We also have concerns that prototypes can give an illusion of true progress while the issues of security, reliability and the ability to easily change the size and configuration have not been fully addressed. In many cases, before these prototypes are rolled out as production systems, very fundamental reworking is necessary.

23.50 Ironically, the government does not always apply to itself the performance measures it demands from its contractors. When the government contracts out its systems development work, it always demands cost and schedule performance measures of the contractor. In the in-house projects we examined, we did not see a project where the government applies these same requirements to itself.

Quality Assurance

Quality assurance needs to be an independent function

23.53 It is in the interests of the project sponsor, the project leader and management to strongly support the quality assurance function by means of an objective review that validates all the plans, standards and procedures applicable to the project. This will help to build confidence in project processes and results and increase the likelihood that the expectations of the clients and project sponsor will be met.

23.54 At Revenue Canada, there is no independent quality assurance function but quality control is addressed throughout. We are concerned that without a quality assurance function, the Department may not have the means to systematically assess and deal with the lessons learned from quality control.

23.55 At Industry Canada, no mechanism exists to ensure that the integration of the service with the content meets the expectations of clients, except for the feedback that the Department obtains from clients who choose to use electronic mail. We feel that this is inadequate for a web site that is intended to be the single window for businesses to obtain strategic information and to transact with the government. An independent quality assurance function could help to ensure that the Department achieved the full synergistic effect of having its technology people working with its business information experts.

23.56 At Human Resources Development Canada, management has taken concrete steps to establish an independent quality assurance function. A director has been appointed to co-ordinate all quality control activities and to examine the Department's development processes to outline deficiencies related to the Enhanced Framework. This quality assurance function is in its early stages and it is too early to predict how effective it will be in the future. Because Human Resources Development has been given the mandate to be the single window for all Canadians into all government job search activities, management believes that a fully supported quality assurance function is necessary to provide all the Department's partners and clients with the assurance that their expectations of a high-quality product will be fully met.

Conclusion and Recommendations

23.58 In contrast to the contract projects examined in the past, the three in-house projects we reviewed this year were better aligned with the business objectives of the departments, partly because senior management was more closely involved in the planning and the execution of each project. In all cases, special care was taken to break down the project into more manageable components and to produce usable results quickly. There was also better use of in-house experience in business and systems and better overall communication because project reporting was complemented with informal communication. In-house projects also have the advantage of not being distracted by contract management.

23.59 We are concerned, however, that the level of discipline we found in contract projects, relative to standards, documentation and the management of change requests, is present in only one of the three in-house projects reviewed this year.

23.60 Furthermore, two of the in-house projects carry a high risk of serious problems in the future because of poor planning methodology. In the area of task definition and tracking methodology, all the in-house projects examined are at high risk. Both in-house and contract projects have major deficiencies relative to setting up independent quality assurance functions.

23.61 Areas for improvement. In this year's audit, we looked at all aspects of project management; however, for reporting purposes, we have identified three areas where the best practices of project management are not universally followed and where corrective action is needed to reduce the risks of in-house projects:

• project planning;
• project tracking and oversight; and
• quality assurance, as an independent function.

• senior management involvement;
• sufficient "maturity" or capability for organizations to carry out major information technology projects;
• priorities properly set up and communicated with respect to time, cost and requirements;
• accurate, timely and effective performance measurement;
• effective user involvement and commitment to the success of the project;
• experience and expertise of the staff dedicated to the project; and
• the separation of large projects into smaller, more manageable components, each of which provides an improved capability to the organization.

• Clients should be properly identified and involved throughout the project, from the planning stage to the final implementation.
• Detailed project plans should be prepared to permit proper project tracking and oversight.
• Business cases should be developed in accordance with the Treasury Board Secretariat's guide, Creating and Using a Business Case for IT Projects (1996).
• Project charters should be used to clearly establish all the parameters for managing the work and the resources needed for all projects and subprojects.
• An independent quality assurance function should be established for all projects.
• Subjective assessments of progress should be supplemented with objective measures such as time tracking.

Treasury Board Secretariat's response: The findings of the Auditor General are consistent with the results of the project reviews conducted by the Treasury Board Secretariat in early 1995 and with the subsequent improvement directions documented in the Enhanced Management Framework of May 1996.

The suggestions and recommendations of the Auditor General are consistent with the views of the Treasury Board Secretariat, particularly as they apply to the first principle of the Enhanced Management Framework - information technology (IT) projects are aligned with, and support, business directions and priorities. The importance of ensuring that the initial planning phase of a project is effectively performed has continually been shown to be critical for the ultimate success of the project.

Additional best practice solutions and a revised training and development program, to be made available in the new year by the Chief Information Officer Branch of the Treasury Board Secretariat, will further assist and support departments in improving their IT project management capacity.

The continuing support of the Auditor General's Office, both formally and informally, in this critical aspect of the government's work is most appreciated.

Revenue Canada's response: The issues have been presented fairly and factually. Accordingly, no additional detailed response was provided.

Human Resources Development Canada's response: The Department agrees with the recommendations presented in paragraph 23.63 and has initiated the following actions:

1. Business unit partners will be involved in project planning, development and implementation.

2. Project plans will be developed in sufficient detail to ensure proper tracking and management review.

3. Business cases will conform to the Treasury Board Secretariat's guide on Creating and Using a Business Case for IT Projects (1996).

4. Project charters will be developed for all existing and new projects.

5. Quality assurance has been implemented as a separate unit within Self-Service Systems and also within Systems Support and Implementation Services.

6. Time tracking will be implemented to the degree sufficient to provide an objective measure of progress.

Industry Canada's response: In the fall of 1994, Industry Canada undertook the formidable challenge of making the information highway better available to Canadian businesses through Strategis, by deciding to take advantage of the Internet, which, at the time, was still an emerging phenomenon. This unprecedented effort, which is still being carried out, required the dedicated commitment of all levels of management, with a focus on results. The Department started "small" and knowingly confronted the risks of rapidly changing technology to develop new relationships with its business clients. Today, almost three years later, this initiative has been extraordinarily successful by any measure and it has been widely commended.

Indeed, the Auditor General's comments in this report must be read in relation to a project that, in terms of the results achieved, has seen remarkable success in regard to clients' acceptance and use, with over one million visits to the site and some 10 million documents downloaded.

Calculated risks, new team-oriented ways of working, rapid prototype development, much faster cycle times, competitive internal business case resourcing mechanisms and a less ponderous management structure were all intentionally used, in order to be at the forefront of the new, fast-moving Internet medium. Quite simply, traditional IT project management methodologies would have been impediments to the timeliness of the required transformation.

Today, given the burgeoning success of Strategis, the Department is having to consider strategies for dealing with growth and increasing client demands that largely surpass initial expectations. Accordingly, we will be implementing steps to provide more stringent quality assurance and business planning, and involving our clients even more closely, but in such a manner as to not bureaucratize the current innovative, dynamic and responsive processes that are working well and providing "real-time" feedback.

About the Audit

Audit objective

Criteria

• system development efforts done in-house would apply high-quality project management practices;
• when compared with contract projects, in-house system development projects would use similar commonly accepted best management practices; and
• government projects managed in-house would rely on frequent consultations with, communications to, and involvement of potential users.

Scope

• the Standardized Accounting and Corporate Tax Redesign project at Revenue Canada;
• Self-Serve Systems at Human Resources Development Canada; and
• the Strategis project at Industry Canada.

• the Income Security Program Redesign at Human Resources Development Canada;
• Transport Canada's Integrated Departmental Financial and Materiel Management System;
• the Common Departmental Financial System at Public Works and Government Services Canada;
• the Public Service Compensation System, also at Public Works and Government Services Canada;
• the Canadian Forces Supply System Upgrade at the Department of National Defence;
• the Tactical Command, Control and Communications System also being developed by National Defence;
• Transport Canada's Canadian Automated Air Traffic System; and
• the Real Property Services set of projects at Public Works and Government Services Canada.

Approach

Audit Team

For information, please contact Eric Anttila, the responsible auditor.