Systems Under Development Best Practices Symposium

Auditor General's Report on Systems Under Development

Introduction

The OAG has always had an interest in auditing major computer systems under development. However, this was mainly on a one-time basis and depended on our comprehensive audit cycle. We will now be auditing these systems on a continuing basis at different points during the development process and our methodology will be updated to encompass this approach.

We approached this program of reviewing systems under development in stages. The first stage involved research with professionals in the public and private sectors to determine best practices. The second stage is the development of an audit methodology. This methodology will be used to audit several systems under development, with a chapter in the 1994 annual Report of the Auditor General. The Office will then continue value-for-money auditing of systems under development.

Our Study team was drawn from the OAG Computer Audit Group, and was assisted by several committees of senior advisors, to which we presented our preliminary findings for comment. In addition to an extensive literature search, we interviewed almost 70 people in 15 government departments and agencies. These organizations were selected on the basis of having done, or presently doing, systems audit work in the organization, or on the basis of their experience in developing major systems. In the private sector we also interviewed 16 people in 6 private sector companies and received briefs on systems development from 11 system integrators.

There is general agreement that the following activities represent "best practices", although everybody had their own ideas as to the best combination of these activities.

Common Themes / Guiding Principles

1. Put the Environment in Order

• Foster an organizational environment that includes an alignment of corporate strategies, plans, policies, methodologies, standards, and a management culture/philosophy that encourages computer literacy and participation, and favours a business approach to systems development over a pure technological approach.
• Develop a corporate Information Technology strategy consistent with the corporate goals to make ranking and selection of proposed projects more successful. When properly managed and exploited, systems are seen as providing a financial benefit and support to the core business activities.
• Adopt a systems development methodology that is sufficiently flexible to encompass any size project and any approach, from the "waterfall" type through iterative pilots to prototyping.
• Continuous reflection, learning and change are essential. Systems development projects fail because organizations tend to use the same mindset and the same approach over and over again.
• Organizations operating in a competitive or rapidly changing environment must be "on the leading edge." To get there and to remain there, they need to be willing to experiment, to try things, to take risks, to make false starts and to learn quickly—from successes as well as from errors. Each organization must look at its environment to determine if there is a need to risk unproved technologies, or if they should stay with proved technologies used innovatively.

2. Good Project Management

• Divide projects into modules that are small, self-contained and "deliverable". Modules are modifiable and replaceable with minimum effect on others. Ideally, no project (or module) should take more than eight to 12 months from design to completion. Rapid change in technology or user requirements may make the system obsolete even before its completion. There should be review and "bail-out" or "off-ramp" points at regular intervals.
• Successful organizations consistently use some form of basic project management techniques. Successful project leaders are trained in all aspects of project management, including establishing performance measurement indicators, cost and scheduling tracking procedures, milestone and acceptance processes, and technology transfer where contractors are involved; cost/benefit control, where the projected benefits are confirmed and delivered.
• Pay a lot of attention at the start. The most intense attention relating to systems development should be given at the beginning of the effort. Planning is crucial to success.
• Watch out for potential problems. Problems with people, structures and relationships rarely occur suddenly; they develop gradually. They are easy to fix at the beginning, but hard to recognize. Once they become serious, however, they are easy to recognize, but hard to fix.
• Systems projects, especially large ones, require quick decision making, funding and procurement processes. Slow processes can make the design of a system obsolete by the time final approval is obtained, i.e. before implementation even begins. Unnecessary external restrictions to the people and to the development process need to be eliminated wherever possible.

3. Develop a Business Case

• Make a business case for the project. Don't just develop the business case and then put it on the shelf for show. Constantly review and reassess it. Confirm that reasons for automating are still valid and timely. This is particularly important for projects of longer duration.
• Projects have to be business-driven, rather than technology-driven. Link the business case to the corporate strategy and departmental program objectives; include feasibility studies, cost/benefit analysis, risk analysis and a post implementation review.
• Confirm that the function to be automated is really necessary. Could it perhaps be eliminated? Simplified? Contracted out? Don't just automate an existing bad system, because the result will simply be an automated bad system. Before automating a process, use the opportunity to "re-engineer" it. Think hard about WHY the system needs to be automated, not simply HOW. This can yield large savings and financial benefits.
• The traditional practice of owning everything, controlling everything and doing everything in-house is no longer considered optimal. It is now seen as a limitation on an organization's ability to concentrate on doing what it does best. Strategic alliances and contracting-out are part of the new vocabulary.

4. Committed User Involvement and Project Ownership

• Include users early and make them project owners. There was unanimous agreement on the need for constant user involvement from the beginning.
• Determine the people and parts of the organization that will be affected by the completed system and in what way. Communicate with those affected throughout the project on impacts and effects; work closely with the unions.
• Successful organizations combine the talent of a user as the project leader and a strong systems analyst. Systems development does not work if led only by the Information Systems Group.
• The preparation for change should be under the control of the user manager whose group is knowledgeable of the existing applications and will benefit from the system, since the savings will occur here, and the user will need to realize them.

5. Senior Management Commitment

• Senior management must be strongly committed to projects once they are selected. This commitment translates into strong support of the project and into appropriate resourcing. Once the decision for project development is made, management must remain visibly committed to it—or terminate it. Their commitment is tied to their need to see immediate quantifiable results.

6. Leadership and People

• Find the right people: leaders who have authority to make decisions, and who have the courage to make choices; people with technical competence, who welcome the views of others, but are capable of reaching their own conclusions; people who can get things done: results-oriented, reliable, dependable people with team spirit, confidence, commitment and enthusiasm. Team continuity is also essential.

7. Management of the HR Impact

• All projects must include plans for the assessment and implementation of training programs and development needs for staff. The need for the proper management of change resulting from realigned or re-engineered processes or programs should not be underestimated.

8. Internal audit

• Internal audit's traditional role is changing. The emphasis is now on prevention of failures and early detection, rather than on identifying them well after they have occurred. Involve auditors in all important phases of the project. Internal auditors must take on more of an advisory role. Critical findings should be reported in time to take corrective action.

Conclusion

There is truly such an enormous range of initiatives under way in the government, from very small microcomputer applications to multi-million dollar mainframe and telecommunications applications, that it is impossible to lay out one fixed set of rules for all the players in the game. Our first concern, however, is in the development of high-risk, major computer systems. There is no "magic solution" to delivering "successful" systems, but there are surely commonsense recognized guidelines, based on proved management principles, that can ensure that a major project, once determined to be essential to the business, has more than a reasonable chance of being delivered on time, being within budget, and satisfying the user's requirements. An adaptive approach using selected components of a systems development methodology may be acceptable for small pilots or medium-sized projects; but, when proposed costs are in the Major Capital Project range ($100 million +), then project management and control issues become critical.

At the highest, most abstract level, there are in reality only two "best practices" for successful systems development: selecting the right project for development, and managing that project from the initiation stage through the implementation and evaluation stages.