This Web page has been archived on the Web.

2007 October Report of the Auditor General of Canada

Main Points

What we examined

The Canada Revenue Agency requires its staff to have specific, unique skills that must be maintained if it is to meet its objectives. We examined how well the Agency identifies and analyzes its needs for technical training and learning solutions to support its tax administration mandate. We also looked at how well it evaluates the effectiveness of its training and learning activities. We focused on three key branches—Compliance Programs, Appeals, and Legislative Policy and Regulatory Affairs—and the training provided to their auditors, investigators, appeals officers, and rulings officers, because they play a key role in interpreting tax legislation and ensuring compliance with it.

Why it's important

The Canada Revenue Agency is responsible for assessing and collecting the taxes that pay for most of the government's programs. The Agency's ability to accomplish its task and to protect Canada's tax base depends in large part on the knowledge and skills of its employees, particularly those whose work involves interpreting technically complex and frequently changing tax laws and regulations. These employees often have to pit their expertise against that of highly specialized tax accountants and lawyers in the private sector who continue to develop complex and creative ways to reduce their clients' taxes. This challenge requires the ongoing development of a highly skilled workforce and a significant investment in training and learning, which must be well managed. In the 2005–06 fiscal year, the Agency spent about $140 million on learning activities.

What we found

• Senior employees are retiring at an increasing rate, taking with them their knowledge and their expertise. Furthermore, in some active labour markets, the Agency is having difficulty hiring new employees and retaining present ones. These conditions are putting pressure on the Agency's training capacity because more time must be spent training new employees, and there are fewer senior employees available to coach them, train them, and transfer their knowledge and expertise to them.
• The Agency invests a large amount in learning and promotes a culture that encourages learning. Most respondents to the Agency's 2005 employee survey agreed that they get the training they need to do their jobs.
• The Agency has not fully evaluated how effective its investment in technical training and learning is. Nor does it systematically capture and assess how well knowledge gained through training and learning activities has been transferred to the workplace so that employees are better equipped to protect the tax base.
• Responsibility for training and learning is shared in the Agency, with many players involved at headquarters and at the regional and local levels. It is not clear who is responsible for managing specific aspects of training and learning. As a result, accountability for the Agency's investment in training and learning is diffused.
• The training given to entry-level auditors is good, although it can be difficult to find experienced auditors with enough time to coach the new auditors.
• Senior auditors, rulings officers, and appeals officers who have to deal with large corporations and complex issues need more learning opportunities to keep their knowledge and skills up to date. Furthermore, some of these employees told us they have to wait long periods of time to take the courses they need.
• The Agency has established a good process for determining training and learning needs. It has not yet implemented the process fully and consistently, nor has it ensured that needs assessment is specifically considered when allocating training resources.

The Canada Revenue Agency has responded. The Agency agrees with all of our recommendations. Its detailed response follows each recommendation throughout the chapter.

Introduction

7.1 The Canada Revenue Agency is responsible for assessing and collecting the taxes that pay for most of the government's programs. Canada uses a self-assessing income tax system—the taxpayer declares his or her income, deductible expenses, and credits on a tax return, and the Agency processes this return and issues an assessment. Canada also uses a self-assessing system for the goods and services tax (GST)—businesses charge and collect the GST on their taxable transactions and send it to the Agency, minus any eligible GST they have paid to their suppliers. The Agency has the right to review or audit tax returns and to issue assessments or reassessments if changes are needed. Taxpayers have the right to arrange their affairs to minimize their taxes, within the confines of the law, and the Agency is often called upon to identify these arrangements and contest them when necessary. Many factors add to this challenge, such as the growing globalization of the Canadian economy, new technologies, and the complex rules for taxing the international transactions of Canadians and multi-national companies.

7.2 In order to discharge its responsibilities well and to protect Canada's tax base, the Agency needs a highly skilled workforce capable of interpreting and applying Canada's complex tax laws fairly and equitably to millions of taxpayers. This workforce must be well-trained and kept up to date with ongoing changes in tax laws, the results of court judgments, and tax plans created by taxpayers.

7.3 In the 2005–06 fiscal year, the Agency spent about $140 million, or the equivalent of five percent of its personnel costs, on formal training and learning activities (excluding language training) in all branches; about $105 million of that amount represents the salaries of staff who were participating in learning activities, either as learners or facilitators. Technical training and learning activities, the focus of this audit, accounted for about 60 percent of the total training investment. The remainder represented soft-skills training (such as interviewing techniques), external courses and seminars, career development training, and the operations of the Training and Learning Directorate at headquarters.

Focus of the audit

7.4 The audit examined technical training and learning provided to staff of the Canada Revenue Agency in support of the Agency's tax administration mandate. Our objective was to determine whether technical training and learning activities adequately support the Agency in fulfilling its tax administration mandate. We focused on three key branches—Compliance Programs, Appeals, and Legislative Policy and Regulatory Affairs—because these branches play a key role in interpreting and ensuring compliance with tax legislation.

7.5 More details on the audit objectives, scope, approach, and criteria are in About the Audit at the end of this chapter.

Observations and Recommendations

Facing the learning challenges

Staff turnover is putting pressure on the Agency's training capacity

7.6 The Agency is facing high staff turnover in some offices, and it expects this trend to increase in the future. Part of this turnover is due to the retirement of baby boomers after years of service. In other cases, the turnover is due to competition for tax professionals in active labour markets such as Toronto and Calgary.

7.7 The Agency's auditors, investigators, appeals officers, and rulings officers are usually classified in the AU and PM categories. They range in both categories from level 1, in the most junior position, to level 6, in the most senior. Our analysis shows that about 17 percent of AU and PM employees will be eligible to retire over the next five years. However, the proportion is much higher for more senior employees (Exhibit 7.1). If not adequately addressed, this significant loss of knowledge and expertise will have a considerable impact on the Agency's ability to fulfill its mandate. It will also hinder opportunities to transfer knowledge from experienced employees to newer ones, and to provide newer employees with technical guidance and support through coaching, mentoring, and training. We note that the Agency has tended to hire people at the entry-level positions of PM1 or PM2 and develop them over time.

Exhibit 7.1—A high proportion of the Agency's most senior and experienced staff are eligible to retire soon

7.8 While the turnover brings new people into the organization, it presents three major challenges from a training and learning perspective.

• It requires that more time be spent on training to bring the new recruits up to speed with the legislation they are administering and with the Agency's processes.
• It means fewer experienced employees are available to coach and mentor the new recruits.
• It requires increased efforts to maintain or rebuild the knowledge base that departing employees are taking with them.

7.9 The Agency is aware of the challenges it faces with staff turnover and is putting succession plans in place. We expected to see training addressed in those plans, because the Agency will need to develop many new employees to the point where they are capable of carrying out its mandate. In the short term, it may also have to place more than the normal emphasis on training new employees if it is to achieve its longer-term objectives. We found that the succession plans are at an early stage of development.

7.10 Recommendation. The Canada Revenue Agency should include in its succession plans the steps that need to be taken and the resources required to ensure that the large number of new employees it expects to hire are adequately trained and that the knowledge held by departing employees is transferred.

Agency's response. The Canada Revenue Agency agrees with this recommendation. Building on existing succession planning practices, the Human Resources Branch on behalf of the Agency is developing a succession planning framework to assist managers in developing succession plans for necessary positions, which will include training and knowledge transfer components, as required. Development of the succession-planning framework will be completed by the end of the 2007–08 fiscal year.

The Agency's culture encourages learning

7.11 The Agency's staff, particularly its auditors, investigators, rulings officers, and appeals officers, are expected to interpret and apply the complex provisions of the Income Tax Act and the Excise Tax Act. Their work requires an in-depth knowledge of these statutes and a solid understanding of how the Agency interprets them, as well as a good ability to research tax issues. Learning is therefore critical if Agency staff are to be successful in their work.

7.12 The Agency has adopted a learning policy designed to support "learning all the time . . . everywhere." The objectives of the policy are to

• manage the Agency's significant investment in learning;
• encourage employee development;
• support employees in pursuing accredited learning activities; and
• engage employees and managers in understanding, communicating, and practising the Agency's learning philosophy.

7.13 We found, through our interviews and focus groups, that employees have a genuine desire to learn. As a result, they are eager to participate in formal and informal learning opportunities that help them broaden their knowledge. They also enjoy sharing their knowledge and experience with colleagues and learning from their colleagues when they have the opportunity to do so.

Clear accountability is needed

7.14 Managing learning well in a large, decentralized organization like the Agency is a challenging task. We expected to find clear roles and responsibilities for training and learning, and clear accountability both for how training and learning budgets are spent and for the results that are achieved.

7.15 We found that learning is a shared responsibility among many players (Exhibit 7.2). The Agency's Learning Policy describes the roles and responsibilities of the various players, but only in general terms. A framework developed and used by the Training and Learning Directorate provides some more detail, but has not been approved by senior management or widely disseminated within the Agency.

Exhibit 7.2—Learning is a shared responsibility

7.16 We found during our audit that managers were not clear about who was responsible at a regional, branch, or Agency-wide level for monitoring specific aspects of the overall investment in training and learning. For example, it is not clear who is responsible for analyzing at a regional or national level the learning needs of employees, and for taking those needs into account in the allocation of learning resources. Nor is it clear who is responsible for evaluating the effectiveness of a technical training product—the branch that manages the product or the Training and Learning Directorate.

7.17 As a result, while we found that a lot of training is going on and, for the most part, employees are satisfied with the training they get, accountability for the Agency's significant investment in training and learning is diffused. As we note throughout this chapter, the Agency needs to improve its management of training and learning. In our view, the first step to making these improvements is clarifying who is responsible for what and putting in place measures to ensure that the right managers are held to account for the Agency's investment.

7.18 Recommendation. The Canada Revenue Agency should clarify its shared-responsibility approach to training and learning by identifying, agreeing on, and communicating clearly which senior managers are responsible for specific aspects of training and learning and then holding those managers to account for achieving results.

Agency's response. The Canada Revenue Agency agrees on the importance of clearly communicating responsibility for the various aspects of training and learning, and will better articulate and formalize accountabilities.

The Agency's ongoing commitment to learning is demonstrated through the inclusion of "well-supported learning" in all executive and management-level performance agreements. In particular, for the 2007–08 fiscal year, senior managers are working toward this through focusing on transferring knowledge and corporate history from experienced staff who are retiring to next generation employees, and developing and completing learning plans based on business needs. Corresponding measures are in place, and managers will be required to report on progress.

Good management information is lacking

7.19 To deal with the learning challenges and to manage training and learning intelligently, management needs good information about training and learning trends and needs, budgets and costs, and results. In a large, decentralized organization like the Agency, such information is doubly important.

7.20 We found very little management information on training, and the information that was available was not presented in a way that management could use to monitor progress toward learning objectives. For example, the Agency Learning Plan is not a strategic planning document linking corporate needs to specific learning solutions, but is simply a schedule of new training initiatives. Similarly, the Agency's annual Report Card on Learning focuses largely on learning activities undertaken, not on the results of those activities. The information currently available makes it difficult for management to determine whether the Agency is obtaining value for money from its learning activities and meeting its learning objectives.

7.21 We are also concerned about the accuracy of data kept on the Agency's training and learning activities. Officials are using the information in limited ways, but they acknowledged that while its accuracy has been improving, further work is needed. We found errors in much of the training data we requested that was contained in the Agency's system. For example, we found employees registered for multiple courses that ran on the same date or employees registered a number of times for the same course, as well as inconsistencies in the dates and numbers of courses held. Managers need valid and reliable data to properly plan, monitor, and report on results achieved from training.

7.22 Recommendation. The Canada Revenue Agency should ensure that its management information on training and learning is reliable and up-to-date.

Agency's response. The Canada Revenue Agency agrees with this recommendation. The Agency will continue to build on the significant progress that has been made since 2005 to improve the integrity of data in its corporate learning system. In the 2007–08 fiscal year, the Agency is implementing a new service delivery model for human resource services. The model includes a dedicated Learning Analyst and Systems Officer in each region to train users to leverage the corporate learning system to its potential.

The Agency will consider enhancing the reporting capabilities of its corporate learning system. Decisions on improvements will be guided by development costs and funding availability.

Assessing training and learning needs

7.23 Assessing training and learning needs is essential to link an organization's objectives with the training and learning that its employees require to help achieve those objectives. We expected that the Agency would have adequate processes in place to identify and analyze its technical training and learning needs.

Several sources are used to identify Agency-wide training and learning needs

7.24 Identifying training and learning needs in the Agency is both a top-down and a bottom-up exercise. In the top-down exercise, the Agency determines the learning solutions that are required to implement corporate priorities or to deal with training gaps that have been identified. In the bottom-up exercise, employees determine their individual training and learning needs and then look for ways to meet those needs, using mainly existing products or activities.

7.25 The Agency uses several sources in the top-down exercise.

• Corporate priorities, such as tackling the underground economy. The operational branches and the Training and Learning Directorate determine what learning strategies would help the Agency achieve its corporate priorities, and they work together to deliver them (see Training to Address a Corporate Priority).
• Changes in the legislation and judgments made by the courts. This type of information is usually circulated to employees through communiqués or briefing sessions from the operational branches at headquarters.
• Informal networking. Managers told us that the Agency is able to identify training needs through day-to-day discussions between headquarters and local offices, conferences attended by a number of employees, and the review of employees' work by their team leaders.

In our view, the sources used to determine the training and learning needs related to corporate priorities and address emerging issues are adequate.

Improvements are needed in the use of individual learning plans

7.26 For the bottom-up exercise used to determine learning needs, the Agency has created individual learning plans (ILPs). Each employee is expected to complete an ILP annually and to discuss it with his or her manager, who also approves the final plan as part of the performance management process (Exhibit 7.3). The ILP process has existed since 2000 and has been refined and improved over the years. Since 2004, ILPs have been recorded in a database for subsequent use and analysis. Given their importance in determining the learning needs of employees, we expected that ILPs would be a realistic statement of learning needs. We also expected that they would be completed by most employees, used to produce regional or national learning plans, and used to determine whether identified learning needs had been met.

Exhibit 7.3—Employees state their learning needs in yearly individual learning plans

7.27 We were told by managers and employees that many ILPs are a realistic assessment of learning needs, and completing them is a useful exercise. However, they also told us that some ILPs continue to be a "wish list" and that the courses listed on them are not necessarily training that the employee needs for his or her current job or for career development. In addition, some employees told us that they have become discouraged because they had indicated a manager-approved need for a course for several years and had never been able to take it. For them, completing the ILP is simply a "paper exercise." The Agency is tracking the number of ILPs completed but is not tracking their quality.

7.28 About 70 percent of employees and managers in the three branches we audited had an ILP in the 2006–07 fiscal year, an improvement from 64 percent the previous year. However, training and learning needs for the remaining 30 percent of staff are not clearly known or identified, making the planning and monitoring of learning activities difficult.

7.29 The ILPs are summarized—or rolled up—nationally, regionally, or locally to determine the demand for particular courses, prepare training calendars, and initiate course registration. However, the roll-ups are not used to produce national or regional training plans. Such plans would bring together information from the top-down exercise and the roll-ups and could include learning objectives for the year, the need for particular learning events to address those objectives, the allocation of training resources, a calendar of learning events, and the method for measuring the results of those events against the learning objectives.

7.30 We also found that the ILPs are not used systematically to determine whether identified training needs have been met and, if not, to determine what corrective actions could be taken. Despite some work at headquarters to identify outstanding training needs, the needs expressed on the ILPs at the beginning of the year are not matched with the learning histories at the end of the year at a regional or national level. We note that this matching is done informally between managers and employees during performance evaluations. We were also told that some training histories are not up to date.

7.31 In our view, while the Individual Learning Plan is a good tool for determining employees' learning needs and monitoring whether those needs are being met, significant improvements are needed before its full potential can be realized. Ongoing reinforcement of the importance of the ILP process is also needed.

7.32 Linking of individual learning plans to training resources and course registration is weak. Given that individual learning plans are the basic tool used to identify learning needs, we expected that they would be closely linked to the allocation of training resources and to course registrations.

7.33 We found that training and learning budgets are set annually, based on historical averages, and that these budgets have not been reviewed for several years. The Agency spends the equivalent of about five percent of its personnel costs on training and learning, a significant amount. However, this amount is not formally linked to a summary of the learning needs expressed in ILPs. Therefore, the Agency has not analyzed whether it is spending the right amount in the right places based on the demand for training and learning. For example, we found no evidence that the Agency had taken its recruitment and turnover challenges into account when allocating training resources.

7.34 There is no calendar available to employees as they are completing their ILPs to show when particular courses will be offered. Instead, the roll-ups of completed ILPs are used to set the calendar by balancing the demand for particular courses with the availability of instructors and training facilities. As a result, completing an ILP identifies a need for training but does not register the employee for the training. In some offices, once a decision is made to offer a course, training coordinators use their ILP roll-up to send invitations to employees who identified the need for the course. In other offices, employees are expected to monitor the quarterly calendar of courses and to register themselves. For many courses, the demand exceeds the supply. In our view, there needs to be a clearer link between the ILP process and the course registration process so that employees have a reasonable expectation of quickly receiving the training they need or of obtaining the required knowledge in some other way.

7.35 Recommendation. The Canada Revenue Agency should ensure that individual learning plans are completed, are realistic expressions of learning needs, and are followed up to ensure that those needs have been met. It should also roll up the individual learning plans on a regional or national level and use that information for planning and budgeting purposes so that available resources are better linked to corporate and individual learning needs.

Agency's response. The Canada Revenue Agency agrees with this recommendation. The Agency will issue detailed guidelines by 31 March 2008, on planning of learning, based on business requirements and individual development needs, that will address both the development of individual learning plans and the overall process for the planning of learning across the Agency.

These new guidelines will consolidate and clarify existing guidance. The guidelines will include

• the business processes to ensure that learning demands identified in individual learning plans are properly entered into the corporate learning management information system and followed up during the planning period to ensure that learning needs have been met on a timely basis,
• requirements to keep learning histories up-to-date, and
• guidance on how information available from the corporate learning system will be rolled up and used in the planning process.

Training coordinators and the regional learning analysts and systems officers are being trained to produce reports based on learning priorities identified in the system, which will assist managers at all levels in strategically focusing their learning investments at local, regional, and national levels.

As understanding and confidence in using the corporate learning system grows, so will the integrity and usefulness of the data for planning, budgeting, and monitoring purposes.

Training needs of entry-level auditors are generally being met

7.36 Most new auditors begin their careers in the Small and Medium Enterprises program and then move to other audit programs or to Appeals. These employees are often recruited because they have post-secondary education with accounting specializations or an accounting designation, but they usually need basic training in the tax legislation and how the Agency interprets it, as well as training in how to conduct audits of taxpayers. This training is critical if these employees are to be effective in fulfilling the Agency's mandate.

7.37 We expected to find a well-organized training program for new recruits that would get them off to a good start in their careers with the Agency. We found that the Agency has such a program and is generally implementing it well.

7.38 Entry-level auditors take a series of classroom training sessions and then spend time working on appropriate taxpayer files with an experienced auditor. The training is based on a well-defined learning path (Exhibit 7.4), which is also used in the ILP process. The sessions and files get increasingly more complex and, by the end of the training program, new employees are usually ready to audit small businesses on their own. During our focus groups, employees told us that the entry-level training was organized and delivered well, but they were concerned about the availability of experienced auditors with the time to coach new employees or provide them with technical support. Given the staff turnover that the Agency is facing and that some offices will be facing for some time, managers will be forced to make difficult choices in order to find sufficient time for experienced auditors to coach new employees.

Exhibit 7.4—Entry-level auditors have a mix of classroom and practical training

Senior staff need more learning opportunities

7.39 Senior auditors and senior appeals and rulings officers are experienced practitioners who deal with the most complex audits and rulings. These employees often have to pit their expertise against that of highly specialized tax accountants and lawyers in the private sector. In addition, many of these employees are the subject matter experts who provide input into course content, teach courses, and provide technical support to other staff. We expected that the Agency would provide these employees with advanced training and sustained informal learning opportunities to allow them to keep their skills and knowledge up to date.

7.40 We found that the Agency offers nine advanced training courses for senior auditors and five training courses for international auditors. Some senior staff told us that they often have to wait several years before being able to attend a course that they and their supervisors consider important to their work. They also told us that the advanced courses they are able to take are almost always worthwhile.

7.41 In some cases, it is not cost-effective for the Agency to develop a course for a few senior employees working in specialized areas. This is particularly true for rulings officers. While senior employees rely to some extent on self-learning approaches, they also need other learning opportunities. The private sector offers courses and conferences that deal with these specialized topics, but the costs are usually quite high and the Agency allows only a few people to attend them. The Agency expects those who do attend to share what they have learned with their colleagues.

7.42 Frequent informal learning opportunities, such as practice-sharing sessions, peer coaching, discussions with the Agency's industry specialists, and informal networking, are critical for senior staff because they offer them the opportunity to keep their knowledge up to date and to build on it. We found that the Agency encourages these learning opportunities. For example, regional conferences for international, large business, and tax avoidance auditors are held every 12 to 18 months. These conferences allow a number of practitioners to meet and discuss common issues and concerns and to learn from each other. There are similar conferences for other senior staff.

7.43 We were also told that practice-sharing sessions are valued opportunities to discuss technical issues and broaden people's understanding of the issues their colleagues are dealing with. However, the team leaders usually determine the frequency of these sessions and the issues to be discussed, and there is a wide variation in practice. Some teams meet monthly to discuss technical issues, while others meet semi-annually or annually but discuss only administrative issues.

7.44 Senior staff also possess knowledge that should be transferred to their peers and to more junior staff, particularly given the turnover the Agency is facing. We found that except for the audits of large corporations, Agency employees usually work alone on a taxpayer audit, objection, or ruling. Senior employees told us that they want more opportunities to share and transfer their knowledge, but the pressure to complete audits, issue rulings, or resolve objections makes it difficult.

7.45 Recommendation. The Canada Revenue Agency should analyze the unique learning needs of its senior auditors, rulings officers, and appeals officers, and ensure that those needs are met on a timely and sustained basis.

Agency's response. The Canada Revenue Agency agrees with this recommendation. The Agency has started to develop learning paths for senior auditors, rulings officers, and appeals officers. The Agency expects to complete the first phase of this work in the 2008–09 fiscal year. The development of learning paths requires an analysis of the learning needs of these senior auditors and officers at different stages in their careers.

Managers and employees will continue to collaborate to identify individual learning needs and methods for delivery based on existing and desired competencies as they relate to program objectives. Additional specialized courses are under development that will meet the needs of senior auditors, rulings officers, and appeals officers in the Agency (for example, Cross-border Tax Avoidance).

The Agency believes that developing learning paths and undertaking the measures described in our response to recommendation 7.35 will lead to better targeted, more sustained, and timely delivery of training and learning activities for senior auditors, rulings officers, and appeals officers.

Ensuring that learning is effective

The Agency is not making sufficient use of its evaluation model

7.46 Training and learning are ways to help an organization meet its objectives. The Agency's primary objective is to ensure that taxpayers meet their obligations and that Canada's revenue base is protected. Training and learning should give its employees the knowledge and skills to understand and administer Canada's complex tax legislation. The Agency needs to evaluate its training and learning activities on a regular basis to ensure that employees are accomplishing this task, that its training and learning needs are being met, and that training and learning resources are being allocated and used effectively.

7.47 The Agency recently adopted an evaluation model to help it determine whether its training and learning activities are effective. The model has five evaluation levels:

• Level 1 provides the learners' and the facilitators' reactions to the design and delivery of the learning event and any action planned by the learner after attending the event.
• Level 2 provides the learners' perception of any change in knowledge and/or skills as a result of the event.
• Level 3 evaluates changes in behaviour on the job as a result of the learning—that is, how the knowledge gained was applied.
• Level 4 evaluates the impact of the learning on the business—that is, how it helped the learners meet short- and medium-term program objectives and the Agency's strategic objectives.
• Level 5 evaluates the return on investment of the learning event in monetary terms, by comparing the benefits obtained with the cost incurred.

7.48 The cost and complexity of evaluating a learning event increases with each evaluation level. We found that the Agency evaluates most courses at levels 1 and 2 through short evaluations completed by participants and facilitators at the end of the course. In evaluating courses at levels 3 to 5, progress has been slow. Based on the model's guidelines, the Agency called for 30 percent of learning events to be evaluated at level 3, 10 percent at level 4, and 5 percent at level 5. These evaluations would be done by a qualified evaluator. We found that of more than 700 courses in the Directory of Learning Products, to date the Agency has done only one level 5 evaluation and two level 3 evaluations; only the level 5 evaluation involved a technical training course. Officials told us that they need to establish more realistic targets given the costs of evaluating learning events at levels 3, 4, and 5.

7.49 Recommendation. The Canada Revenue Agency should establish targets for evaluating the effectiveness of training and learning events using the model it has adopted and a reasonable time frame for meeting those targets, complete the evaluations, and report the results to senior management.

Agency's response. The Canada Revenue Agency agrees with this recommendation. The Agency plans to establish targets for evaluations of formal learning products, including blended and e-learning products, by 31 March 2008. The Agency will develop an action plan with time frames to meet these targets. The action plan will be aligned with available resources.

Course evaluations are underused

7.50 Course evaluations at levels 1 and 2 provide important information about the content and presentation of a learning event, what the learners believe they have learned, and the actions they will take (Exhibit 7.5). Participants usually complete these evaluations for all the Agency's training courses. We expected that training managers would review and analyze the evaluations to assess the quality of the courses and to determine whether revisions are needed.

Exhibit 7.5—Course evaluation by participants provides important information

7.51 We found that local managers review the evaluations of courses that are delivered locally, primarily to determine the participants' satisfaction with the instructors. We also found that the manager of the national trainers reviews the evaluations of the courses they deliver after each course offering and takes corrective action when necessary. Course evaluations are captured in a database, but the information in the database is not reviewed regularly to help assess the effectiveness of the courses. Instead, before a course is revised, an account manager in the Training and Learning Directorate reviews all the previous evaluations of that course with the operational branch to get a sense of the overall level of satisfaction with the course and to identify suggestions for change.

7.52 In our view, the evaluation database is underused. Analyzing the information in the database regularly would provide important information about

• how learners reacted to course content and presentation,
• what they felt they learned,
• whether their learning needs had been met, and
• how they planned to put that learning into practice.

Other indicators provide some assurance that training is effective

7.53 As we have noted, the Agency has done little to formally evaluate how effectively its training programs transfer knowledge to its employees so they can perform their duties better and help the Agency achieve its objectives. The Agency therefore has only indirect assurance that it is getting value for its investment in training and learning:

• The Agency's 2005 employee survey indicates that most employees believe they get the training they need to do their jobs.
• Most quality assurance and program monitoring reviews show that there are few technical errors in reassessments issued by the Agency's auditors or in decisions made by its appeals officers. However, those same reviews also show weaknesses in other aspects of the work, such as audit planning, that likely point to the need for better training.
• Team leaders told us that they usually see an improvement in an employee's performance after training has been taken, if the employee has an opportunity to work on issues covered in the training and apply the learning.
• Informal networking at all levels highlights the strengths and weaknesses of learning events but provides little assurance that what has been learned was transferred to the workplace.

Coaching and mentoring are critical but employees face barriers

7.54 The transfer of knowledge from those who have it to those who need it is a critical component of all learning activities. This transfer takes place in formal classroom settings, but in the Agency it more frequently occurs on the job. Coaching and mentoring allow experienced employees to share their knowledge with others as they work together on a particular issue or audit. The Agency's learning policy states that it is committed to promoting learning through sharing of knowledge, experience, and best practices, and that it is implementing coaching and mentoring programs to help meet this commitment.

7.55 We found that the Agency has put in place a framework that includes tools, guides, and learning products for coaching and mentoring. During our focus groups, employees told us that they strongly support coaching and mentoring but that they need more of both. They identified several barriers to effective coaching and mentoring, including a shortage of experienced employees who have time available; the priority given to meeting targets for completed audits, issued rulings, or resolved objections; and a lack of recognition for those who take the time to coach others. In many situations, the team leader is the coach. However, team leaders told us they were stretched by increasing administrative demands and, on some teams, by the presence of several new employees. Furthermore, some team leaders do not themselves have the technical expertise to be effective technical coaches.

7.56 In our view, the staff turnover that the Agency is facing and will continue to face underscores the need for sound technical coaching and mentoring of all employees, in line with the commitment set out in the Agency's learning policy.

Conclusion

7.57 The Canada Revenue Agency is facing significant challenges as it works to protect Canada's tax base—complex and changing legislation, taxpayer expertise and creativity, and staff turnover. These challenges mean that technical training and learning activities need to be well managed if they are to adequately support the Agency in fulfilling its tax administration mandate.

7.58 The Agency's culture encourages learning, particularly in the branches that we examined, and the Agency has put in place many of the processes needed to manage training and learning. However, it has not fully implemented those processes, nor has it established a clear accountability regime for learning. It does not have an overall learning plan, with learning priorities, to meet current and future challenges and to use in measuring progress toward learning objectives. Furthermore, it only has limited data to measure its progress. While entry-level auditors receive good training, senior auditors, appeals officers, and rulings officers need more learning opportunities.

7.59 The Agency has not fully evaluated how effective its technical training and learning activities are. However, indirect evidence suggests that the Agency has some assurance that its training and learning activities are transferring the technical knowledge that staff require to support the Agency's operational needs and corporate goals. At the same time, important improvements are needed before the Agency can be fully assured that it is getting value for its investment.

About the Audit

Objectives

The objective of the audit was to determine whether technical training and learning activities adequately support the Canada Revenue Agency's operational needs and corporate goals.

Scope and approach

We examined two key elements related to technical training and learning. First, we assessed how well the Agency identified training and learning needs in support of its corporate mandate and staff needs. Second, we assessed the extent to which the Agency measured the effectiveness of its training and learning activities.

We examined technical training and learning in three branches—Compliance Programs, Appeals, and Legislative Policy and Regulatory Affairs. Our selection of these branches reflects their importance in interpreting complex tax legislation and in ensuring compliance with the legislation.

We interviewed officials at headquarters in the Training and Learning Directorate of the Human Resources Branch and in the three branches that were the focus of the audit. We obtained and reviewed relevant documents and analyzed information and extracted data related to demographics, training budgets, individual learning plans, and course histories. We also examined the 2005 employee survey for issues that related to training and learning, as well as some internal audit, quality assurance, and program monitoring reports.

In addition to headquarters, we visited seven Tax Services Offices and four regional offices. During those visits we interviewed senior officials in the Compliance and Appeals programs. We also conducted 21 focus groups with employees and managers to obtain their perceptions of the training needs analysis process and the effectiveness of training and learning.

Criteria

We expected that the Agency

• would have an adequate process in place to identify and analyze corporate technical training and learning requirements linked to operational objectives, and the resources necessary to meet those requirements;
• would have an adequate process in place to identify and analyze staff technical training and learning needs, and the resources required to ensure that they are met;
• would evaluate the effectiveness of its technical training and learning programs in meeting operational and individual needs.

Audit work completed

Audit work for this chapter was substantially completed on 29 June 2007.

Audit team

Assistant Auditor General: John Rossetti
Principal: Jamie Hood
Directors: Claude Brunette, Martin Ruben

Wilson Ford
Lisa Harris
Sophia Khan
Rod Newcombe
Robyn Roy
Ludovic Silvestre

For information, please contact Communications at 613-995-3708 or 1-888-761-5953 (toll-free).

Appendix—List of recommendations

The following is a list of recommendations found in Chapter 7. The number in front of the recommendation indicates the paragraph where it appears in the chapter. The numbers in parentheses indicate the paragraphs where the topic is discussed.

Definitions:

Training—In this chapter, training refers to organized events designed to upgrade the knowledge and skills of employees so that they can successfully perform to acceptable standards on the job. (Return)

Learning—In this chapter, learning refers to any process where employees acquire or improve the competencies that are necessary for their present jobs or for their future careers. Learning can be obtained formally, such as through scheduled training courses, or informally, such as through on-the-job coaching. (Return)

AU—Positions classified under the auditing group that audit the accounts and financial records of taxpayers or that provide technical interpretations of the tax laws. (Return)

PM—Positions classified under the program administration group that have responsibility for planning, developing, delivering, or managing government policies, programs, and services dealing with the collection of taxes and other revenues from the public. (Return)

Coaching—Techniques and processes used to improve work performance, solve problems, exchange ideas (peer coaching), or help employees in their professional development (coaching as a management practice). (Return)

Mentoring—A structured, supportive relationship between an individual who shares his or her knowledge, experience, and wisdom with another individual who is willing and ready to benefit from this exchange to enrich his or her career. (Return)