5.1 The reform of classification and job evaluation was one of the important systemic changes associated with public service renewal. Because of important links with other systems such as labour relations and pay administration, classification and job evaluation have been considered the "cornerstone" of human resource management in the public service.

5.2 In response to recommendations for change, the Treasury Board Secretariat undertook to develop a single job evaluation plan to replace the some 70 currently in use, and to reduce significantly the number of occupational groups and sub-groups into which jobs are classified. Given the difficult context, this was quite a challenging task.

5.3 Based on our assessment of the June 1995 version of the Universal Classification Standard, our audit concluded that, after some six years, the challenge posed by the reform has been only partially met:

The new job evaluation plan developed by the Secretariat is a definite improvement and has many promising features, but a number of significant issues need to be addressed to ensure that it is a valid and reliable tool, capable of evaluating all jobs in the public service while meeting requirements for pay equity that stem from the Canadian Human Rights Act.
Opportunities exist to simplify further and/or to reduce the costs of classification and job evaluation and some of the related systems and practices, such as pay administration.

5.4 While the Treasury Board Secretariat has undertaken the reform with a commitment to "cost neutrality", we found that the cost and benefits of the reform were not known with any degree of precision.

5.5 At the conclusion of the audit the Secretariat was taking a number of important measures to address issues identified by the audit. The Standard has potential and can be made to work. Success will depend, however, on the willingness and ability of the stakeholders to work together to achieve the objectives of the reform at a reasonable cost to the taxpayer.

Introduction

5.6 In 1989, the government launched Public Service 2000 , an initiative to renew the public service. Its aim was to achieve a culture more oriented toward serving clients and delivering results. The government wanted to simplify the employment and personnel management regime so that the resources devoted to these activities could be redirected toward better service. A task force headed by a deputy minister and composed of senior managers was set up to review classification and job evaluation in the public service.

The Current Classification and Job Evaluation Systems

5.7 Classification. In the public service, jobs are currently classified into 72 occupational groups and 106 sub-groups, and into a varying number of levels within these groups. Exhibit 5.1 provides examples. The current occupational group structure was created in the late 1960s, mainly to serve as a basis for bargaining units. The structure has remained essentially unchanged and continues to be the main framework for negotiating pay. As of July 1990, there were 78 bargaining units. As a result of these complex occupational group and bargaining structures, there are, according to Treasury Board Secretariat, currently some 840 pay rates and some 70,000 rules governing pay and allowances in the public service. This impacts heavily on the complexity and cost of related administrative systems. In 1994 we reported that, partly due to its complexity, the annual administrative cost of the federal pay system per employee was some 1.5 times higher than in provincial governments. In October 1995 we reported that the project initiated to improve the pay process through better use of technology was cancelled after more than $60 million of the estimated $120 million total cost of this project had been spent. The complexities of the pay system were a significant factor in the difficulties encountered in that project.

5.8 Job evaluation. Job evaluation is the process used to determine the relative value of jobs, based on content and requirements. Jobs are evaluated using factors and rating guides (the job evaluation plan) and reference (benchmark) positions contained in a "classification standard". The resulting evaluations are used to determine which jobs within the organization should receive the same rate of pay and which should be paid at different rates. Approximately 70 different classification standards are used in the public service.

A complex and lengthy process

5.9 In the public service, classification and job evaluation are intertwined. The process is both complex and lengthy, and thus costly. It requires several steps, including:

gathering and reviewing information about a job;
writing a detailed and thus lengthy job description with appropriate organizational charts that accurately depict the job and its relation to other jobs in the organization;
determining the occupational group of the job by assessing how the job description fits, or does not fit, globally with definitions contained in various classification standards;
evaluating the requirements of the job, using the job evaluation plan and the reference positions (benchmarks) contained in the applicable classification standard to determine the value of the job, usually expressed in points; and then
determining the level of the job by comparing its point value to the predetermined range of points established for each of the levels in the occupational group. Exhibit 5.2 illustrates the factors currently used to evaluate positions in the Secretary sub-group of the Secretarial, Stenographic and Typing occupational group and the maximum number of points allocated to each. It also shows the point ranges set for each level of the sub-group. Exhibit 5.3 presents the rating guide used to evaluate the factor "consequences of error" for the Secretary sub-group.

Pay equity requirements: more complexity

5.10 The concept of "equal pay for work of equal value" has had the force of law in organizations under federal jurisdiction - including the public service - since 1978. Such organizations must now be able to compare the value of jobs in predominantly female groups with jobs in predominantly male groups, using a common standard and taking into account four factors specified in section 11 of the Canadian Human Rights Act: responsibilities, skills, effort and working conditions.

5.11 Developed in the late 1960s, the current classification and job evaluation plans were not designed to compare the value of jobs in one occupational group with those in another group. In addition, current job evaluation plans do not always include the factors specified in the Act. The current classification and job evaluation systems thus cannot be used effectively to achieve or maintain pay equity. Several complaints have been filed with the Human Rights Commission in relation to pay equity issues, and a Human Rights Tribunal has ruled that some practices are discriminatory.

5.12 To deal with pay equity issues, complaints and rulings, the Treasury Board Secretariat has had to resort to using an additional, separate "common" job evaluation plan and a cumbersome system of equalization payments to female-predominant groups. This has added a significant layer of complexity to already complex job evaluation and pay systems.

Why are classification and job evaluation important?

5.13 There are very important and complex links between the classification of a position and other human resource management decisions or systems, such as staffing, collective bargaining and pay administration. For these reasons, classification and job evaluation are considered the "cornerstone" of human resource management in the public service. For example, because appointments are usually made "to a position", reclassifying a position to a higher level leads to initiating a staffing process to find a candidate or promote the incumbent. The resulting appointment will trigger a pay transaction. Exclusion from, or membership in, a bargaining unit will be also determined by the occupational group and level of the position and the duties performed by the incumbent. Because of these links, any change to the current classification and job evaluation systems, such as simplification, can have significant multiplying effects. The reform of classification and job evaluation was considered by many as one of the important systemic changes resulting from Public Service 2000 . Exhibit 5.4 describes in more detail some of the links between classification and job evaluation and other aspects of human resource management in the public service.

Previous Audits of Classification and Job Evaluation

5.14 In our 1983 Report chapter titled "Constraints to Productive Management", we noted that managers found it difficult and time-consuming to apply the current job evaluation plans. They felt it hampered productive management. In 1984 we reported that job evaluation in the public service was subject to great pressures at a time when the overall size of the public service was being restrained. We also noted significant deficiencies in the way the system was managed. In our follow-up report in 1986 we expressed concerns about the pace of Treasury Board Secretariat's review of a significant number of outdated classification standards. In 1990, another follow-up noted the relatively high number of occupational groups in certain departments and agencies. We reported that six departments had at least 40 occupational groups, and that one of them had 56. We concluded that a fundamental change was necessary. In our 1994 Report chapter "Management of Scientific Personnel", we recommended that consideration be given to creating a single occupational group for employees doing scientific research, to simplify the occupational group structure in research establishments.

Proposals for Reform: Significant Change

5.15 The task force set up under Public Service 2000 was charged to examine the occupational group structure and the administration of job evaluation in the public service. In its July 1990 Report, the task force recognized that both were cumbersome, too complex, very time-consuming and too costly to operate, to say nothing of their impact on the costs of other human resource management systems. According to the Treasury Board Secretariat, in 1989-90 alone there were some 216,000 classification actions, more than half of them involving major changes. Yet it has been established that the total number of staffed "full-time permanent" positions in the public service at that time was only 181,355. Partly due to the restructuring of departments, the Treasury Board Secretariat could not provide us with more recent information on the number of classification transactions over the last five years.

5.16 The task force also pointed out that the classification and job evaluation systems posed many obstacles to redeployment of resources, and that changes in technology and work organization had made these systems obsolete in many cases. The task force noted that the current systems were highly susceptible to manipulation.

5.17 The task force concluded that classification and job evaluation should be greatly simplified, and recommended that the current 72 occupational groups and 106 sub-groups be reduced to 23 and 8 respectively. It also recommended amalgamating 15 occupational groups into a single group, the General Services group (GE), which would include some 125,000 public servants. The task force recommended that job evaluation and its administration be simplified by developing a single job evaluation plan. It suggested that job descriptions be no more than three pages long and in a computerized format wherever possible. Exhibit 5.5 presents the guiding principles that the task force proposed for the new job evaluation plan.

5.18 In the November 1990 White Paper Public Service 2000: The Renewal of the Public Service of Canada , the government endorsed many of the task force recommendations. In January 1991, Treasury Board Secretariat started to implement them with a commitment to "cost-neutrality", which was never clearly defined.

Undertaking the Reform: A Formidable Challenge

5.19 Following up on the recommendations of the task force and the White Paper, the Treasury Board Secretariat decided to develop, within a self-imposed timeframe of 24 months, a totally new job evaluation plan that would be capable of evaluating the job content of over 200,000 positions, widely different in nature and scope. The plan had to be suitable for use in more than 50 departments and agencies whose size and activities vary greatly. Moreover, reducing the number of occupational groups meant dealing with several unions, some of which could eventually disappear with the reduction in the number of occupational groups - all of this at a time when labour relations were difficult, salaries had been frozen and the government and unions had been in litigation for several years over pay equity. This was quite a challenging task.

5.20 Since 1991, the Treasury Board Secretariat, in collaboration with departments, has developed a new job evaluation system, the Universal Classification Standard . The Standard is intended to replace some 70 current classification standards, using a single approach to job evaluation.

5.21 Major changes, at times unexpected, interfered with the project - such as the June 1993 decision to restructure departments, and the downsizing of the public service by some 45,000 positions over three years due to changes in government policies, programs and service delivery.

5.22 When we began our audit, in the fall of 1993, a "final draft" version of the Universal Classification Standard had been issued. By then, according to the Secretariat, some 42,000 positions had been evaluated by departments for eventual inclusion in the new General Services (GE) group. The Secretariat had anticipated that implementing the Standard across the public service would be spread over five years. During our audit, new versions of the Universal Classification Standard were issued and implementation was delayed several times. At the conclusion of our audit the Standard had not been submitted to Treasury Board ministers for approval. Exhibit 5.6 provides a schematic description of the Standard and its key characteristics as they appeared in the most recent version (June 1995) issued by Treasury Board Secretariat prior to the conclusion of the audit.

Why Audit Before Implementation?

5.23 We decided to conduct our audit before the new system was fully implemented because our experience with job classification and evaluation has shown that good design is essential to a new job evaluation plan: if significant problems in design are discovered after implementation, it is difficult to take effective corrective measures. Moreover, a number of concerns had been expressed by experts, unions and the Canadian Human Rights Commission about the design and development of the Standard.

5.24 Furthermore, several million dollars had already been spent by the Secretariat and departments to develop and test the Standard. Another reason for auditing before implementation was that, because classification and job evaluation are central to human resource management in the public service, implementing the Standard was likely to have a major impact on the payroll, labour relations, pay administration and many other human resource systems as well as efficiency of program delivery.

5.25 Audit objective. Our audit was aimed at assessing whether the Universal Classification Standard would be capable of evaluating all jobs in the public service (that is, universal) as well as ensuring that work performed by women would be made visible and valued in the same manner as men's work to eliminate possible systemic bias (that is, gender-neutral). Another objective was to determine whether classification and job evaluation with the Standard and the occupational group structure would be simpler. Finally, we wanted to assess how the Treasury Board Secretariat had taken into account the cost/benefit factor.

5.26 Audit scope and approach. Our audit examined the way Treasury Board Secretariat managed the design, development, testing and validation of the new job evaluation plan and applied it to a number of positions using the June 1995 or earlier versions of the Standard. We also looked at the work done for the task force or by the Secretariat to simplify the occupational group structure. We interviewed senior officials and human resource managers in a number of departments, and examined how other organizations in the public and the private sectors have simplified their occupational group structure and job evaluation systems. We also interviewed and consulted experts in compensation management. Details about the conduct of the audit are presented at the end of the chapter.

5.27 When we began our audit, the government was committed to a policy of renewing the public service, providing flexibility for different needs but in the context of a single public service. While it can be argued that other organizational forms are possible - including decentralized human resource management - the audit was conducted against that policy.

Observations and Recommendations

The Status Quo Is Unacceptable

5.28 There is a consensus in departments and among Treasury Board Secretariat officials that the current classification and job evaluation systems are both obsolete and too costly, given the current pressures for fiscal restraint, more efficient service delivery and downsizing. To them, the status quo is unacceptable. We concur with this assessment.

5.29 We found, however, that while departments agree with the guiding principles identified by the task force for a new job evaluation system, they emphasize that any new system must allow for flexibility because of the significant variations in the nature and operations of the programs and services they provide. Departments also believe that current and future budget reductions and ongoing restraint require that the design of the new system, its implementation and its ongoing management entail the lowest possible cost in labour, time and dollars, as well as in human terms.

Job Evaluation: A Significant Improvement but Not Yet Meeting the Mark

The initial timetable for the project was unrealistic

5.30 Given the context, and the technical difficulties associated with the project, Treasury Board Secretariat's initial two-year timetable to develop a totally new job evaluation system as complex as the Standard was clearly too optimistic. Because of the initial deadline, the debate that should have taken place with numerous stakeholders on key issues did not take place. Issues conflicting with the deadline were ignored, although there were clear signs that significant repercussions lay ahead. In particular, because of the tight deadline, the thorough testing necessary to assess the universality and gender neutrality of the Standard was not carried out.

Universality and gender neutrality were not assured

5.31 Treasury Board Secretariat took a number of measures to ensure universality and to reduce the risk of systemic bias against work predominantly performed by women, including:

the development of a single job evaluation plan that will make it easier to compare the value of jobs done by women and by men;
the incorporation into the Standard of the four factors set out in the Canadian Human Rights Act ;
a definition of gender neutrality that recognizes the need to identify and value properly the historical and traditional characteristics of the work of women and men;
the selection of 16 elements capable of covering a wider range of job content in the public service; and
extensive training of managers and human resource specialists on the new Standard and on the requirement for gender neutrality when drafting job descriptions and during the job evaluation process.

5.32 The Treasury Board Secretariat concluded, in April 1995, that it was satisfied with the universality and gender neutrality of the Standard. In our opinion, however, the measures taken by the Secretariat were insufficient to conclude that the Standard could evaluate all jobs in the public service and in a gender-neutral manner. In fact, the following evidence suggests that the version of the Standard we reviewed was neither universal nor free of gender bias.

The method of collecting information on job content needs to be strengthened

5.33 If the Standard is to be capable of evaluating all jobs in the public service, comprehensive information needs to be gathered on the full range of significant job content present in the public service. However, we found serious deficiencies in the method that Treasury Board Secretariat initially used to collect job content information for the development of the Standard. The Secretariat collected job content information on a sample of 1,000 positions throughout the public service. Although the sample seems large, the Secretariat acknowledged that it did not statistically represent the whole public service population. For example, it did not include all combinations of occupational groups, sub-groups and levels. Positions in predominantly female groups were also under-represented.

5.34 Furthermore, the information gathered on job content was not subject to quality control to ensure that it was valid, reliable and gender-neutral. One pay equity expert consulted by Treasury Board Secretariat observed that the quality of the job content information gathered was inconsistent, varying from satisfactory and reasonable to incomplete and inaccurate. In addition, no effort was made before the first phase of the Standard's implementation to verify whether the new job description format and content could collect information on all the significant aspects of the work performed by people in either predominantly male or predominantly female occupational groups.

5.35 In spite of these deficiencies, the Secretariat used this job content information as a basis for making key decisions during the development of the Standard, particularly in choosing and defining the elements of the Standard that would serve to evaluate all jobs in the public service. This eventually led to work traditionally associated with women not being sufficiently visible, and to problems in applying the Standard to certain jobs.

5.36 At the conclusion of our audit, the Treasury Board Secretariat had undertaken a thorough review of the literature to identify overlooked job content in predominantly female groups and was committed to ensuring that the Standard would make visible all significant job content. In addition, it was developing sampling techniques as well as quality control mechanisms, with the objective of ensuring universality and gender neutrality.

The various components of the Standard had significant shortcomings

5.37 To assess the universality and gender neutrality of the Standard, one must assess whether its components (elements, degree definitions and degree illustrations):

are clearly defined;
incorporate all significant content of all jobs in the public service;
include frequently overlooked aspects of work performed by women;
measure unique aspects of work performed, to avoid crediting the same job content more than once;
are consistent and appropriate within each level and for all levels; and
can properly value all significant job content found in the public service, including work performed by women.

Exhibit 5.7 illustrates the various components of the Standard for an element that was entitled "Thinking Challenge" in the version we reviewed.

5.38 Our assessment of each of these aspects using the June 1995 version of the Standard revealed that the Treasury Board Secretariat did not have the basis for asserting that the Standard was universal and gender-neutral. In particular, the version of the Standard we reviewed would, to a large extent, perpetuate gender bias in job evaluation. Unless corrective measures are taken, this could lead to numerous costly and time-consuming grievances or complaints under the Canadian Human Rights Act and cast doubt on the Standard's credibility as a universal and gender-neutral job evaluation plan. We noted the following deficiencies:

Except for a few elements, definitions were vague, broad and at times ambiguous. It was not clear what was actually being measured.
Degree definitions aimed at providing, for each element, a rating scale representing a progression from a minimum to a maximum value were also extremely vague and ambiguous.
Degree illustrations intended to exemplify, for a specific element, degrees in concrete job content and to show the progression from one level to another contained the most significant deficiencies. In many cases, the criteria embedded in them were different from those in the element definition or in the degree they were supposed to illustrate. Aspects of women's work including work at a high level were missing. Also, progression was unclear because consistency was a problem in many cases. (The Appendix provides a summary of the significant deficiencies found in the various components of the Standard.)
Statistical tests to diagnose for systemic effects by gender revealed that six of the 16 elements of the Standard measured male and female job content equally well. The 10 remaining elements favoured the content of predominantly male jobs. Such results are indicators of potential bias.

5.39 At the conclusion of our audit the Treasury Board Secretariat was taking a number of important measures to address our concerns, such as: redefining, renaming and regrouping the elements; identifying gaps and examining the need for new elements; developing matrices and grids to discipline progression among degrees; and developing a framework to guide the review and the rewriting of degree illustrations.

The weighting of the Standard's elements needs to be reassessed

5.40 All job evaluation plans deem some job content to be more important than other content. The weighting - that is, the percentage of points awarded to each element in a job evaluation system - is intended to establish the relative importance of each element, reflecting the organization's objectives and values. The values themselves must be free of gender bias. Exhibit 5.6 shows the maximum possible number of points allocated to each of the 16 elements of the Standard.

5.41 To determine the importance of each element, 300 positions drawn from a sample were ranked using a method known as "whole-job-ranking". The method consists of comparing jobs using job descriptions, to rank them on the basis of their overall global "value".

5.42 Experts recognize that the "whole-job-ranking" method is fraught with difficulties for ensuring gender neutrality and validity, especially when applied to a large number of positions. Because it is subjective, the ranking can be influenced by a number of irrelevant factors, such as stereotyping. Evaluators are unlikely to question their preconceived opinions of the overall importance of the work, including their possible gender bias. If such a method is to be used, a significant number of safeguards must be in place to prevent or to detect bias potential. We found that the measures taken by the Secretariat were insufficient.

5.43 Although the results of the ranking exercise showed that the positions of some predominantly female groups moved up in the new hierarchy, our review of Treasury Board Secretariat's statistical analysis of these results nevertheless suggested the presence of significant problems. For example:

aspects of women's work were valued unfavourably in the ranking exercise;
implicit criteria used to evaluate jobs were very limited: in fact, a few aspects of job content alone could have served to predict the ranking of all 300 jobs; and
there were indications that the same aspect of work was credited more than once.

5.44 In spite of evidence of significant deficiencies, the ranking results were used to develop the weighting of the elements contained in the version of the Standard we reviewed . We found that the resulting weighting was highly imbalanced. Four of the 16 elements were assigned more than 55 percent of the points; 10 elements were each assigned a value of 5 percent or less, and are therefore given negligible weight. In the version of the Standard we reviewed, the only element designed explicitly to take into account work traditionally associated with women, "Care of Individuals," received a value of approximately 3 percent.

5.45 Overall, we found that, in the version of the Standard we reviewed, the importance given to three elements, "Influence", "Theories and Principles" and "Thinking Challenge" - often associated with administrative/managerial work - was such that the other 13 elements would not likely affect job evaluation results to any great degree.

5.46 The imbalance in the weight of the elements limits the Standard's ability to value all jobs in the public service properly. Furthermore, it results in the undervaluation of work performed by women. The high concentration of weights in only a few elements, the large number of elements to which no appreciable value is attributed and the low value assigned to work traditionally performed by women are due to shortcomings in the "whole-job-ranking" exercise noted earlier.

5.47 At the conclusion of our audit, the Treasury Board Secretariat was planning to examine the weighting of the elements of the Standard.

Thorough testing of universality and gender neutrality is essential

5.48 Because of the self-imposed tight schedule, the Treasury Board Secretariat did not conduct a full-scale pilot project to test the universality and validity of the Standard. Instead, it conducted three limited tests at different phases of the Standard's development. In our opinion, this testing was not enough to assert that the Standard can evaluate all positions in the public service.

5.49 For example, during the development phase, a sample of about 800 positions was used to assess the completeness and applicability of the Standard. Statistical tests for universality conducted by the Treasury Board Secretariat indicated problems in applying the Standard to particular occupational groups, such as scientific personnel or professionals. No analysis could be done for positions in many other occupational sub-groups because samples were too small. Nor could statistical tests be conducted for gender neutrality because the sample of positions from predominantly female groups was inadequate.

5.50 A second test began with the application of the Standard to some 35,500 positions (about 25 percent) of the 125,000 positions to be included in the new General Services group (GE). The objective was to verify whether the weight given to each element was appropriate and to determine the number of levels this occupational group could have. Although this was to be a test of the Standard, the Treasury Board Secretariat did not consider it appropriate or necessary to establish adequate quality control mechanisms to ensure that all departments were conducting the test properly. We found significant deficiencies in this test. For instance:

No guidelines were issued to ensure that departments would use a valid and the same sampling method.
While testing was being carried out, the Secretariat made the weighting scheme available to departments upon request. Usually, the weights for the elements of a new job evaluation system are not disclosed at that stage to ensure that raters are not unduly influenced by the rating points instead of by the job content.
"Jobbing" was used by many departments. "Jobbing" means describing and evaluating one position and automatically giving the same value to similar positions without necessarily evaluating them individually. While perfectly acceptable and even desirable at the implementation stage, "jobbing" is not advisable for testing purposes because it can seriously distort overall results.
Although the Treasury Board Secretariat reported at the end of this test that some 35,500 positions had been evaluated, we found that Secretariat officials did not know how many and which individual positions had really been evaluated. Nor did they know the results of the evaluation for the specific positions actually evaluated. One department, using the jobbing technique, reported 6,900 evaluation results when in fact only 30 positions had been evaluated.

5.51 Because of these deficiencies, the test results cannot be used to assert the validity and universality of the Standard for positions to be included in the new General Services (GE) group.

5.52 In July 1994, in response to our initial concerns about universality, Treasury Board Secretariat conducted additional testing on positions in occupational groups other than the General Services group. We found that some 70 occupational groups or sub-groups were excluded from the sample for this test - including some that were predominantly female. In a number of cases, the sample also included too few positions to be able to determine patterns, if any.

5.53 Our examination of the results of the various tests conducted by the Secretariat indicates that:

positions at the same level under the existing job evaluation system were distributed over three, four and sometimes five different levels in the new proposed General Services group;
positions that had been given about the same number of points using the Standard would have salary differences ranging from $5,000 to $50,000 using the current salary scales; and
a number of positions were exceeding the maximum number of points proposed for the highest level of the General Services group.

While such results might be justified and valid given the new values imbedded in the Standard , they might also indicate the presence of problems or anomalies. A detailed examination of the results is necessary to determine if they are valid or stem from shortcomings in the Standard's components, misapplication of the Standard, skewed results due to "jobbing", improper classification of jobs under the current system or the way pay rates have been established. The Secretariat has yet to determine if such results are valid or not.

5.54 We believe that the Secretariat needs to conduct more rigorous and convincing testing to assert the validity, universality and gender neutrality of the Standard if it wants to persuade the employees, unions and other stakeholders that the Standard is indeed valid, universal and gender-neutral.

5.55 Treasury Board Secretariat, in collaboration with departments and representatives of employees and with the support of external experts in pay equity and job evaluation, should ensure that the Standard is capable of evaluating all jobs in the public service while meeting criteria for gender neutrality stemming from the Canadian Human Rights Act . More specifically, the Secretariat should ensure that:

the method used for collecting information on job content in public service jobs is valid, reliable, credible and free of bias, with appropriate quality control mechanisms to permit the collection of information on all significant job content, including work associated with women's work that is often overlooked;
the sample of jobs used to gather information on job content is adequate and credible, with adequate gender representation;
the various components of the Standard (elements, degrees and degree illustrations) are reviewed to ensure that they can effectively capture and properly value all the significant job content found in the public service, including that associated with women's work. This should include ensuring that:

all components of the Standard are clearly defined and measure unique aspects of work to the extent possible;
the progression from minimum to maximum value in each element is clear; and
illustrations intended to assist raters provide a reasonable spectrum of job content, do not undervalue work performed by women and are consistent with the definitions and the desired progression in value;

the weight allocated to each element is appropriately distributed to permit the valuation of all jobs, including work performed by women, on a fair and equitable basis while giving proper recognition to the objectives and values adopted for the public service and the reform; and
thorough testing, on a valid sample, of the validity, reliability and gender neutrality of the Standard is carried out with adequate quality control measures and, if necessary, review and additional testing before implementation.

Treasury Board Secretariat's response: Treasury Board Secretariat (TBS) agrees with these recommendations.

TBS acknowledges that the version of the Universal Classification Standard (UCS) that was reviewed by the Auditor General had shortcomings. However, a lot was accomplished and a leading-edge tool was produced at that time. TBS agrees that some things could have been done differently given more resources and better co-operation among the stakeholders.

TBS has since adopted a consultative approach to develop improvements to the UCS, enhance the collection of data, re-weight elements and add new elements, and simplify the process. All of the main stakeholders have recently agreed upon definitions for gender neutrality, universality and simplification.

The current active participation of the Canadian Human Rights Commission, the Auditor General's Office, consultants, departments and unions on a UCS Advisory Committee, a Technical Committee, and other committees and working groups dedicated to specific aspects of the improved UCS should ensure that we can address previous shortcomings and succeed in achieving these recommendations.

Statistical testing will also supplement the qualitative approaches being developed in consultation with all stakeholders to achieve a universal, gender-neutral plan.

Simplification and Operating Costs of Job Evaluation

Job evaluation will be improved but still time-consuming and costly

5.56 Using only one classification standard will make it much easier for managers to make classification decisions and to explain the results. Still, job evaluation with the version of the Standard we reviewed would be relatively complex and costly, according to experts.

5.57 One of the reasons for the continuing high cost is that, with the Standard, job content information is still gathered through job descriptions. Job descriptions in the new format we reviewed were long, in many cases ranging from five to 11 pages. The Treasury Board Secretariat estimated that it would take approximately two days to write a job description using the new format. Departments we visited generally confirmed this estimate, some indicating that, in many cases, it took as long as three days. The time and thus the cost of writing job descriptions is about the same with the Standard as with the present job evaluation plans used by departments. However, Treasury Board Secretariat estimates that with the Standard, a committee will be able to evaluate five positions a day on average, a notable improvement compared with the one position a day using the existing systems. Some departments we visited, however, thought that this estimate was optimistic.

5.58 Writing detailed job descriptions and evaluating jobs with the version of the Standard we reviewed entail significant training costs. It was estimated that individuals need two or three days of training to be qualified in writing job descriptions in the new format, and another two or three days to be able to participate in job evaluation committees. One department reported that the cost of training managers to write job descriptions and evaluate jobs with the Standard had been $3.4 million in salary alone, for some 18,000 positions. On this basis, the cost in salary for training managers, let alone incumbents of positions, in the public service as a whole could be substantial.

5.59 We found that some departments are exploring avenues to reduce the burden and operating costs of job evaluation after the Standard is implemented. For instance, one department with some 25,000 positions intends to use the "jobbing" technique extensively. It has identified 82 generic job descriptions that could be applied to about 65 percent of all its positions proposed for inclusion in the General Services (GE) group. This would reduce significantly the time required to evaluate them. Other departments have developed or are planning to introduce computerized job descriptions or a computer bank of job descriptions.

Effort to computerize job evaluation failed

5.60 In the context of a suggestion made by the task force, the Treasury Board Secretariat launched in April 1991 a research and development project to determine whether artificial intelligence technology could provide managers with a tool for writing job descriptions and for evaluating jobs in a few minutes.

5.61 With co-operation and financing from 10 departments and with assistance from the Artificial Intelligence Research and Development Fund of Industry, Science and Technology Canada, the Secretariat hired two consultants to produce a prototype with two modules - one for writing job descriptions and one for evaluating jobs. In March 1993, the prototype was delivered.

5.62 Consulting and Audit Canada was asked by the Secretariat to assess the results. The report revealed that, in spite of impressive achievements in research, there were a number of serious problems with the design and the different technology used for the two modules. As a result, the prototype could not be made operational, and the effort required to develop an operational system would mean almost starting over. The prototype would have to be redesigned. Consulting and Audit Canada estimated that, to produce an operational computerized tool, work would have to continue for another two to three years and could cost $3.5 million to $6 million. Furthermore, Consulting and Audit Canada considered that the project would involve significant technical risk. For these reasons it recommended a cost/benefit analysis before pursuing the project. After $1 million had been spent, the project was terminated. Departments that had contributed resources to the project expecting an operational system were disappointed by the results.

5.63 We believe there are ample opportunities to simplify job evaluation further and to reduce the associated costs. Between a job evaluation system that requires detailed and lengthy job descriptions and the individual evaluation of jobs and a system that does the same thing using artificial intelligence technology, there are intermediate solutions that can take advantage of current computer technology, and new trends in job evaluation such as developing a structured computerized questionnaire. Exhibit 5.8 presents new trends in job evaluation. Exhibit 5.9 presents an illustration of the structured questionnaire approach to job evaluation for "Work of Others". The simplification of job evaluation and, in particular, reduction of the operating costs is an important issue for departments. They will bear most of the implementation cost, as well as the ongoing operating costs resulting from any job evaluation plan. Their demands for keeping such costs as low as possible should not be ignored.

5.64 At the conclusion of our audit, the Treasury Board Secretariat was examining various options to improve the collection of information on jobs at lower cost, including the feasibility of using a questionnaire during the implementation phase. Other options being considered include data banks of job descriptions and job evaluations, and computerizing a questionnaire at a later stage.

5.65 Treasury Board Secretariat, in collaboration with departments and representatives of employees and with the support of external experts in pay equity and job evaluation, should ensure that the implementation costs and future operating costs borne by departments are the lowest possible, notably by exploring alternatives to detailed job descriptions, such as a structured questionnaire, and by finding ways to simplify the job evaluation process further.

Treasury Board Secretariat's response: Treasury Board Secretariat agrees. However, it is important to separate out investment that goes into the design phase of the Standard to ensure that work characteristics are not overlooked. Streamlined and more economical means of writing and evaluating work descriptions are being addressed in anticipation of a positive decision on implementation.

Simplification of the Occupational Group Structure

5.66 As we note in paragraph 5.17, the task force set up to review the design of the occupational structure recognized its complexity and recommended reducing the 72 occupational groups and 106 sub-groups to 23 and 8 respectively. It also recommended the creation of the General Services group that would include some 125,000 employees. The rationale for whether to merge or maintain occupational groups was based mainly on the general nature of the work and the community of interests. Other factors were also considered, such as the need for professional certification, comparability with outside labour markets and respect for union affiliations.

The occupational group structure: less simplified than recommended

5.67 The Treasury Board Secretariat endorsed the criteria used by the task force for the simplification of the occupational group structure but, under pressure, placed greater importance on its commitment to respect union affiliation. A review of representations and correspondence revealed the extent of the lobbying efforts on this point by groups, associations and unions. With a few exceptions, all the requests made to both the task force and Treasury Board Secretariat were aimed at maintaining current occupational groups or creating sub-groups within the occupational group structure. As a result, instead of the 8 sub-groups recommended by the task force there could be as many as 30 to 45 sub-groups. In that case, simplification would be much less than initially envisaged.

The proposed structure: varying degrees of support

5.68 The proposed new occupational group structure does not appear to have the support of all departments. For example, in February 1992, deputy ministers reaffirmed their support for the objectives of the reform but they expressed concern about the large number of sub-groups and the apparent lack of consultation with departments on decisions about the types and number of occupational groups. Many felt that respect for union affiliations had taken precedence over departmental priorities and operational requirements.

5.69 Most of the departments we visited were satisfied with the creation of the General Services group, especially those with a large proportion of employees proposed for inclusion in it. These departments saw the creation of the group as a major step in the right direction. They also felt that this new occupational group would make it easier to deal with expenditure reductions, restraint and restructuring.

5.70 Departments less affected by the creation of the General Services group said that the number of occupational groups and especially sub-groups was still too high. They saw the proposed occupational group structure as only a marginal improvement. For example, some would like to have seen a number of occupational groups in the department amalgamated into a single, broader group to facilitate career advancement, maximize flexibility or meet operational requirements more efficiently. Some departments have made representations to the Treasury Board Secretariat stressing the importance of simplifying the occupational group structure further.

5.71 Departments also expressed concern about the appropriateness of maintaining current union accreditations in the transition period. For many departments, this runs the risk of significantly diluting the anticipated benefits of simplification. Experts also said that maintaining a large number of occupational groups and the current collective bargaining structure, where each bargaining agent representing an occupational group negotiates salaries separately, will likely conflict with the principle of equal pay for work of equal value and the use of internal relativity as the underlying principle of compensation in the public service. In their opinion, maintaining the current bargaining structure and process for negotiating salaries could eventually have a negative impact on any improvement in internal equity brought about by the application of the Standard.

Significant opportunities exist to simplify further

5.72 Our review of the work and studies carried out in the context of the task force indicated that other options would have permitted greater simplification of the occupational group structure and might have offered more flexibility. There is no evidence that such options were considered fully. For example, one option advocated retaining six occupational groupings to replace the current 72 occupational groups. That option also assumed that all jobs would be evaluated using a single job evaluation plan. Its proponents claimed that it would eliminate barriers among the existing groups and would provide much greater flexibility.

5.73 We also found that other organizations - including public sector organizations - have undertaken more vigorous reforms of their occupational structure than those envisaged by the task force. For instance, one organization has reduced its occupational groups from 27 to one. Instead of some 260 pay rates it now has a 16-level pay structure, ranging from cleaner to vice-president. According to senior management, this new structure provides simplified pay administration, increased mobility and management flexibility.

5.74 As part of its reforms, the Australian public service has also simplified its occupational group structure. It now has only five broad groupings, including an equivalent of the General Services group, the Administrative Service Officer.

5.75 Because of the close links between the occupational group structure and other aspects of human resource management in the public service, such as staffing, pay and labour relations, further simplification of the occupational structure and related administrative systems could result in substantial savings, while improving efficiency. For instance, an analysis of some 630 pay scales in the public service (regional rates were excluded) revealed that often differences among them were not significant. When we applied a differential of 4 percent and compared the highest rates of these pay scales, we found that there were fewer than 50 significantly different pay scales in the public service.

5.76 For all these reasons, we believe that more could be done to either simplify the occupational group structure and/or reduce the administrative cost of other systems that results from maintaining a complex occupational structure, such as the current one or even the one proposed.

5.77 Treasury Board Secretariat, in collaboration with unions, departments and other stakeholders as appropriate, should review the proposed occupational group structure to determine how it could be simplified further and/or how its impact on the complexity and cost of related human resource systems such as collective bargaining, staffing, pay administration and others could be significantly reduced.

Cost Neutrality: Not Clearly Defined and Possibly Unrealistic

5.78 At the beginning of the reform project, Treasury Board Secretariat made a commitment that the reform would be "cost-neutral". We found that the term had not been defined clearly and that, as a result, there were several interpretations of its meaning. Furthermore, we found that the cost of operating the existing job evaluation systems was not known with any degree of precision. Treasury Board Secretariat had taken steps to determine the cost of the current systems and the potential savings stemming from the implementation of the new reformed system, but without success. Scenarios and cost estimates we reviewed ranged from $65 million to $200 million a year for some 200,000 classification actions. Partly because of this, at the conclusion of our audit in the fall of 1995, the Secretariat was not in a position to determine with any degree of precision the costs and benefits of the new system.

5.79 Departments' officials did not know how the objective of cost neutrality could be achieved, given the requirements for pay equity and the guarantee contained in current collective agreements that any employee whose position is evaluated to a lower level will not see a reduction in salary. Most departments anticipated that payroll costs would increase as a result of converting to the new system, though they did not know by how much. Some expressed concerns that they may incur higher cost without much significant benefit, because the impact of the new General Services (GE) group on the efficiency of their operations would be more limited than in other departments. In times of restraint, return on investment is an important issue for departments. One department reported having invested some $17 million on the first phase of applying the Standard to approximately 18,000 positions. This figure excluded the cost of converting to the new system. We must recognize, however, that departments whose programs are offered on a standardized basis may have incurred much lower costs up to now. Departments need to know the cost implications as well as the benefits they will reap from the reform.

Treasury Board Secretariat's comments: Treasury Board Secretariat asserts that costs during the development of a Standard, when the fullest range of information is being sought and the training curve is at its steepest, will be higher than those incurred during the actual implementation of the Standard. Therefore, these costs should not be projected as being the likely or typical departmental costs of implementing the Standard.

Treasury Board Secretariat agrees that costs can be reduced by simplified methods during implementation and, to this end, is pursuing a variety of mechanisms including computer-based questionnaires, generic work descriptions, and cost-effective training modules.

5.80 Two years following the tabling of this Report the Treasury Board Secretariat should conduct an evaluation of the progress of the reform. In conducting its evaluation the Secretariat should take into account the views of the various stakeholders and the impact of the reform on them.

Treasury Board Secretariat's response: Treasury Board Secretariat agrees in principle with this recommendation and its scope, and will raise the matter at an early meeting of the UCS Advisory Committee with a recommendation that it be endorsed.

Need for a Different Approach

Treasury Board Secretariat's initial positions contributed to problems

5.81 We feel that some initial positions adopted by the Treasury Board Secretariat contributed to an impasse with some stakeholders and experts and were partly responsible for many of the problems we identified. For example, staff in the Secretariat took the position throughout the audit that as long as male and female work was measured by the same standard, the evaluation of the work was gender-neutral. Such a view supports systemic discrimination: one must ensure that all aspects of men's and women's work are reflected in the components of the Standard.

5.82 We also found that until recently, the Secretariat did not give sufficient consideration to advice it received. For instance, in April 1991 a pay equity expert informed the Secretariat that the proposed methodology for developing the weighting of the Standard's elements was flawed and possibly did not meet the requirements of section 11 of the Canadian Human Rights Act . We have no evidence that the Secretariat examined this serious allegation thoroughly when it was raised. As we have noted in paragraph 5.46, this methodology was one of the causes of the significant imbalances in the weights given to the elements of the Standard.

5.83 The Secretariat's initial interpretation of the exclusive and unilateral right of Treasury Board to develop a classification and job evaluation plan also led to problems. It is true that the Public Service Staff Relations Act explicitly excludes job classification or job evaluation from collective bargaining, and provides for no consultative or negotiation framework for the meaningful joint work required to develop a valid and credible classification and job evaluation plan. However, we think that the complaint mechanism provided in the Canadian Human Rights Act of 1978 requires a change in attitude and approach on the part of the employer, as well as some form of framework to discuss issues related to job evaluation. Employees or their representatives can challenge a job evaluation system developed by the employer, including some of its underlying premises, if they think that it discriminates on the basis of sex. If challenged, the employer must defend its decisions before an independent third party. Also, empirical evidence and jurisprudence point to the fact that systems developed in collaboration with employees are more likely to be balanced and representative of the values of all employees, and less discriminatory.

Until recently, lack of sustained consultation with stakeholders and experts

5.84 We noted a lack of sustained consultation, in particular with certain stakeholders such as the Canadian Human Rights Commission, external pay equity or job evaluation experts and unions, throughout the Standard's development. Although the Secretariat conducted numerous consultations with these stakeholders, they often were on very specific topics, were conducted separately and lacked continuity. No fundamental issues were debated with all the stakeholders or experts present, such as the best method to use to determine the weighting of the elements of the Standard. The lack of sustained consultation resulted in increasing distrust of the Secretariat's approach. Some departments had similar problems with consultations held by the Secretariat. They perceived these consultations as intended more to convey or confirm decisions than to really solicit points of view or analyze and discuss options for solutions before decisions were made.

5.85 At the conclusion of our audit, the Treasury Board Secretariat was putting in place a number of committees aimed at bringing stakeholders and experts together.

Conclusion

The Challenge Only Partially Met

5.86 Almost six years have passed since the task force on classification and job evaluation made its recommendations to simplify the occupational group structure and job evaluation in the public service. Although the development of the Universal Classification Standard is, itself, a remarkable achievement in many ways, given our findings we have to conclude that the challenge posed by the reform has been only partially met.

Significant Issues Need to Be Addressed

5.87 While some significant progress has been made, substantial work must be done, notably to ensure that the Universal Classification Standard is a valid and reliable tool capable of evaluating all jobs in the public service. Similarly, greater efforts have to be made to ensure that work performed by women and traditionally overlooked is made visible and valued appropriately and in the same manner as work performed by men. If the version of the Standard we have reviewed had been implemented, it would not have met the requirements for pay equity and most likely would not have been able to answer a challenge under the Canadian Human Rights Act .

5.88 Though job evaluation with the Standard will be simplified because there will be a single classification standard instead of the 70 now used, the cost to operate the job evaluation system will remain high, particularly if detailed job descriptions continue to be used to collect information about jobs. We believe that more effective and less costly options exist that could significantly reduce the operating costs of job evaluation. In times of constraint, such opportunities must be sought vigorously. Further reduction of the operating cost of job evaluation would ensure greater support from departments.

5.89 Unless there are changes to the proposed occupational group structure, simplification is likely to be significantly less than recommended by the task force or anticipated and desired by many departments for optimum efficiency. Departments agreed, however, that even the proposed structure would be an improvement over the current occupational group structure. Significant opportunities exist to further simplify the structure to take into account the need for improved departmental efficiency and/or reduce the cost of some of the associated systems and practices, such as pay administration. If not enough flexibility is provided to departments, the cost and benefits of the reform - in dollars as well as in human terms - may be spread unevenly.

5.90 While the Treasury Board Secretariat has undertaken the reform with a commitment to "cost-neutrality", this objective was never clearly defined and was possibly unrealistic. At the conclusion of our audit it was not yet in a position to assess with any degree of precision the cost of the current systems or the cost and benefits of the reform.

The Standard Has Potential and Can Be Made to Work

5.91 In our opinion, the Standard is a step in the right direction and its design represents an improvement over the many job evaluation plans now used by departments. The promising features of the Standard, the very significant developmental work accomplished to date and the general satisfaction of most departments lead us to conclude that the Secretariat should pursue its efforts to develop the Standard to meet both the objectives of the reform and the requirements of the Canadian Human Rights Act . All job evaluation or pay equity experts we consulted agreed that, although tough technical issues need to be addressed to ensure that the Standard is valid and credible, these difficulties can be overcome with commitment by all stakeholders, time, resources and improved project management.

Important Measures Are Being Taken to Address Issues Identified

5.92 At the conclusion of our audit, the Secretariat was taking a number of important measures to address issues that we, and other stakeholders, have identified. Although initially they reacted slowly to our concerns about the Universal Classification Standard , we have noted since the fall of 1995 a significant change in attitude among Treasury Board Secretariat's officials in relation to our findings. This change has paved the way for improvement. As mentioned, the Secretariat has taken several measures to obtain advice and points of view from various stakeholders and experts. In addition, important work was being carried out diligently with an objective of ensuring that the Standard will be a valid and reliable instrument capable of evaluating all jobs in the public service, while meeting the requirements of the Canadian Human Rights Act . Although it is too early to assess the net effect of the many measures being taken, they are encouraging.

The Challenge Ahead

The Key to Success: Stakeholders Working Together

5.93 To a large extent, the success of the reform depends on the willingness and ability of the various stakeholders to work together on developing an effective tool for job evaluation and reducing the cost of the system. However, because classification and job evaluation impact significantly on other human resource systems and on the efficiency of program delivery in departments, it is the whole approach to human resource management in the public service that is being challenged by this reform. While the initial focus may be job evaluation and the simplification of the occupational structure, the reform cannot be a piecemeal effort. A significant number of related issues will need to be addressed, such as:

accommodating the need for departmental flexibility while ensuring that job evaluation is managed consistently across departments in which benefits and costs - in human terms, as well as in dollars - may be spread very unevenly;
negotiating salaries in the future and simplifying pay administration in a way that maintains internal relativity in the public service while ensuring that people whose skills are at a premium in the labour market are attracted and retained;
staffing under a simplified occupational group structure while considering the need for specialized knowledge, skills or experience; and
reconciling departments' accountability for results and efficient program delivery, given their limited degree of latitude in such matters as job evaluation, the occupational group structure, collective bargaining and pay.

Reform at a Reasonable Cost to the Taxpayer

5.94 Simplification is necessary to improve efficiency and flexibility and reduce administrative costs, in order to free up the financial resources needed to implement the reform while continuing to provide quality of service to Canadians at a reasonable cost. One reason for the reported success of the simplification of classification and job evaluation in Australia was, according to Australian authorities, the strong incentive provided by the direct link between compensation and improvement in overall productivity. In Australia, labour legislation governing the public service and the "structural efficiency principle" require that increases in pay rates be linked to overall productivity improvement. In the current context of restraint, there might be merit in such an approach.

Treasury Board Secretariat's comments: Treasury Board Secretariat agrees with the Auditor General that classification and job evaluation are considered "the cornerstone of human resource management in the public service", that "the status quo is unacceptable" and that the "standard has potential and can be made to work".

Human resource management systems use departmental resources (including management time) - some of which might be better targeted at providing quality services to taxpayers and reducing the deficit. However, to achieve significant efficiencies in all these human resource systems, we need first to modernize how we value work and classify jobs.

TBS is committed to working co-operatively with all of the principal stakeholders to try to achieve one simple, consistent, affordable work evaluation tool that is capable of evaluating all public service work in a consistent, bias-free way, and that reflects new public service values.

About the Audit

Objective

We wanted to assess whether the development and design of the Universal Classification Standard was appropriate to achieve critical objectives of the reform, such as universality (applicable to the public service as a whole) gender neutrality (free of bias), and simplification. We also wanted to assess whether the proposed occupational group structure would support efficient program delivery, and how thoroughly Treasury Board Secretariat had taken into account the cost/benefit factor.

Scope

Our audit included work performed by the Secretariat to: design and develop tools to collect information on job content; design and develop the new job evaluation plan; determine the relative importance of each element used to evaluate jobs (the "weighting"); test and validate the new plan; apply the new job evaluation plan to a number of positions; and simplify the occupational group structure.

Our audit of the simplification and operational costs of job evaluation as well as of the occupational group structure consisted of a review and an evaluation of the representations made by departments and unions, and of the work and studies conducted in the context of the task force or by Treasury Board Secretariat.

We interviewed deputy or assistant deputy ministers or directors general of human resources in 10 departments, ranging from large to small. These departments represented more than 70 percent of the employees proposed for inclusion in the new General Services group.

We visited six private and public sector organizations that had recently developed a new job evaluation system and that, in many cases, had simplified their occupational group structure, to assess how they had gone about it. We also conducted numerous interviews and consultations with nine other organizations and three compensation management consulting firms.

We also examined documentation on classification reform in Australia's public service, and recent literature in the field of job evaluation and work design.

Criteria

Our examination of the development and design of the Universal Classification Standard used recognized criteria in the field of job evaluation and took into account the government's objectives for the new system.

In the absence of specific criteria for gender neutrality, we developed our own from a variety of sources, including: the Canadian Human Rights Act, the Equal Wage Guidelines, 1986, a publication by the Canadian Human Rights Commission entitled Implementing Pay Equity in the Federal Jurisdiction, rulings by the Ontario Pay Equity Review Tribunal and the Canadian Human Rights Tribunal, and academic research on job evaluation and gender neutrality. Treasury Board Secretariat officials and pay equity experts reviewed our criteria, and changes were made accordingly. Officials from the Canadian Human Rights Commission found our criteria accurate and appropriate. A number of pay equity experts provided us with opinions on the universality and gender neutrality of the Standard.

Audit Team

Ghislaine Côté directed this audit.

For information, please contact Jacques Goyer, the responsible auditor.

Appendix

SUMMARY OF SIGNIFICANT DEFICIENCIES IN THE VARIOUS COMPONENTS OF THE UNIVERSAL CLASSIFICATION STANDARD

Elements

The elements-definition component is intended to explain what the element is supposed to measure. Except for few elements, definitions are vague, broad and at times ambiguous. For example:

"Thinking Challenge" is defined as measuring "the effort of the thinking challenge in terms of complexity of the demands and outcomes associated with the work performed." This definition does not explain what a "demand" and an "outcome" are. Furthermore, it is not clear if the demands and outcomes always vary together in complexity. Although Notes to Evaluators are intended to provide guidelines on how each element is to be applied, they do not provide enough instructions to make sure the evaluator has the appropriate interpretation of what is measured.
"Care of Individuals" is intended to measure "the responsibility to provide direct care to individuals...." It is not known what types of direct care are measured, such as short-term, long-term, critical or non-critical and how these factors should or should not be taken into consideration. Notes to Evaluators include no additional explanation to assist the evaluator in interpreting the purpose of the element.

Degrees and Their Definitions

Degree definitions have been defined on a rating scale to represent a progression from a minimum to a maximum value. In their present form, degree definitions are extremely vague and ambiguous. In fact, except for two elements, degrees are not really defined. They have been framed in "generic" words such as some, moderate, significant, extensive, etc. and they cannot assist the evaluator in establishing meanings for these words or providing rules of differentiation among jobs. For example:

For the "Physical Demands" element, "Limited effort" was defined as "Limited physical effort is required."

Degree Illustrations

Degree illustrations serve to exemplify degrees in concrete job content and to show the progression from one level to another. Because element definitions and degree definitions are too general and broad, degree illustrations constitute the actual criteria against which the job is evaluated. Evaluators rate jobs mainly by examining various degree illustrations at different levels and then deciding the "best fit" for the job evaluated. The actual scales embedded in the degree illustrations drive ratings. As it stands, degree illustrations are the critical component in the Standard and they contain the most serious shortcomings.
The range and breadth of elements are undermined by degree illustrations. In many cases, the criteria included in them are different from those in the element definition and degree definition. For example:

The "Thinking Challenge" element is supposed to measure the complexity of demands and outcomes, while the primary difference between degree illustrations at different levels is in scope of authority.

The criteria embedded in degree illustrations in several elements concentrate too much on the same managerial/hierarchical aspect of work, resulting in too narrow a range of job content. There is too much emphasis on organizational placement, which is the major discernible difference among degrees in elements, even ones that should not be measuring scope of authority.
Some aspects of work are missing, particularly aspects of women's work and work at a high level such as complex collegial decision making, the responsibility of maintaining confidential and sensitive information, the requirement to keep up-to-date in technology, the effort and skills to negotiate without authority, human relations skills, etc.
Based on degree illustrations for some elements, what is measured and the progression from one degree to another are unclear. For example:

The element "Influence" aims at measuring "the extent to which activities, decisions or recommendations influence the service being delivered." Based on degree illustrations, it is not clear what type of influence is being measured, i.e. direct, indirect, contributory, shared or other.

Consistency of degree illustrations is problematic in several cases: within the same degree, there are non-equivalent degree illustrations, and between two degrees there are illustrations with no discernible differences. For example:

The element "Context" aims at measuring "the extent to which knowledge of the context is required to perform the work." The following degree illustration, "The work involves scheduling the output of a data processing centre and determining application specifications, service levels and equipment resource requirements. It requires knowledge of the organization and work flow of the centre and the various operational requirements of the centre's client", is clearly not equivalent to another: " The work involves conserving and protecting the fishery resources in a sub-district of a region and requires knowledge of the mandate and objectives of the department. It also requires knowledge of the fishing industry, the geography of the area, and the jurisdictional relationship within the commercial, sport and native fisheries." Yet both are at the same degree level.

Degree illustrations systematically devalue women's work, notably by placing job content of female-dominated jobs at inappropriately low levels compared with male-dominated jobs. For example:

In the "Thinking Challenge" element, the degree illustration, " The work involves changing dressings to wounds or incisions, taking vital signs, observing and recording changes in condition of patients, reporting any equipment failures and assisting in arranging clinics and scheduling opportunities" was placed at the same degree level as " The work involves establishing cleaning priorities, deciding on the types or strengths of chemical cleaning products to be used and monitoring the quality of services provided by other cleaners."

Another way degree illustrations systematically devalue women's work is by having degree illustrations that do not adequately differentiate job content at various degree levels among female jobs. For example:

In the "Care of Individuals" element, the two nursing jobs in the following degree illustrations have been placed at the same degree levels but are very different in terms of responsibility. The degree illustration, "The work involves providing nursing care to patients in a hospital. Direct care includes carrying out therapeutic measures and diagnostic tests, monitoring symptoms and reactions to treatment and monitoring the progress of patients" requires greater responsibility than the nurses' aide, "The work involves caring for patients. Direct care includes feeding and providing basic hygiene, remedial treatments, comfort measures such as skin care and emotional support." This demonstrates that this element does not differentiate very well among the various degree levels for female jobs.

Instructions given in Notes to Evaluators to credit specific job content are not always reflected in degree illustrations, which means that the job content in those cases is unlikely to be credited by evaluators. For example:

For the element "Influence", the notes suggest considering the responsibility for independent action, or as a team member. This was intended to recognize frequently overlooked female job content, collegial decision making. There is, however, no degree illustration that credits such a responsibility.

A qualitative examination of degree illustrations revealed significant overlapping. Statistical analysis was consistent with this observation and showed a high correlation for 8 out of 16 elements. This has the effect of limiting the universality of the Standard. The more overlapping there is, the higher is the risk that not enough attention will be paid to the full range of job content in an organization.
Statistical tests have been performed to diagnose for systemic effects by gender. One of the tests revealed that 6 of the 16 elements of the Standard measure male and female job content equally well. The 10 remaining elements favour the content of male-dominated jobs. These results are indicators of potential bias in the Standard.

Examples are drawn from the June 1995 version of the Universal Classification Standard issued by the Treasury Board Secretariat to departments.

1996 May Report of the Auditor General of Canada

Chapter 5—The Reform of Classification and Job Evaluation in the Public Service

Main Points