1991 Report of the Auditor General of Canada
Chapter 20—Department of the Secretary of State Official Languages and Translation—Translation Bureau
Important changes in the Translation Bureau environment since 1985 have created a climate of uncertainty
The cost accounting system does not allow the identification of full cost per word translated in-house and contracted out
Direct production costs of translation in the Department of the Secretary of State are higher than in the private sector
The absence of performance standards makes it impossible for the Translation Bureau to determine whether performance is at an acceptable level of efficiency and to identify the potential for improvement
The available management information does not allow management to manage with due regard to economy and efficiency
The absence of an appropriate strategy for human resource management, in an environment of downsizing and increasing use of contractors, has led to loss of expertise and to changes in duties that may explain the level of productivity achieved and the decline in quality
The Department of the Secretary of State has carried out no analysis to identify the costs and benefits, as well as the advantages and disadvantages for its human resources and its organization, of contracting out while maintaining translation quality
Contracting out has resulted in average annual savings of $7.5 million, in direct costs, for the last three years.
During 1990, only 11 of the 24 cases where contractors failed at least four times to comply with deadlines were referred to the Certification Committee
The Certification Committee is very lenient toward contractors who do not comply with contract requirements
The Department of the Secretary of State does not have information essential to the efficient management of contracting out
20.2 Our calculations indicate that direct production costs of translation in the Department of the Secretary of State are higher than in the private sector - 27.3 cents compared to 18.3 cents. Several factors may explain this variance: productivity, salaries and benefits and the regionalized structure of the Department (20.47 to 20.50).
20.3 Given that the Translation Bureau was not identifying its indirect costs related to translation operations, it was unable to monitor them closely enough to identify potential savings (20.52 to 20.54).
20.4 Our 1984 Report recommended that quantitative production standards be established to monitor translator performance. However, management decided instead to set individual quantitative and qualitative objectives. Although translators at the TR2 level have increased their production by 20 words per hour since 1983, 75 percent of them performed below the benchmark of 200 words per hour. We estimate that, had that standard been reached, annual savings of $1.9 million would have resulted. Moreover, translators have little incentive to produce at their highest possible levels. (20.65 to 20.73).
20.5 The Translation Bureau cannot explain the overall decline in the quality of translations since 1987. Consequently, it cannot take appropriate corrective measures (20.83 to 20.92).
20.6 After more than five years of effort, and with annual operating costs of about $2.5 million, the Department's management information system still does not provide, in a timely manner, the information it needs to manage with due regard to economy and efficiency (20.96 to 20.99 and 20.176 to 20.181).
20.7 The absence of an appropriate strategy for human resource management, in an environment of downsizing and of increasing use of contractors, has led to a loss of skills by the Bureau and to changes in duties that may explain the performance level and the decline in quality (20.105 to 20.116).
20.8 Our consultation with employees of the Translation Bureau indicates that the prevailing work climate is poor. Employees have clearly identified negative effects contracting out has on their work, but they are more concerned by the way contracting out is managed than by its use (20.122 to 20.127).
20.9 In the absence of a complete analysis, management was not able to determine the ultimate advantages of contracting out (20.129 to 20.133).
20.10 Since 1988-89, contracting out has resulted in average annual savings of $7.5 million in direct costs. However, this does not take into account the repercussions of contracting out on human resources and on the quality of translation. Better management of operations and human resources would have reduced the negative impacts on the work environment and made the Translation Bureau more efficient (20.137 to 20.141).
20.11 The translator certification process lacks rigour, and the imposition of sanctions on contractors is inequitable (20.151 to 20.154 and 20.160, 20.161).
20.12 The information provided to Parliament in Part III of the Estimates is incomplete (20.187 to 20.190).
Mandate of the Department of the Secretary of State and Objective of the Translation Bureau20.13 The mandate of the Department of the Secretary of State is to foster a sense of belonging to Canada, to assist Canadians in understanding and celebrating their identity and to increase, in one official language or the other, opportunities for the enjoyment of our country's educational, social, economic, political and cultural resources.
20.14 As stated in Part III of the Estimates for the Department of the Secretary of State, the objective of the translation activity is "to provide translation and interpretation services and to standardize terminology for Parliament, and federal departments and agencies, in order to enable the federal government to communicate in the two official languages".
Evolution and History of the Mandate20.15 The Translation Bureau Act of 1934 placed the Translation Bureau under the aegis of the Department of the Secretary of State. In 1987 an amendment to this Act added to the Bureau's mandate the provision of interpretation services, sign language interpretation and terminology.
20.16 From 1939 to 1965 the Translation Bureau faced a gradual increase in the demand for translation services. When the Official Languages Act (1969) became law, new needs and increasing demands compelled the Bureau to intensify its recruiting efforts and extend its operations to regional offices, particularly in Quebec.
20.17 As a result, from 1969 to 1974 the increase in the volume of translation was spectacular. However, from 1985 to 1991 the total number of words translated, including multilingual translation and regional operations, increased from 290 to 313 million words, or by 8 percent in six years - a relatively weak growth rate. The Department is forecasting a demand for 324 million words by 1993-94.
20.18 In 1982 Treasury Board policy made the Translation Bureau a common services agency. All departments and agencies listed in Appendices I and II of the Financial Administration Act were thereby compelled to use the Translation Bureau for all translation services.
20.19 The Translation Bureau has always played a leadership role in the field of translation and linguistics. It has had, and still has, a considerable influence on the translation profession and on the development of the translation industry - an influence that serves to maintain high standards of quality for translations in Canada.
Key Activities and Resources20.20 Exhibit 20.1 illustrates the departmental activities, carried out in a structure of six sectors. The functions described in the Translation Bureau Act are delegated to the Official Languages and Translation Sector and to Regional Offices. The Official Languages and Translation Sector includes Translation Operations, Terminology and Linguistic Services, and Planning, Management and Technology (also called Administration and Special Projects).
20.21 In 1990-91, of the total number of words translated (313 million, as indicated in paragraph 20.17), 42.6 percent were contracted out to the private sector.
20.22 Translation Operations is divided into six directorates of which the largest, the Departmental Translation Services Directorate, has 27 work units. These units, which comprise 52 percent of the person-years in Official Languages Translation Operations, are located in government departments and agencies in the National Capital Region and are responsible for approximately 60 percent of the volume of translation into official languages. In January 1991, 16 of the 27 units were already contracting out more than 50 percent of their workload.
20.23 As can be seen in Exhibit 20.2 , translation expenditures amounted to approximately $108.5 million in 1990-91 and used 1,252 person-years. These figures do not include regional translation operations, estimated at about $4.5 million and 96 person-years.
20.24 The terminology service facilitates research work for numerous users, particularly translators, interpreters and writers in the public service, as well as translation contractors and the general public. The Planning, Management and Technology Directorate is mainly responsible for providing operational support services.
Audit Objective and Scope20.25 The general objective of our audit was to determine whether Translation Bureau activities were managed with due regard to economy and efficiency and whether appropriate procedures were in place to evaluate and report on effectiveness.
20.26 We examined the management of the Bureau, taking into account the activities that are contracted out. Contracting out plays an increasingly significant role in translation operations, resulting in changes that impact on the organization and on the optimization of its resources. We examined the most significant aspects of the management of the Translation Bureau: costs; operational effectiveness in terms of overall productivity and translators' performance; and results measured in terms of client satisfaction and quality of translation. We also examined management information and human resources management. In addition, we looked into the work climate prevailing in the spring of 1991.
20.27 In examining the management of contracting out, we paid particular attention to its justification, the savings it has made possible, and the certification and inventory of contractors. In addition, we reviewed the planning and implementation of the management information system. Finally, we analyzed the information contained in Part III of the Estimates and the program evaluation completed by the Department of the Secretary of State in 1989.
20.28 Our audit focussed on official languages translation because it accounts for approximately 81 percent of the human resources of the Official Languages and Translation Sector (excluding Promotion of Official Languages). Moreover, it is this sector that has seen the most significant increase in contracting out since 1985. Where data permitted, we carried out analyses covering the period 1985-86 to 1990-91.
20.29 In view of the smaller scope of their operations, our audit did not cover the activities of Regional Operations, Interpretation, the Multilingual Directorate, and Terminology and Linguistic Services. However, in calculating costs, we included the activities of Regional Operations and the Multilingual Directorate.
20.30 We interviewed management of Translation Operations, and staff at various levels of the organization. We also had extensive discussions with managers of translation services in a number of quasi-public and private organizations, as well as with contractors who act as suppliers to the Translation Bureau.
20.31 We examined such internal audit reports, reports of studies relating to the Translation Bureau and documents dealing with Bureau activities as we deemed necessary.
The Environment Since 1985
Important changes in the Translation Bureau environment since 1985 have created a climate of uncertainty20.32 Exhibit 20.3 describes four external events that have had a bearing on Translation Bureau operations since 1985. These events have significantly altered the operational framework in which the Bureau has evolved, and have raised several questions about what kind of organization it will become.
20.33 In 1991, the Bureau finds itself in a situation where it contracts out over 40 percent of its operations. Moreover, pilot projects are under way to eventually permit departments to choose between the Translation Bureau and the private sector when they need translation services. The Bureau will thus have to give increasing consideration to the private sector's performance. This is an important turnabout since, for about 15 years, until 1985, the private sector served as a safety valve allowing the Bureau to handle a surplus of work. Since 1985 the private sector has been considered a partner in the achievement of Bureau objectives. Today, the private sector presence demands that the Bureau develop a concerted strategy for defining its goals and the means it will use to achieve them.
20.34 In March 1990, to identify the means of adjusting its resources and operations to Treasury Board requirements, the Translation Operations Branch created a task force called Strategies 1991-94. Ten sub-working-groups formulated important recommendations for maintaining quality, streamlining procedures, and training staff on new methods. Some of these have been implemented. As of July 1991, however, implementation of most recommendations was still at the planning stage and remained dependent on available resources.
20.35 From 1985 to 1991, there were several changes in official languages translation operations affecting workloads and resources. On the one hand, annual fluctuations aside, the volume of translation in official languages alone remained static at around 250 million words. On the other hand, again in official languages alone, the Bureau was cut by 307 person- years, a decrease of 23 percent.
20.36 The use of contracting out was in line with a government objective to realign resources between the public and private sectors, and was intended to result in savings. In the Translation Bureau, contracting out compensated for the loss of person-years. Exhibit 20.4 shows the extent to which contracting out for official languages translation has consistently increased since 1985-86. At that time the private sector produced 44.4 million words, whereas today its market share is 113.7 million words. So in 1990-91, 41 percent of official languages translation was contracted out. Annual expenditures for translation contracts have risen from $6.4 million to $21.6 million over the last six years.
20.37 The Translation Bureau is operating in a changing environment even as it tries to redefine the type of organization it wants to become, and its future is uncertain. We took these factors into account during our audit.
Management of the Translation Bureau20.38 The management of operations requires information on cost and performance. As we indicated in paragraph 8.99 of the special study on Efficiency in Government published in our 1990 Report, cost and performance information "will allow managers to set performance standards, monitor results, and analyze variances, which helps in identifying potential improvements. Such information is critical to any accountability process where program managers are expected to provide an acceptable level of service at the lowest possible cost."
20.39 Results in translation are obtained largely through the use of human resources. The efficiency of the Bureau is therefore based on meeting appropriate translator performance standards with consideration to the quality of the product.
Translation Costs20.40 For a service organization such as the Translation Bureau, the cost of an output - words translated, in this case - is a significant indicator of performance. A good cost system provides the costs of production, both direct and indirect, and allows for their monitoring at various level of responsibility.
20.41 An explanation of some terms used in the following paragraphs may be helpful here. Direct costs are those most directly allocated to an output, without additional computations. Indirect costs are not easily linked to an output; they are added to direct costs through various allocation formulas to arrive at the full cost. To find the unit cost, the total cost is divided by the number of words translated. Exhibit 20.5 shows the elements that make up in-house and contract costs.
The cost accounting system does not allow the identification of full cost per word translated in-house and contracted out20.42 In paragraphs 14.12 and 14.13 of our 1984 Report chapter on the Translation Bureau, we recommended the implementation of an appropriate cost accounting system to allow the Bureau to analyze the respective costs and benefits of using in-house resources or private sector contractors. This system also was to facilitate the development of a strategy for achieving the Bureau's objectives in the most economical manner possible. A June 1984 independent study, as well as internal studies, have also stressed the need for the Translation Bureau to develop such a cost system, to allow it to determine such things as the optimum threshold for contracting out.
20.43 Since 1985, the Translation Bureau has developed a cost accounting system based mainly on the direct cost per word translated. We evaluated the methodology the system uses and identified three main deficiencies:
- Except for the costs of computer equipment and software, which represent only a small portion of indirect costs, the method does not take into account indirect costs (see Exhibit 20.5 ) incurred by the Official Languages and Translation Sector and other sectors of the Department of the Secretary of State that provide support services.
- The cost per word translated on contract adds, to the amount paid to contractors, certain other costs incurred by the Department of the Secretary of State for contract management. Yet it does not include the costs for certain services provided free of charge to contractors, such as photocopies, access to the library and, in some cases, an office.
- The present cost accounting system does not provide, on a monthly basis, information on the direct costs per word translated in-house for the whole of the Translation Bureau, even though such information is available from detailed reports on each responsibility centre.
20.45 We thus concluded that, as of January 1991, the cost accounting system of the Department of the Secretary of State was not providing management with complete information on the cost per word. In the interest of value for money and sound decision making, it is imperative that the cost accounting system calculate accurately and completely the cost per word translated, both in-house and contracted out. In July 1991, departmental officials informed us that they were in the process of modifying the system.
20.46 In modifying its cost accounting system, the Department of the Secretary of State should:
- amend its method of determining the cost per word translated, both in-house and contracted out, to determine the full cost and thus to better manage its resources; and
- amend the cost accounting system so that it will provide management, on a monthly basis, with information on the direct cost per word translated for the organization as a whole.
Direct production costs of translation in the Department of the Secretary of State are higher than in the private sector20.47 The price paid to contractors per word translated includes three elements: direct costs, overhead and a profit margin. These three elements are an integral component of any viable business, regardless of its size and its legal and operational structure.
20.48 Exhibit 20.6 compares the direct production cost per word translated in the Department of the Secretary of State to the cost in the private sector. The Department pays contractors, on average, 18.4 cents per word translated. To make a fair and equitable comparison, we estimated the direct production cost in the private sector. We considered the average price of 18.4 cents paid to contractors to include 4 cents for their overhead and profit margin, which leaves an amount of 14.4 cents in direct production costs. This estimate is based on our discussions with several contractors and we consider it reasonable. In other words, this amount of 14.4 cents represents what it costs per word in the private sector to translate for the Department of the Secretary of State. However, to make a fair comparison, 3.9 cents must be added for direct costs of activities that the Department must carry out to manage contracting. Those functions are already included in the cost per word translated in-house.
20.49 Exhibit 20.6 indicates that the direct production cost per word translated in the Translation Bureau is 27.3 cents, compared to 18.3 cents in the private sector. We note, however, that the Bureau's computer-assisted translation allowed it to increase its overall efficiency and reduce the direct cost per word translated in-house, for 1990-91, from 29.9 cents ( Exhibit 20.17 ) to 27.3 cents ( Exhibit 20.6 ).
20.50 Although the Bureau did not attempt to analyze these differences, several factors may explain this cost variance:
- the level of productivity in the Translation Bureau compared with that in the private sector;
- respective salaries and benefits; and
- the regional structure of the Department of the Secretary of State .
Department's response: Agreed. The Department of the Secretary of State will examine the various factors that might explain the gap between internal and external direct production costs in order to determine to what extent the gap can be reduced.
Potential reduction of indirect costs20.52 Total indirect costs related to translation - those of the Official Languages and Translation Sector as well as those of other sectors of the Department of the Secretary of State - are equivalent to 46 percent of the amount for direct costs.
20.53 While these costs increased from $30 million to $36 million in the last three years, in real dollars they have remained relatively stable. Unlike direct costs, total indirect costs do not vary proportionately to the volume of production. Nevertheless, given the increase in contracting out in the last three years, we expected to find a decrease in indirect costs, or an analysis of the reasons why they had not been reduced.
20.54 In view of the deficiencies mentioned in paragraph 20.43, the Department of the Secretary of State has been unable to identify indirect costs and thus to reduce them, where possible. The Department maintains that, in the short term and during this period of transition, such reductions are not attainable.
20.55 Given that the Department of the Secretary of State will have to give increasing consideration to the private sector's performance, it should identify and monitor its indirect costs to better determine the potential for savings.
Department's response: Agreed.
20.56 Full cost. Full cost, as defined in paragraph 20.41, is the sum of direct and indirect costs. It is used mainly for setting prices, since it helps to ensure the recovery of all costs.
20.57 Treasury Board policy on common services advocates the use of full cost to set prices in an environment of revenue dependency. However, for organizations that "compete directly with private sector organizations providing similar services", market price is to be used in cost recovery. If the market price does not allow the organization to recover all its costs, the resulting losses "require analysis to determine their cause and to develop plans to eliminate them." A special operating agency could be called upon to comply with similar terms and conditions in setting prices for its services.
20.58 The Department of the Secretary of State accepted the recommendation of a May 1989 program evaluation that a cost recovery policy be implemented, and is currently reviewing this subject. At the time of our audit, departmental managers informed us that they were investigating a number of possibilities, such as the Bureau's becoming a special operating agency.
In-house full cost is higher than the market price20.59 Exhibit 20.7 shows the full cost per word translated in-house, which has increased since 1988-89 from 39.2 cents to 41.6 cents.
20.60 However, the market price is harder to determine. The starting point would be the average price currently paid to contractors, 18.4 cents per word ( Exhibit 20.6 ). If we take into consideration the costs of certain services, which would no longer be provided to contractors free of charge in a context of direct competition, the market price would be somewhere between 21 and 24 cents per word.
20.61 If the Translation Bureau relied on charging the market price to recover its costs, it would incur large annual losses. This underlines the importance of our preceding recommendations on identifying where in- house costs could be reduced.
20.63 To measure the efficiency of its internal resources, the Translation Bureau uses overall productivity. This measure of performance, expressed as the ratio of production (the number of words translated to person-years used in translation operations), deals exclusively with hours devoted to translation activities and therefore excludes the time spent on the management of contracting out, or on the provision of linguistic advice. For the last three years, 80 percent, on average, of all person-years assigned to translation in official languages has been included in overall productivity calculation. This includes the time of translators, revisors, operations managers and support staff.
20.64 The production of each translator, expressed as the number of words translated, is the main element in the measurement of overall productivity. Although we recognize that translation is an intellectual activity, setting performance standards is nevertheless essential to sound management.
The absence of performance standards makes it impossible for the Translation Bureau to determine whether performance is at an acceptable level of efficiency and to identify the potential for improvement20.65 In the Department of the Secretary of State, the translator group, known as the "TR group", includes translators, interpreters, and terminologists. As of March 1991 the Official Languages and Translation Sector had a complement of 992 in this group, including 790 official languages translators. Exhibit 20.8 shows the distribution of translators by level and outlines their duties.
20.66 We reviewed, for the period 1988-89 to 1990-91, the production of 19 percent of translators at the TR2 level, or 287 translators. These translators were in 10 different work units, selected for their representative size.
20.67 In 1984 we reviewed the performance of translators and recommended in paragraph 14.19 that production standards be established and monitored. In June of that year, an independent study made the same recommendation. In addition, it stressed that several Translation Bureau managers considered standards one of the few objective means of assessing the performance of translators.
20.68 In 1980 management abandoned the idea of a universal standard prescribing the minimum number of words expected at each translator level. Current policy instead provides for the setting of quantitative and qualitative objectives for each translator by his or her immediate supervisor, usually at the time of the annual performance reviews. Moreover, we found that the Bureau has used quantitative standards as advancement criteria for translators.
20.69 Although there are no universal standards in the translation profession, the private sector nevertheless uses various methods of performance measurement. Sometimes a minimum required number of words is specified on employment contracts; bonuses may also be given for work beyond the established threshold but, in most cases, private sector translators are paid by the word.
20.70 We found that the Bureau's translators over the last three years have consistently, on average, translated 180 words per hour, an increase of 20 words per hour since 1982-83. Against the same production capability, used in 1984 as a benchmark to differentiate between translators at the TR1 and TR2 levels for promotion and planning purposes (300,000 words per year or 200 words per hour), we found that 75 percent of our sample of translators at the TR2 level did not meet this level. We also noted that 35 percent of translators in our sample had not had individual quantitative objectives set in their annual performance evaluations, as required by current policy.
20.71 We found that translators in the Department have little incentive to produce their best possible performance. We noted that, between 1985-86 and 1989-90, barely one percent of translators at the TR2 and TR3 levels were promoted. As of 15 July 1991, 73 percent of all translators had reached the maximum of their pay scale. We found no other measures aimed at rewarding good performance. This situation can have a significant negative impact on motivation.
20.72 In our opinion, the absence of performance standards means that management is unable to objectively evaluate whether translators are performing satisfactorily. This makes it difficult for the Bureau to identify the optimal level of productivity for comparison to the actual level reached. Thus, the Bureau does not know whether its improvement in productivity of 1.7 percent per year since 1986-87 represents what it should have achieved. Moreover, it is not able to make the management decisions needed to improve efficiency.
20.73 Translators' performance also has a marked impact on in- house costs of production, and optimizing it would result in substantial savings. Based on our sample, if all TR2 level translators had achieved a performance level of 200 words per hour while assigned to translation duties, we estimate that, from 1988-89 to 1990-91, an average of 10.6 million additional words per year would have been translated in-house. This would have produced annual savings of $1.9 million for the government.
20.74 To optimize translators' performance, the Translation Bureau should, in consultation and with the participation of employees:
- establish, by employee group or by sector, standards that would allow it to objectively measure the performance of translators and to plan the Bureau's workload; and
- implement measures to recognize and reward those who meet or exceed established performance standards.
The Bureau has already taken a series of measures to recognize employee contributions (meetings, awarding of plaques, publication of achievements and so forth), uses existing programs such as Suggestion Awards and, with Personnel, has studied a series of possible additional measures that conform to the recommendations of PS 2000.
Client Satisfaction and Quality of Texts Translated20.75 An important objective of the Translation Bureau is to ensure a high level of translation quality and service to clients. Thus, the Bureau has defined important indicators which are used to measure client satisfaction and quality of translations.
20.76 To achieve this objective, the Translation Bureau in 1986 implemented a "Continuous Evaluation System". It provides for the evaluation of client satisfaction and quality of translations from a sample of approximately 1,600 texts, and for quarterly reports to management on the results.
20.77 Each month, a questionnaire is attached to a number of translated texts upon their return to the originator, asking for an indication of the degree of satisfaction with various aspects of translation services. From a sample of these translated texts, the Linguistics Services Directorate evaluates their quality and, based on the standard in use called SICAL, rates them from A (superior) to D, depending on the number of major and minor errors detected.
20.78 We reviewed the results of surveys of client satisfaction and quality of translation for the last five years. We also reviewed the two reports of the Government Consulting Group commissioned by the Department. The first, in February 1990, was an evaluation of the Continuous Evaluation System; the second, in February 1991, dealt with the size of the sample required to draw conclusions on the quality of texts from various subgroups.
20.79 This second report recommended a sample size of about 400 texts from each group or subgroup on which conclusions are desired for instance, translations from French to English and translation done in-house or contracted out.
20.80 We believe that, to evaluate client satisfaction and quality of translation:
- surveys should be carried out in accordance with recognized sampling methods; and
- evaluations should be objective and based on reasonable standards and results should be communicated to management in a timely manner.
The level of client satisfaction remains high20.81 The results of the last five years' surveys show that overall client satisfaction remains high. Results from 1990-91, shown in Exhibit 20.9 , are representative of those from previous years. It will be noted that 96 percent of respondents described the official languages translation services as satisfactory or better. At the same time, we found that adherence to deadlines has consistently been the element with which clients have expressed the least satisfaction.
20.82 In surveys used to measure client satisfaction, we noted that the exact number of questionnaires sent is not noted. As a result, the Bureau cannot determine the percentage of responses received from clients; the response rate for 1988-89 was estimated at approximately 50 percent. Moreover, it does not follow up with a sample of non-respondents to determine whether their level of satisfaction corresponds to that of respondents. Finally, surveys do not take into account whether texts were translated in-house or were contracted out. Management plans to correct this last situation in 1991-92.
The Translation Bureau cannot explain the overall decline in the quality of translations20.83 In addition to measuring client satisfaction, the Translation Bureau carries out its own objective evaluation of translated texts in order to maintain high standards. The Bureau aspires to deliver texts that are faithfully translated, grammatically correct and at levels A and B of the SICAL standard. Exhibit 20.10 shows that, from 1987-88, the overall quality of translated texts began to decline. Indeed, the proportion of texts at levels A and B went from 71 to 54 percent, a 24 percent decrease, between 1987-88 and 1989-90. At the time of this report, in July 1991, results of the 1990-91 survey were not yet available.
20.84 Surveys of translation quality give only general results that cannot explain the level of quality attained. Three main factors explain this: the objectives, the size of the samples, and the methodology used.
20.85 The objectives of the Continuous Evaluation System are not designed to provide management with data that are sufficiently comprehensive to enable it to analyze the results.
20.86 In 1988-89 the Translation Bureau used a sample of 407 texts and in 1989-90 a sample of 377 texts to evaluate the quality of translations. Such a small sample does not make it possible to attribute the decline in quality to specific significant factors. For example, the Bureau cannot conclude whether the decline is attributable to in-house or contracted work. To do so, the sampling method would have to take into account factors that could explain the results. The Bureau is also unable to precisely link the decline in quality to other significant factors, such as the nature of texts, the skills in the organization, the work climate or the increasing amount of translation done autonomously, that is, not subjected to supervisory review.
20.87 Moreover, we noted that, from 1986-87 to 1989-90, the size of the sample for purposes of quality evaluation went from 712 to 377 texts, a decrease of 47 percent. During the same period, the number of words translated in official languages increased by 11 percent.
20.88 With respect to the methodology, we noted that quality evaluation currently is done without taking into account the expected level of quality related to the SICAL standard, established by the manager, based on the nature of the text and other factors.
The Translation Bureau is not able to take appropriate corrective measures20.89 Although the system provides for quarterly reports, the results of surveys are analyzed only once a year, several months after the end of the fiscal year. Consequently, besides the fact that only limited analysis of the surveys is possible, reports are not available in a timely manner and do not allow for identification of the possible causes of any lack of efficiency or decline in quality.
20.90 We also noted that the Department of the Secretary of State has not determined the level of satisfaction and of quality below which corrective measures are to be taken. Analysis is limited to a comparison with previous years' results.
20.91 The February 1990 report of the Government Consulting Group noted that, in general, the method for surveying client satisfaction was appropriate; but it voiced reservations about the low rate of response and the limited use of the survey. It also underlined the lack of monitoring by management of compliance with system guidelines. The survey of quality noted the same weaknesses as those mentioned in paragraphs 20.83 to 20.89.
20.92 In our opinion, the Continuous Evaluation System as implemented does not provide management with the information it needs to explain the level of satisfaction and of quality achieved and to take appropriate measures to improve them.
20.93 The Department of the Secretary of State should:
- review the objectives and methodology of the Continuous Evaluation System;
- carry out a cost-benefit analysis to determine the factors to be taken into account in the choice of a sampling method; for instance, the nature of texts, the expertise required, and deadlines;
- segregate the results of the surveys of client satisfaction and quality into the two most significant categories, namely, texts translated in-house and those translated by contractors;
- determine the level of satisfaction and of quality below which corrective measures should be taken; and
- carry out quarterly analysis of survey results and report the results of these surveys promptly to management.
Management Information20.94 Management should have at its disposal timely and relevant information to manage costs and performance.
20.95 Cost data, performance indicators, and operational data are the three main sources of information used in the management of translation operations. Computer systems of the Department of the Secretary of State record most of the data related to the cost per word and the quality of texts translated. They also accumulate operational data such as volume of work, distribution of hours of work, nature of texts, deadlines, work not subjected to revision, etc. These data make it possible to establish the direct cost per word translated and to ascertain the overall results of quality surveys. They are also used to measure productivity and provide significant information on translation operations.
The available management information does not allow management to manage with due regard to economy and efficiency20.96 Most cost and operational data are available only at the work unit level. They cannot easily and rapidly be consolidated for timely access by management.
20.97 The Bureau's performance indicators, such as overall productivity, quality of texts and client satisfaction, are general in nature. Data that could explain the results achieved are neither collected nor provided to management on a regular basis.
20.98 In the present circumstances, the wealth of data available cannot be used effectively and regularly to highlight the causes of and factors influencing differences in cost, performance, productivity, and quality between in-house production and translations contracted out. Paragraphs 20.176 to 20.181 outline the deficiencies we identified in the implementation of management information systems.
20.99 In its reports, management has offered several explanations for costs and performance that it is unable to substantiate. Consequently, it cannot take the appropriate measures needed to improve organizational performance.
20.100 The Department of the Secretary of State should put in place the mechanisms necessary to make available to management, in a timely manner, the information it needs to manage costs and performance.
Department's response: See response to the recommendation at paragraph 20.182.
Human Resource Management20.101 Notwithstanding technological advances, translators are essential to the achievement of the federal government's communication objectives, in both official languages. Their skills, commitment and motivation are, therefore, factors of prime importance that must be valued and cultivated.
20.102 Our analyses are based on computerized data gathered by the Department of Supply and Services and on internal studies completed by the Department of the Secretary of State at the time of our audit. They cover the six-year period from 1985-86 to 1990-91.
20.103 Specifically, we analyzed variances in the translator group profile, i.e. group distribution by level, seniority, specialization, and age. At the same time, we reviewed the reasons for separations, absenteeism, and rates of turnover and retention in the organization. Finally, we noted several characteristics of translators who left the Department of the Secretary of State and of those recruited since 1985-86.
20.104 The reduction in staff, coupled with an increase in the use of contractors, created significant changes in the Translation Bureau. Such an environment places special demands on human resource management; for instance, it requires a management philosophy based on a recognition of the value of human resources, a well-defined strategy and action plan, and frequent and timely communication aimed at keeping staff informed of senior management's intentions and at answering concerns. Recognizing competent employees and the importance of their contribution, assessing and rewarding their performance and doing whatever is needed to retain them are recognized as sound principles of human resource management. They help to minimize the negative effects of change and to create a climate conducive to good performance.
The absence of an appropriate strategy for human resource management, in an environment of downsizing and increasing use of contractors, has led to loss of expertise and to changes in duties that may explain the level of productivity achieved and the decline in quality20.105 Although the Translation Bureau has recognized the importance of its human resources, it has not yet developed a strategy or a policy to manage downsizing, to plan for its future needs and the skills required of its staff. Such a strategy would, among other things, have allowed the Bureau to develop plans to manage the downsizing and to prepare employees for the type of organization which it is becoming. The precise effect that the use of contractors has had on employees has not been determined, and no plans have been developed to deal with it.
20.106 The five-year plan for personnel reduction developed by the Department of the Secretary of State in 1985-86 defined its needs in mainly quantitative terms. Following lay-offs during the first year of the plan's implementation, management considered that the rate of attrition would thereafter accomplish the required reduction in personnel. In December 1990, as part of the Strategies 1991-94 study, a Translation Bureau task force made recommendations aimed at adjusting resources and operations to the new context. In July 1991 the Bureau developed action plans. Today, translators and support staff still do not know the details of the plans and how management intends to implement them, or their role in achieving the Bureau's medium- and long-term objectives. This has had a number of repercussions that we outline below.
20.107 In March 1991 the Department of the Secretary of State had lost 33 percent of the TR group on staff at the start of 1985-86. During this period, 410 employees in the TR group working in the Official Languages and Translation Sector left the Department. This number rises to 475 if separations in March 1991 due to early retirement are included. Consequently, during this period the Department had to recruit 227 TR group employees for the Official Languages and Translation Sector.
20.108 We found that, from 1986-87 to 1990-91, the average rate of turnover was 50 percent higher than in 1984-85, i.e. 7.6 percent for the Sector as a whole. Of 410 translators who left, 50 percent resigned voluntarily. Of these same 410 translators, 85 percent left the federal public service.
20.109 During this period, two factors in particular contributed to the exodus of translators. On the one hand, the Department of the Secretary of State decided to help develop the inventory of contractors, which had not been sufficient to absorb the excess workload; in effect it encouraged its translators to try working as contractors. On the other hand, the private sector held an attraction for the Bureau's good translators, in part because of the assurance of an ample supply of work, piece-work remuneration, and the prospect of concentrating almost exclusively on translation. As of 31 March 1991, at least 34 percent of official languages translators who had left the Translation Bureau were listed in the Department's inventory of contractors.
20.110 Several indicators show that the Translation Bureau has suffered losses of skills that it has not been able to measure accurately. Translators' skills are not recorded and the Bureau is not able to measure its losses. However, there are several indications that the Bureau has lost some of its more experienced translators and has had to replace them with untrained recruits.
20.111 In official languages translation, 50 percent of translators who left had more than 10 year's service; 68 percent were still in the early or middle stages of their careers since they were less than 45 years old. Several of these translators (the number cannot be determined accurately) worked as specialists. Furthermore, the Bureau has had to fill more than half of the positions vacated by experienced translators with recruits in need of training. As shown in Exhibit 20.11 , 84 percent of the 208 recruits were at the TR1 level, whereas 82 percent of those who left had been at the TR2 level or higher. Moreover, the Translation Bureau is encountering difficulties in retaining its recruits: almost one-third of the translators hired by the Bureau at the TR2 level or higher left within five years.
20.112 The loss of expertise explains several of the negative repercussions that the Department of the Secretary of State began to experience in March of 1991. For instance, it has left weaknesses in critical areas that, in the medium and long term, will have an impact on costs and cause the organization to lose the economic advantage it once had. Moreover, the departure of so many experienced translators diminishes the quality of supervision that can be provided to trainees. The ensuing intense staffing activities also generate recruiting and training costs. For many years, the Translation Bureau has found it difficult to recruit translators. According to Bureau managers, it usually takes five years of experience in translation to become fully operational. Finally, the inexperience of recruits is likely to have a negative impact on the organization's overall level of productivity and on the average performance of translators.
20.113 Changes in duties mean that translators in the Department of the Secretary of State spend less and less time translating. An analysis of hours and work distribution for official languages translation personnel shows ( Exhibit 20.12 ) that, in 1990-91, translators spent only 58 percent of their time translating - 8 percent less than in 1987-88. Translators now spend three times as much time as in 1987-88 managing contracts, and twice as much providing linguistic advice to their clients. As for support staff, Exhibit 20.13 shows that they spend 9 percent less time typing but 12 times more on activities related to contracting out.
20.114 Moreover, because of the increased volume of work contracted out, in April 1990 the Translation Operations Directorate created the position of controller, responsible for ensuring that contractors comply with the requirements of their contracts, including those related to quality. Controllers have been selected from among experienced translators and their time is spent almost exclusively in this function. They represent 4 percent of all TR3 translators working in official languages translation. Furthermore, managers maintain that in-house translation increasingly involves urgent texts and, more and more, short texts. There are no data available on this subject to determine to what extent this change has occurred.
20.115 Professionals in the field stress the need for concentration and the fact that skills and performance improve with experience. Changes in the nature of the translation, the impact of the distraction of communicating with contractors, and the diversification of tasks may, in part, account for the levels of overall productivity and performance of translators.
20.116 We recognize the demands on, and difficulties for, human resource management during periods of change and transition. However, our audit identified some significant repercussions on the Bureau's human resources. In our opinion, they have been due primarily to an inappropriate management strategy, insufficient analysis of the impact of change, and communication that leaves much to be desired. Management's exercise of leadership has been more reactive than proactive.
20.117 To improve the management of its human resources, the Translation Bureau should:
- develop, in consultation with employees, a management strategy as well as its medium- and long-term objectives, the means of achieving them and the kind of resources needed. This strategy as well as the plan for its implementation, should be communicated to staff;
- identify the skills at its disposal to better define those it needs to acquire;
- develop methods to counter the negative effects on its staff of downsizing and contracting out.
The Work Climate20.118 The work climate in any organization constitutes an important element of operational effectiveness. Despite the significant changes during the six years covered by our audit, the Department of the Secretary of State has not conducted a study of the work climate. In view of these two factors, we consulted all the staff working directly in official languages translation, including Regional Operations personnel.
20.119 The objectives of our consultation were: first, to survey as accurately as possible the state of the work climate prevailing in the Translation Bureau in the spring of 1991, and second, to determine whether there was a relationship between the increasing use of contractors and this climate.
20.120 To this end, we met with 117 middle managers, translators and support staff, in groups, to identify major trends with positive and negative impacts on the work climate. To validate what we learned from these consultations, we later submitted a questionnaire to 1,021 persons working in official languages translation. We received 621 completed questionnaires, a response rate of 61 percent. The results were discussed with Bureau management.
20.121 Finally, we note that the profile of replies was essentially the same regardless of whether respondents were French- or English-speaking, worked in the National Capital Region or elsewhere in Canada, had little or a lot of experience or were managers or translators.
In general, the present work climate in the Translation Bureau is poor20.122 As Exhibit 20.14 shows, 68.5 percent of respondents perceived the general working climate as somewhat negative or very negative, while barely 3.6 percent saw it as somewhat positive or very positive. Moreover, Exhibit 20.15 shows that three quarters of respondents perceived a deterioration since April 1988.
20.123 The consultation attempted to identify the factors that had the most positive or negative influence on morale in the workplace. Support from colleagues, professional autonomy, the nature of the work, work tools and client response were the most important sources of work satisfaction. In other words, everything touching on the practice of the profession was seen as very positive. As a sign of the translators' professionalism and the importance they attach to their work, we found that absenteeism among translators has remained fairly stable since 1986-87.
20.124 Conversely, we found that respondents viewed the future of the organization, the increasing use of contracting out, the lack of support from senior management, and job security as factors with the most negative impact on the work climate.
20.125 The results of our survey indicated that Translation Bureau employees enjoyed their work, but were highly dissatisfied with the environment in which they had to perform it.
20.126 Moreover, the results showed an obvious link between the deterioration of the work climate and the increase in contracting out. Indeed, for over two thirds of respondents, contracting out had a negative impact on morale. The increased use of contracting out symbolized, for them, the decreasing relevance of the organization and its anticipated disappearance. Three quarters of the respondents perceived it as a factor that reduced the quality of both the products and services of the Translation Bureau and diminished the significance of their duties.
20.127 Still addressing perceptions, we note that the staff considered contracting out to be poorly managed. In fact, respondents were more concerned with the management of contracting out than with its use. Three quarters of them judged the quality of communication to be negative. They viewed the management of downsizing as ad hoc, without consultation or clear and accurate information. People had the impression of being cut adrift and left to their own devices. We noted, in our analysis of communiqués, that management had provided little information to the staff about the repercussions of contracting out and about the future of the organization.
20.128 To improve the work climate, the Department of the Secretary of State should:
- decide on the type or organization it wants to become and on the concrete steps it intends to take to achieve this;
- communicate to employees the direction it will take and the measures it plans to counteract the anticipated impact on the staff; and
- monitor changes in the work climate.
Management of Contracting Out
Justification of the Use of Contracting Out: The Costs and Benefits20.129 For a long time now, the Translation Bureau has been contracting out translation work. As far back as 1968-69, 17 percent of translations were being contracted out.
20.130 The Treasury Board and the government have often stated that contracting out must be considered only when it is clearly demonstrated to be advantageous. In September 1985 the President of the Treasury Board declared to the National Joint Council that:
"Contracting out is not an end in itself and the Government does not intend to simply expand its activities in this area no matter what the cost. Our approach is one which involves a detailed analysis of all proposed initiatives from a human, financial, economic and political point of view. The results of the analysis must demonstrate a particular proposal will result in real dollar savings to the Canadian public, not just artificial person-year savings."He added that restraints would not be undertaken on the backs of Public Service employees.
20.131 In this environment, we expected that such analysis would have been done to support the growing trend in the Translation Bureau toward contracting out. Exhibit 20.16 outlines the criteria that should be used in support of the decision to "make or buy".
The Department of the Secretary of State has carried out no analysis to identify the costs and benefits, as well as the advantages and disadvantages for its human resources and its organization, of contracting out while maintaining translation quality20.132 A 1984 independent study recalled that, at the beginning of the 1980s, the Bureau maintained that translation contracted out was costing less than that done in-house. From 1985 to 1989, as the proportion of contracted translation grew from 20 to 35 percent, the Bureau carried out no analysis to justify this increase. However, a March 1989 program evaluation established the financial advantages of contracting out. In fact, the Translation Bureau has not done any comprehensive analysis of the impact of contracting out so as to determine whether there are overall advantages and to set its optimal level.
20.133 In March 1991, following the federal budget, which resulted in the early retirement of 65 experienced translators, the Department of the Secretary of State pointed out to the Treasury Board, for the first time since 1985, some effects of the increasing use of contracting out. It outlined the critical situation in which the Bureau could find itself if staff reductions continued and the demand for translation increased as much as anticipated. It showed, among other things, the negative impact on translation quality and on the potential for recruitment, and the expected increase in unit costs due, in part, to the emergence of specialized sectors in which the Bureau would lack in-house translation skills.
20.134 The Department of the Secretary of State should carry out comprehensive analyses of the costs and benefits of contracting out and its effects on its human resources and organization to ensure that it is managing translation operations with due regard to economy and efficiency.
Department's response: The Department of the Secretary of State recognizes the usefulness of analyzing the costs of in-house work and contracting out. The evaluation of the direct purchasing pilot projects in 1992-1993 will provide an opportunity to respond to this recommendation and to set up a process of efficient management. Meanwhile, the priority will remain the delivery of quality services that meet clients' needs.
Savings from Contracting Out20.135 We tried to estimate the savings realized by contracting out. In calculating savings per word, we took into account various factors that could affect our comparison of the cost per word translated in-house and contracted out. Thus, the cost per word translated in-house was recalculated to exclude the costs related to computer-assisted translation, since this activity is not contracted out. This adjustment raises the in-house cost in 1990-91 to 29.9 cents, as shown in Exhibit 20.17 , from 27.3 cents indicated in Exhibit 20.6 .
20.136 We included, in the 1990-91 cost per word contracted out, an amount of 4.7 cents that represents supplementary costs relating to contracting out. These costs cover such things as procurement and quality control activities and services supplied free of charge to contractors.
Contracting out has resulted in average annual savings of $7.5 million, in direct costs, for the last three years.20.137 As shown in Exhibit 20.17 , the savings in direct costs resulting from contracting out in 1990-91 were $9 million. Our analysis of the previous two years shows similar results. In 1989-90, these savings amounted to $6.6 million and in 1988-89, to $6.8 million. These savings do not take into account the effects of contracting out on human resources and on translation quality.
In the final analysis, has contracting out been advantageous?20.138 It is up to management to determine the costs and benefits of contracting out. Throughout this chapter, we have stressed the absence of such analysis and have discussed various aspects of contracting out and its impact on the Translation Bureau. Based on available data, we have attempted to establish the pros and cons of contracting out and to determine whether, in fact, it has been advantageous.
20.139 As mentioned in the previous observation, the Department has realized average annual savings of $7.5 million in direct costs for the last three years. This amounts to approximately 7 percent of the full cost of translation.
20.140 At the same time, as noted previously, there have been certain negative repercussions that may somewhat offset the savings and that have certainly been caused, in part, by contracting out:
- the loss of expertise;
- changes in responsibilities; and
- deterioration in the work climate.
- differences in direct costs between in-house and contracted out work;
- the potential for savings in indirect costs;
- the cost of managing contracts and of services provided free of charge to contractors; and
- the capacity of the private sector.
20.144 Despite its substantial use of contracting out, the Department of the Secretary of State has not analyzed indirect costs to identify the potential for savings. If such an analysis were carried out, additional savings might be found to be possible.
20.145 The Department devotes significant resources to the management of contracting out and to services provided free of charge to contractors. If the resources devoted to these two activities were reduced, the cost of contracting out would fall and the savings would increase.
20.146 The potential impact on the translation market of increased contracting out must also be considered. If the private sector were not able to handle the workload, prices for work contracted out could increase rapidly. This would effectively reduce the savings resulting from contracting out.
20.147 The difficulties in forecasting future trends in these four variables highlight the importance of closely managing contracting out and of constantly monitoring in-house and contract costs.
Certification and the Inventory of Translation Contractors20.148 In view of the large amount of translation contracted out, we reviewed the contractor certification process, the operation of the Certification Committee and the evaluation of the capacity in the inventory of contractors expressed in number of words.
20.149 Certification. The standards governing the process for certifying translation contractors must be sufficiently high to ensure that translated texts will meet contract requirements.
20.150 Generally, Translation Bureau policy requires that contractors who wish to be listed in the Department's Services Contracts Directorate inventory to obtain translation contracts meet two requirements:
- they must possess, as a minimum, sufficient writing experience in both official languages and knowledge in an area of specialization recognized by the Translation Bureau, and this knowledge must be confirmed by a diploma, a certificate or a similar degree; and
- they must pass a certification examination.
The certification process lacks rigour20.151 The certification examination is carried out without supervision. This examination, which is delivered to the candidate by mail, consists of a 600-word text to be translated within three weeks. To pass, the candidate must obtain level B of the SICAL standard.
20.152 Our audit indicates that in 1990 the rate of success for the certification examination was 13 percent. But for an identical level of qualification, level B of the SICAL standard, the rate of success for the Translation Bureau's recruiting examination - which is taken under supervision - was 1 percent. Two important factors explain this variance: on the one hand, the difference in the amount of candidates' experience and their supervision during the examination and, on the other, the difference in the amount of time allotted for the examinations.
20.153 In our opinion, the current process can allow the certification of contractors who do not have the required skills. Given that, as we note in paragraph 20.162, the Certification Committee is very lenient in imposing sanctions against contractors who do not meet contract requirements, it is even more important that the certification process be sufficiently demanding.
20.154 As early as 1983, the Audit Services Bureau of the Department of Supply and Services recommended that the examination for the assessment of contractor skills be supervised.
20.155 The Department of the Secretary of State should revise its process for certifying translation contractors so that candidates taking the examination are supervised, to ensure a valid assessment of contractor's qualifications.
Department's response: Agreed. In August 1991, the Translation Bureau adopted a policy of professional recognition for translators, interpreters and terminologists. Accordingly, it will review as soon as possible its accreditation policy and the various provisions it contains with respect to accreditation, sanctions and their application.
20.156 Certification and quality control policy. The Translation Bureau must ensure, before authorizing payment, that the quality of texts translated by contractors meets the requirements stipulated in the contracts and that translations are delivered within the stated deadlines.
20.157 Cases where a contractor's translation does not meet the quality standards or the prescribed deadlines are referred to the Secretary of the Certification Committee. When a contractor has accumulated four evaluations that are unsatisfactory in terms of quality, or has delivered a significant number of texts late, the case must be referred to the Certification Committee. The Committee's mandate is to apply the policy on certification and quality control and impose sanctions where required.
20.158 Under the existing policy, when a contractor does not comply with contract provisions, the Department may, in accordance with a decision by the Certification Committee, take any measures deemed appropriate, including:
- a warning;
- a suspension;
- the withdrawal of a specialty;
- decertification; or
- a provision for general damages for texts delivered late.
During 1990, only 11 of the 24 cases where contractors failed at least four times to comply with deadlines were referred to the Certification Committee20.159 Our audit revealed that, during 1990, only 11 of the 24 cases where contractors failed at least four times to comply with deadlines were referred to the Certification Committee. This is due mainly to the absence, in the present policy, of specific directives on when such cases should be referred to the Committee.
The Certification Committee imposes sanctions in an inequitable manner20.160 We found that sanctions imposed on contractors were not always related to the infractions committed. For instance, in 1990 one contractor who had accumulated six quality infractions (representing 9.6 percent of his production) had his specialty withdrawn, while another contractor who had accumulated 14 quality infractions during the same period (representing 10.3 percent of his production) received no sanctions.
20.161 In our opinion, inequities exist because the "certification and quality control policy" does not specify criteria for imposing sanctions.
The Certification Committee is very lenient toward contractors who do not comply with contract requirements20.162 We found that the Certification Committee has been very lenient in imposing sanctions against contractors who did not comply with contract requirements for both quality of translations and deadlines. For example, during 1990:
- of 41 contractor files referred to the Committee for unsatisfactory quality , 10 were not subjected to any sanctions, while 11 received only a warning;
- of 11 contractor files submitted to the Committee because of failure to deliver texts on time , as mentioned in paragraph 20.159, 4 were subjected to only a warning.
- specify its policy and directives on evaluating cases of contractors that must be referred to the Certification Committee;
- develop criteria for imposing sanctions on contractors who do not comply with quality and deadline clauses in their various contracts; and
- ensure that sanctions are imposed uniformly
20.164 Contractors' translation capacity. The Services Contracts Directorate of the Department of the Secretary of State has an inventory of about 600 contractors, 70 percent of whom are individual translators. The rest are translation companies that employ translators or hire subcontractors. When registering in the inventory, contractors must indicate to the Department what their daily translation capacity is, in number of words per day and by area of specialty.
20.165 To efficiently manage its financial and human resources and to determine how much translation it can send out on contract, the Department of the Secretary of State needs a reasonable estimate of the translation capacity of contractors in the inventory, by area of specialty. Detailed information about areas of specialty among contractors where there may be a shortage would help management in planning its use of contracting out and in developing in-house the skills that cannot be found in the private sector.
The Department of the Secretary of State does not have information essential to the efficient management of contracting out20.166 We found that the translation capacity of contractors as computed by the Department of the Secretary of State is theoretical since, in general, contractors cannot translate simultaneously in each area of specialty. Moreover, in their statement of capacity, some contractors include estimates of the capacity of subcontractors they use, and frequently these subcontractors are also registered on the inventory, so their capacity is counted twice. The Department of the Secretary of State is therefore unable to estimate by area of specialty the translation capacity of contractors registered in the inventory.
20.167 The lack of reliable data on the extent to which private sector translators can absorb, by area of specialty, the workload the Translation Bureau wants to contract out is compounded by the lack of documentation on the skills of its own translators, which we outlined in paragraph 20.110.
20.168 Consequently, the Department lacks relevant information, on both the volume of translation to be contracted out and its own human and financial resources needs, with which to plan its use of contracting out.
20.169 The Department of the Secretary of State should, from time to time, carry out evaluations of the translation capacity of translators in its inventory in order to better plan its use of contracting out.
Department's response: The Department of the Secretary of State agrees with the recommendation, but sees some difficulties in implementing it. Nevertheless, with the development of a professional recognition policy for translators in August 1991, the Department of the Secretary of State will now consider developing and implementing a method of evaluating translation capabilities at a given point in time.
Planning and Implementation of the Management Information System20.170 An operational management information system must provide, in a timely fashion, complete, reliable, and relevant data that allow management to evaluate and control translation activities.
20.171 Following studies launched in 1983, the Department of the Secretary of State implemented a new management information system in September 1986, called the "Operational Information System" (OIS). This system replaced the obsolete Translation Data System.
20.172 The OIS was designed to facilitate the management of translation work in terms of workload, work distribution, production capacity, utilization of human, financial, and material resources, productivity, etc. In response to observations in our 1984 Report and in our 1986 follow-up (paragraphs 15.137 to 15.139), the Department stated that the new Operational Information System would permit improvements in analyzing and comparing in-house and contract costs of translation, in monitoring compliance with deadlines, and in providing management with the required data to measure achievement of objectives.
20.173 The system includes 120 microcomputers distributed in client departments and linked to a central computer operated by a private company in Ottawa. An interface between the OIS and other departmental systems was planned.
20.174 To temporarily compensate for deficiencies in the OIS, an information system to supplement it, called the "Computerized Work Plan" (CWP), was implemented in April 1986 by the Translation Operations Branch.
20.175 This system collects data on time utilization, average salaries, and operating expenses for each work unit. The data are supplemented manually with quantitative data from the OIS and used to determine the average cost per word translated.
The Operational Information System does not meet user needs20.176 Because of numerous problems while the system was being implemented, the Department commissioned an impartial review of the system's quality by consultants. Their report, dated 18 April 1988, noted serious weaknesses in the architecture and implementation of the system such as the difficulty of maintaining synchronization between the two data bases of the OIS system. This created delays in capturing data and raised questions about data integrity.
20.177 In November 1988 the Department launched an improvement project to temporarily correct some of these deficiencies. Completed in mid-July 1991, this project has made possible significant changes, particularly with respect to data input, that facilitate management's use of the system.
20.178 In the meantime, in January 1991, a report submitted by the Government Consulting Group on the review of the monthly management report confirmed the lack of synchronization.
20.179 In short, it took five years to correct some major conceptual problems in the OIS, and it will be a few more years before an integrated management system can be implemented. In the meantime, as we mentioned in paragraphs 20.96 to 20.99, management does not have a system that readily provides all the information it needs to manage its operations. For example, because the Computerized Work Plan is not linked to the OIS, operational data, the ratio of in-house production to production contracted out, and cost data are not directly accessible for consolidation and analysis purposes.
20.180 Not including OIS purchasing and improvement costs, which we estimated, in the absence of actual costs, at over $2 million, operating the two systems costs about $2.5 million per year, most of it related to the OIS.
20.181 The Department is currently doing a feasibility study on the integration of Computerized Work Plan and Operational Information System data as well as data from all other departmental systems. This study would be followed by another, on replacing the OIS and CWP with a single system. At the present time, informatics management favours operating OIS on its own equipment. This will require recompiling all software.
20.182 The Department of the Secretary of State should implement, as quickly as possible, a management information system that will address all user needs and management expectations, and should ensure that its efforts in this regard are properly co-ordinated.
Department's response: Agreed. The Department of the Secretary of State has recognized the inherent difficulties in the co-existence of separate systems and has undertaken to integrate them by simplifying them, a process which it plans to complete for 1992-1993. The general migration of systems will be included in the departmental informatics plan.
Information to Parliament20.183 The Office of the Comptroller General requires that Part III of the Estimates include sufficient information to allow Members of Parliament and others to understand and evaluate a program's performance in terms of its planned and actual results and resources.
20.184 Our audit was limited to the section of Part III for 1991-92 related to the Translation activity and its three sub-activities: Translation Operations, Terminology and Linguistics Services, and Administration and Special Projects. We also considered an internal audit report on the Part III for 1990-91.
Significant performance indicators were not disclosed20.185 We found that, in general, information on the Translation activity was relevant and presented on a comparative basis. Data related to the Main Estimates are reliable. The areas dealt with are relatively important. Nevertheless, we found the following deficiencies.
20.186 Performance indicators for the Translation Operations sub-activity are included in Part III. However, we found that the Translation Bureau does not use performance indicators, as required by the Office of the Comptroller General, for Terminology and Linguistic Services and for Administration and Special Projects. The Department told us that, since 1989-90, it has been waiting for the creation of the new Department of Multiculturalism and Citizenship before developing a new operational planning framework to be used in preparing Part III. In our opinion, this need not have prevented the preparation and use of performance indicators for these two sub-activities.
Incomplete information20.187 Descriptions of the three sub-activities mentioned in paragraph 20.186 can be found in Part III under the titles "Translation", "Interpretation", "Linguistic Services", and "Computer-Assisted Translation". It is thus difficult to establish a link between the estimated expenditures and the relevant information. Moreover, some information is incomplete. No mention is made of the extent to which deadlines for the delivery of texts are respected, although this is a significant measure of the translation service. Moreover, mention is made of a downward trend in the cost per word translated, but no indication is given of the reasons for this trend.
20.188 We consider the indicator "cost per word translated" can mislead the reader because the Department does not specify what this cost includes. Paragraphs 20.42 to 20.45 outline our detailed observations on the calculation of the cost per word translated.
20.189 For the last six years, Part IIIs have repeated the statement "Decentralization of translation services has been completed in seven of the nine regions." No explanation is provided as to why decentralization is still not complete in the other two regions.
20.190 The Department's Internal Audit Bureau also noted the lack of performance measurement in its October 1990 report on the preparation of Part III of the Estimates.
20.191 With respect to the translation activity, the Department of the Secretary of State should provide in Part III of the Estimates:
- performance indicators for the main outputs of the activity;
- a brief description of the elements included in the cost per word translated both in-house and contracted out; and
- sufficient information to ensure a sound understanding of the data and justification for the funds requested for each sub-activity in the sector.
Program Evaluation20.192 Program evaluation is an important control mechanism, allowing managers to account for the achievement of program objectives and, using the results of the evaluations, to make programs more effective.
20.193 In March 1989 the Program Evaluation Directorate submitted an evaluation report on Official Languages Services. The evaluation was aimed, in general, at evaluating the rationale for Official Languages Services, the achievement of its objectives, and their impacts, as well as suggesting alternative approaches to delivering translation and interpretation services.
20.194 The objective of this segment of our audit was to determine whether the Department had established satisfactory procedures to measure the effectiveness of its programs. We considered the results of the program evaluation in carrying out our audit.
20.195 Our examination was limited to an analysis of reports and documents related to Official Languages Services, which was sufficient to make a judgment on the reliability of the information. We also interviewed Program Evaluation Directorate personnel who participated in the study. Our examination did not cover the management and organization of the program evaluation function.
20.196 We concluded that, in general, the evaluation was satisfactory given the previously established framework. It was well designed, carefully conducted and well documented. The evaluation provided valid information on the questions it addressed, allowing us to confirm that the mandate to provide translation services is clear, that the need for this service is obvious, and that, overall, the achievement of its objectives is satisfactory.
20.197 We also noted that an action plan has been developed to follow up on the findings and recommendations highlighted in the evaluation. This plan has been approved by senior management.