U.S. Commission on Civil Rights

Federal Efforts to Eradicate Employment Discrimination in State and Local Governments: An Assessment of the U.S. Department of Justice's Employment Litigation Section

Chapter 3

Measuring Performance: An Analysis of ELS Management Tools

The accountability of any civil rights enforcement program hinges on the establishment of clearly defined impact measures that are closely tied to the program's mission and priorities. Thus, in order to assess the performance of ELS, the Commission sought to identify specific indicators used to measure productivity and outcomes. The Commission's detailed review of how ELS tracks its own performance, however, revealed inconsistencies across time and reliance on measures that are insufficient to track the relationship between priorities, goals, and impact.

The following discussion will demonstrate that the reporting measures ELS has relied on have evolved over time, from basic input and output indicators to more complex indicators that attempt to track goals and outcomes. The fact that ELS work is not appropriately planned for or measured, combined with its poorly constructed record-keeping system, made it difficult for the Commission to analyze and draw conclusions about its impact. For the purposes of this report, the Commission had only a few constant indicators for analysis of performance in earlier years, and more specific outcome-related indicators for more recent years. Even though ELS has significantly improved its established performance indicators, it still does not have indicators in place that appropriately measure its productivity or impact. This criticism is not intended to detract from the important work ELS does, nor how well it does it, as the report discusses in greater detail below, but rather to emphasize that recognition of the Section's good performance would be bolstered by the establishment of a more precise method of measurement.

PAST MEASURES OF WORKLOAD AND PERFORMANCE

ELS has historically assessed its performance in terms of workload and accomplishments. The measures of performance have been fairly consistent over the past two decades, but until recently have retained their emphasis on workload and the processing of matters and cases.

As early as fiscal year 1980, budget requests show that ELS was tracking cases and matters filed, closed, and pending at year end. In addition, the text of the submission distinguishes the numbers of pattern or practice lawsuits filed against state and local government employers and defensive suits, and mentions decrees obtained and right-to-sue letters issued.[1] From these, workload and performance measures have evolved over the years. In 1982, five ELS workload and performance measures appeared in the budget submission.[2] In fiscal years 1984 through 1989, ELS represented workload and performance in budget requests with a fairly stable set of nine measures, including the numbers of agency referrals, right-to-sue notices (requests received and letters issued), investigative matters received and terminated, cases commenced and terminated, and decrees obtained and enforced.[3]

In the 1990s, the performance measures became more specific. In FY 1992, the number of decrees enforced was replaced with the number of decrees actively monitored, and the number of decrees requiring monitoring was added.[4] In fiscal years 1995 and 1996, the workload and performance measures were clarified to show that they included pattern or practice, individual, or defensive cases; and the number of cases in active litigation was added, bringing the total number of measures to 10.[5] The more specific measures set the stage for a ballooning of workload and performance measures in the FY 1997 budget submission, when Congress began requiring that federal agencies performance be evaluated on the basis of the agencies overall goals.

THE EFFECTS OF GPRA ON PERFORMANCE MANAGEMENT

In 1993, Congress passed the Government Performance and Results Act (GPRA)[6] with the intent of improving the effectiveness and accountability of federal programs. It requires executive agencies to establish a system of goals and prepare performance plans providing information on the expected progress toward achieving those goals within their requested budgets. The plan must express the performance goals in objective and quantifiable measures, and define the level of performance to be achieved.[7] A key intent of this legislation was to force agencies to evaluate performance according to their broader missions rather than their ability to keep up with their day-to-day workload.[8]

The Department of Justice, along with most other federal agencies, became subject to GPRA requirements that agencies develop strategic plans by September 30, 1997, and every three years thereafter. Agencies must also develop an annual performance report and plan. The first plan was to cover FY 1999; the first performance report was due March 31, 2000.[9]

Since the passage of GPRA, the Department of Justice has developed two strategic plans. Both established the goal of protecting the civil rights of all Americans by legal representation, enforcement of federal laws, and defense of U.S. interests.[10] To achieve this goal, the first plan identified six strategies. The key ones are (1) promote compliance with the country's civil rights laws; (2) emphasize pattern or practice cases and pursue effective remedies within existing law; (3) defend challenges to laws and programs that promote opportunity for traditionally excluded individuals; and (4) strengthen relations with other federal agencies that enforce civil rights laws. Performance indicators intended to measure the progress toward meeting the civil rights goal are (1) the impact of pattern or practice cases initiated; (2) the impact of cases brought in conjunction with local U.S. attorneys and/or state governments; (3) use of settlement techniques, where appropriate, as an alternative to litigation; and (4) judgments, convictions, or settlements in the highest priority enforcement areas.[11]

The later strategic plan placed more emphasis on education. It stated a strategic objective to uphold the civil rights of all Americans through enforcement of and education about, federal civil rights laws and protections. [12] The strategies it mentions include targeting discriminatory public and private institutional conduct through pattern or practice litigation, investigating and prosecuting individuals accused of civil rights violations of federal laws, and educating state and local governments regarding federal civil rights laws and requirements.[13]

Companion documents to its strategic plan are DOJ s annual performance reports and performance plans that express more specific and short-term goals.[14] For the annual strategic plan's civil rights goal, the FY 2000 performance report lists enforcing federal civil rights laws and three sub-goals, one of which is prosecuting pattern or practice civil rights violations. ELS is only one of four sections contributing to the successfully litigated pattern or practice cases that address this sub-goal. Furthermore, ELS contribution to this sub-goal is small, never more than three pattern or practice cases of the Department's 44 to 48 per year during FY 1998 to 2000.[15] The report thereby reveals how insignificant ELS is in both DOJ's strategic plan and annual performance reports.

Nevertheless, detailed measures of ELS workload and performance continue to appear in the annual budget submissions. Indeed, the number of performance measures burgeoned for a few years before dropping back to a smaller set. Compared with the 10 performance measures in the FY 1996 budget submission, the post-GPRA FY 1997 submission had 29.[16] Then 37 performance measures were listed in the budget submission in FY 1998,[17] and 36 measures in FY 1999.[18] The FY 2000 budget streamlined the measures, presenting only 25;[19] and the FY 2001 budget has one fewer.[20]

In addition to its effect on the number of measures, GPRA has forced federal agencies to establish performance indicators that assess the relevant outputs, service levels, and outcomes of each program activity.[21] Thus, the budget submissions began classifying types of measures. Workload measures are indicators of the work received and do not reflect activity or effort.[22] Output measures are indicators of activity or effort, i.e., completed work. Outcome measures assess the results of a program activity compared with its intended purpose.[23]

ELS has struggled to develop meaningful measures in light of GPRA's new requirements and the types of indicators used. First, the Section tried to show more detail in its measures of workload. The added detail in workload measures also became reflected in some output measures.[24] Second, ELS developed a number of measures that tracked the processing of cases and matters, nearly all of which were dropped in recent years. Third, ELS added additional outcome measures. Finally, ELS developed a number of measures of the quality or efficiency of the Section's work.

The modifications in ELS performance measures are reflected in the tables that follow. Table 2 shows the list of performance measures in place in the FY 2001 budget submission and the fiscal year in which each measure was first used to measure workload or performance in the budget submissions. It also shows any earlier measures that were similar. Table 3 shows measures used between fiscal years 1997 and 2000 that are no longer used.

ELS performance measures have gone through considerable evolution and have improved since GPRA was passed. However, they need to continue evolving. As will be demonstrated further, the Commission found that ELS current measures fail to appropriately track its productivity. In particular, to meet the goals of DOJ's strategic plan, ELS should be measuring the outcomes of its outreach and technical assistance to employers, employees, and EEOC staff, and measuring its coordination activities with other agencies that refer complaints. But ELS is no different from other federal agencies that have been required to comply with GPRA without sufficient training. The senior staff at ELS should undergo thorough training in performance measurement so that they can more adequately use GPRA and other tools to assess performance and link the Section's performance measures to the Department's goals.

ESTABLISHING APPROPRIATE PERFORMANCE MEASURES AND GOALS

During the course of this review, the Commission found that ELS is lacking in necessary program planning devices, and it became evident that in ELS, strategic planning is not a priority. There is no adequate formal written plan providing a roadmap for enforcement priorities, litigation-related activities, or expected outcomes. The introduction of GPRA was an effort to induce federal agencies to plan, prioritize, and monitor progress toward goals, but this has not been achieved with respect to ELS. The Commission found the Section's GPRA measures lacking as indicators of performance.

ELS officials admitted difficulties when developing GPRA measures and have concluded that it is unfeasible to quantify litigation activity, which represents the main aspect of the Section's enforcement program. The acting assistant attorney general for civil rights stated that setting standards for GPRA is rendered more difficult because the courts and opposing parties exercise a degree of control, and preferred outcomes are not always clear. For example, bringing many cases could be indicative of a failure in obtaining consent decrees and settlements, rather than an aggressive stance against employment discrimination. Moreover, measuring improvements in civil rights is difficult because gauging outcomes, he believes, is nebulous.[25] Succumbing to these perceived limitations, ELS has quantified aspects of its program that officials believe are quantifiable, but not all, including certain measures that some might consider most important. In the final analysis, many of the measures ELS uses are not effective indicators of work impact.

For example, while one intermediate measure tracks the pattern or practice goal, there are no measures at all for outreach and technical assistance, or outcomes of defensive matters. However, considerable resources are devoted to these initiatives. Other shortcomings are the measures used to track the Section's 706 program, several of which refer to processing EEOC referrals, but only charges after receipt, not ELS efforts to provide technical assistance to EEOC staff in the development of charges. Such measures are important because they could lead to streamlining steps in the pre-referral process with EEOC and conceivably reduce the amount of work required to conduct supplemental investigations for the charges that ELS determines worthy of further pursuit. Even more crucial, there is no precise planning process that leads to judgments about program priorities or areas of discrimination to pursue. Furthermore, no measures are in place that account for the numbers of employees or employers who are affected by ELS work.

A similar finding was made by the General Accounting Office (GAO) in its review of the Civil Rights Division's FY 2000 performance plan, which concluded that the plan needs improvement.[26] ELS intermediate and end outcome measures are more representative of the Section's daily workload processing EEOC referrals and developing pattern or practice cases than the goals stated in DOJ's strategic plan. GAO also found that ELS goals generally show little relationship to the Division's goals.

The Commission suggests that ELS sharpen its ability to determine when appropriate outcomes are accomplished, however difficult that assessment might be. So doing is paramount to successful planning. ELS may benefit from dialogue with other sections of the Division and other federal agencies with similar litigation programs to glean ideas for how its work can be effectively measured. The collection of best practices for complying with GPRA could result in the development of realistic goals and better measures of progress toward those goals. The Division should also give careful consideration to contracting with a business performance consulting firm that can help its officials to translate priorities and activities into concrete measures.

ELS does not appear to have made significant changes since receiving the GAO report. Although ELS has added a deputy chief to conduct education and outreach, there are no planning documents that indicate new emphasis on this aspect of enforcement, nor are efforts to improve coordination with EEOC or to develop common goals demonstrated. Surveys and focus groups could be conducted of employers, employees, and outside attorneys to gain perspective on its planning and measurement process. Finally, ELS performance measures do not capture the extent to which the Section's efforts have met the broader goal of reducing the amount of employment discrimination against state and local government employees.

MEETING EXPECTED LEVELS OF PERFORMANCE

In the 1980s and early 1990s, information reported in budget submissions was often measures of workload, rather than work outcomes. This, in part, explains why the performance projections presented in the budget submissions were estimates based on current performance and not established priorities. The advent of GPRA brought the objective of setting goals that drive performance to new levels of efficiency, instead of relying on past performance to predict future outcomes.

A few measures illustrate the expected levels of performance that ELS established and how they compare with actual performance and future goals. Figure 2 shows the projected and actual number of right-to-sue notices issued from 1980 to 2001. The expected performance is below actual in most years. In fact, expected performance was set at 1,000 in FY 1981 and remained set below 2,000 through FY 1994. Projected performances of 4,900 (a two-year projection) and 3,300 (a one-year projection) in FY 1995 raised expectations considerably, and the expected number of notices issued remained over 4,000 until an actual performance of 3,500 notices issued in FY 2000 drove the FY 2001 expectations down to 3,500.[27] The sharp rise in expectations in FY 1995 corresponded with a period when EEOC was changing its charge handling procedures and referring more charges to ELS, and ELS itself was streamlining its processing of right-to-sue notices. Thus, because of these changes, ELS actual performance surpassed the higher goals in fiscal years 1997 and 1998.

Figure 3 shows cases commenced for 1980 through 1996. Thereafter, the measure was split into components. From 1996 to 2001, the figure shows the expected and actual performance with regard to pattern or practice suits commenced (or filed) and with suits based upon EEOC referrals commenced (or filed). Here the expected and actual performance levels show a more erratic pattern. From FY 1981 to 1991, the expected number of cases commenced was sometimes higher and other times lower than actual performance. However, from FY 1992 through FY 2001, actual performance was below the expected levels whether measured in the earlier years by cases commenced, or in the later years by pattern or practice cases or suits based on EEOC referrals commenced.

Notably, 10 to 15 suits based on EEOC referrals is the most expected to be filed each year. Four to six pattern or practice cases were expected to be commenced each year, although between two and four were actually filed in any given year.[28] Thus, planned performance levels were not being met.

No particular trend emerges in Figure 4 showing the expected and actual performance with regard to consent decrees (or consent decrees and out-of-court settlements during fiscal years 1996 to 2001). Generally, more consent decrees were actually obtained, and expectations were the highest, in FY 1992 to 1994, when 30 or more were expected each year. Since FY 1996, the goals for obtaining consent decrees and out-of-court settlements have been much lower, ranging between 10 and 16. When 31 consent decrees and out-of-court settlements were obtained in FY 1999 and 40 in FY 2000, the number far exceeded the expected amount. However, rather than reflecting outstanding performance, the numbers may simply indicate that ELS was switching to a more inclusive performance measure. The 2002 budget submission includes decree modifications and extensions.[29] In short, ELS has often exceeded its expected performance, although the Commission is unable to determine whether performance has been set at an appropriate level.

FIGURE 2: ELS Expected and Actual Performance, Fiscal Years 1980 2001 

Note: The FY 1998 budget submission reports the FY 1996 target at 3,300, in contrast to the 3,800 and 4,200 shown in the table as the 2- and 1-year projections. The FY 1999 submission shows the current plan for the FY 1998 target as 1,175; but the FY 2000 budget submission shows the final target for FY 1998 as 5,000. Finally, the FY 2001 shows the final target for FY 1999 to be 4,200, not the 4,500 given in the FY 2000 plan. 

Source: U.S. Department of Justice, Civil Rights Division, FY 1980 2002 Congressional Budget Submissions.

FIGURE 3: ELS Expected and Actual Performance, Fiscal Years 1980 2001

Note: The FYs 1997, 1998, and 1999 budget submissions show some discrepancies in performance goals. The FY 1996 budget submission did not include an FY 1996 target (the 2-year projection) for suits based upon EEOC referrals commenced; the FY 1997 reported this target (the 1-year projection) as 10; the FY 1999 submission reported it as 15. Also the FYs 1998 and 1999 submissions reported the initial FY 1998 targets for EEOC referrals and pattern or practice cases commenced as 10 and 6, respectively. The FY 2000 submission shows the same number as the final plan for FY 1998. But, according to the FY 1999 submission, the then current plan for EEOC referrals and pattern or practice suits commenced was 1 and 0, respectively. Finally, the 2001 submission shows the FY 1999 goal for pattern or practice cases as 3, not 4, as is indicated in the FYs 1999 and 2000 submissions.

Source: U.S. Department of Justice, Civil Rights Division, FY 1980 2002 Congressional Budget Submissions. Actual performance was verified with ELS.

FIGURE 4: ELS Expected and Actual Performance, Fiscal Years 1981 2001

* The FYs 1999 and 2000 actual performance and the FY 2001 1-year projected performance include decree modifications and extensions.

Note: The FY 1997 budget submission shows the FY 1996 target (the 1-year projection) as 14; however, the FY 1998 submission reports the target as 10. No 2-year projection was given in the FY 1996 submission.

Source: U.S. Department of Justice, Civil Rights Division, FY 1980 2002 Congressional Budget Submissions.