The Employment Security Department’s (ESD) Training Benefits Program provides additional unemployment benefits to eligible workers while they train for high-demand occupations. The state paid $36.6 million in the previous biennium.
Participants can train through programs that teach skills needed for high-demand occupations. Participants have a high completion rate and 72% report working in the industry they trained for.
Participants who started the Program in 2002 and 2003 earned higher wages than the comparison group beginning one to two years after completing training. However, participants in later years did not. It is unclear whether those later participants will catch up over time.
ESD’s administration of the Program complies with statutory requirements. However, there are opportunities to improve customer service and performance. These include: user-testing the application, improving guidance to partners, improving decision timeliness, and establishing a quality assurance review for the Program.
Recommendation #1: The Employment Security Department should prepare a plan to identify reasons why outcomes improve for some training benefits participants and not others, and determine whether there are opportunities to change the Program to improve outcomes for all participants.
Recommendation #2: The Employment Security Department should develop a plan and associated cost estimate to improve its administration of the Training Benefits Program, to include: improving the application form, improving guidance to its Program partners, improving the timeliness of its decisions, establishing quality assurance review, and improving its performance measures.
The Employment Security Department indicated they concur with the Legislative Auditor's recommendations.
The Training Benefits Program, operated by the Employment Security Department (ESD), provides additional unemployment benefits to eligible workers while they train for new, high-demand occupations. The Program does not pay for training costs such as tuition, fees, and books. More information about the Program is in Appendix 1.
A first step in assessing whether unemployed workers benefit from the Program is to determine whether they complete their training and change occupations. The majority do. Sixty-five percent of participants who entered the Program between 2008 and 2012 reported completing their selected training (Exhibit 1).
Participants may train only through programs that teach skills needed for occupations identified as high demand in the participants’ geographic locations. Participants must receive training from community or technical colleges or other vendors on the state Eligible Training Provider list. These vocational training programs generally lead to a certificate or an associate degree. Typically, bachelor degree programs are not eligible.
Seventy-two percent of the participants who completed training and were working at the time of the survey indicated that they were working in the occupations for which they trained.
A legislative goal for the Program is to enhance workers’ earning power. The Employment Security Department (ESD) compared estimated outcomes for Program participants to outcomes for a comparison group of non-participants who had similar characteristics and were also unemployed. Characteristics included prior earnings, occupation, and age (see Methodology tab). JLARC staff reviewed the analysis and learned that outcomes varied based on when participants entered the Program.
In 2002 and 2003, 4,015 participants entered the Program, compared to 17,018 participants entering between 2004 and 2012.
Program participants experienced a drop in employment and earnings during the first three years, as compared to non-participants. During this period, many participants were in training instead of working. In contrast, non-participants may have entered the job market and earned income.
After the first three years, employment outcomes varied based on when participants entered the Program (Exhibit 2).
|Training Benefits Program Participants||Post-training Employment Compared to Non-Participants|
|Entered the Program in 2002 or 2003||Eight percentage points more likely to be employed than the comparison group of non-participants.|
|Entered the Program between 2004 and 2009||No statistically significant difference from comparison group. It is uncertain whether these participants will achieve different rates of employment than non-participants in the coming years.|
|Entered the Program between 2010 and 2012||Data not available.|
Exhibit 3 shows the average difference in earnings, compared to non-participants, for years after the three-year initial training period. Again, outcomes varied based on when participants entered the Program. The 2002 and 2003 participants had gains in wages over the comparison group beginning one to two years after the training period. Program participants who began in later years had lower earnings than the comparison group.
The 2002 and 2003 participants had some characteristics that were different from participants in later years. It is unclear whether these different characteristics explain the different outcomes for the first two years.
The Employment Security Department should prepare a plan to identify reasons why outcomes improve for some training benefits participants and not others, and determine whether there are opportunities to change the Program to improve outcomes for all participants.
Training benefits, like regular unemployment insurance claims, are paid from a federally-managed trust fund. The amount individuals receive in unemployment insurance payments is affected by how often they become unemployed and their earnings.
As shown in Exhibit 4, the trust fund paid more unemployment insurance to Program participants than to non-participants. These results are based on unemployment insurance payments through 2013.
This use of the trust fund may impact Washington employers because they pay taxes that support the fund. However, individual employers do not pay an increased tax rate if former employees participate in the Program.
JLARC staff evaluated ESD’s administration of the Program by reviewing data and documents, and conducting site visits and interviews with Program staff, partners, and participants. The Training Benefits Program is run by a specialized unit within the Employment Security Department (ESD). Since FY 2005, the unit’s size and spending has expanded and contracted:
The evaluation found that Training Benefits Program staff were thorough in their review of training benefits applications and attentive to eligibility factors set in statute. Factors include whether:
One indicator that eligibility decisions comply with the law is the percent of the unit’s decisions that are upheld following an appeal by participants or employers. Appeals are heard by administrative law judges with the Office of Administrative Hearings.
The Training Benefits unit’s decisions were upheld more often than decisions on regular unemployment claims. In 2014, 511 decisions (16 percent) were appealed. Of these, judges upheld 78 percent. In the same year, 65 percent of regular unemployment insurance claims in Washington were upheld on appeal.
While the Program complied with legal requirements, JLARC staff’s site visits, surveys, and discussion with partners revealed opportunities for improvements. ESD can improve its application, guidance to partners, decision timeliness, quality assurance, and performance measurement system.
JLARC surveyed Program participants and interviewed WorkSource employees who assist them. Fifty-four percent of participants said they found the questions on the application confusing. Employees at all 11 WorkSource centers that JLARC staff visited reported that applicants had difficulty with the application.
ESD has two main partners in helping applicants for the Training Benefits Program: the state’s 58 WorkSource employment centers, and the state’s system of community and technical colleges. ESD previously provided guidance and support for these partners but no longer does (Exhibit 5).
|ESD Partner||Partner’s Role||Previous ESD Guidance||Current ESD Guidance|
|WorkSource centers||Tell recently unemployed workers about the Program and help them with the application process||Before 2012, ESD provided training to WorkSource center staff about the Program, as well as updates on changes to Program policies and procedures||ESD no longer provides training and updates|
|Community & technical colleges||Tell students about the Program and provide training to participants||Before 2005, ESD funded staff positions at the colleges to assist Program applicants||ESD has eliminated funding for these positions|
WorkSource center staff interviewed for this study indicated they are not informed when ESD changes policies or procedures for the Training Benefits Program. The amount and quality of assistance WorkSource staff provide to Program applicants varies across the state.
ESD has not met the federal guideline for benefit eligibility decisions (21 days) since June 2013. Performance before that date was mixed (Exhibit 6). If applicants do not receive an eligibility decision for an extended period, the uncertainty can impact their ability to make decisions on educational plans.
ESD uses two quality assurance programs to review its regular unemployment insurance decision, but it does not use these programs to review its training benefits decisions. ESD could make use of this existing structure for quality assurance review of Program decisions.
The unit tracks application volume and decision timeliness to determine staffing levels. Unit staff have proposed but not implemented additional measures including: training completion rates, pre- and post-training occupation and wage information, and application questions where applicants frequently made errors. Program completion and occupation information is collected by ESD and included in annual reports to the Legislature, but it is not actively used to manage Program performance.
The Employment Security Department should develop a plan and associated cost estimate to improve its administration of the Training Benefits Program, to include: improving the application form, improving guidance to its Program partners, improving the timeliness of its decisions, establishing quality assurance review, and improving its performance measures.
The 2011 legislation that modified the Training Benefits Program also called for additional reporting by Employment Security Department and additional study by JLARC staff.
The information presented in this report on the statistical impact of the Program on participant employment and earnings ended with 2013 data.
Program participants who began in later years had lower rates of employment earnings than the comparison group. Adding more years of data may provide insight into whether this pattern continues. More resources would be required for ESD and JLARC staff to collect the necessary information and update the impact analysis in five years.
The Training Benefits Program provides additional unemployment benefits to eligible workers while they participate in training for a new occupation. The Program is intended to help workers move from occupations that are in low demand to occupations in high demand. A legislative goal for the Program is for the training to enhance workers’ earning power. The state spent $38.5 million in total on benefits and administration of the Program in 2013-2015.
Unemployed workers who may be eligible for the Program include workers who:
Each year, $20 million of the federally-managed unemployment insurance trust fund is set aside to pay training benefits in Washington. Any unspent money rolls over to the next year. All Washington employers pay the same “social” tax rate to support the Training Benefits Program. Individual employers do not pay an increased tax rate if former employees participate in the Program.
As shown in Exhibit 7, the Program has spent less than $20 million in all but three fiscal years. An unspent balance of $134 million has accumulated.
The Program is run by a seven-person unit within the Employment Security Department (ESD). Program staff review applications, determine eligibility, and monitor participants’ progress in training. Expenditures to administer the program were $839,000 in FY 2015.
In 2011, the Legislature directed JLARC to evaluate the efficiency and effectiveness of the Program. The study mandate was part of a bill that made significant changes to the unemployment insurance system, including training benefits (EHB 1091).
The methodology for this study included a gross impact analysis, a net impact analysis, and a review of Training Benefits Program administration. The net impact analysis section included an independent review of Employment Security Department’s (ESD) use of “propensity score matching” to create a comparison group of individuals who did not participate in training benefits but who are most alike to the participants.
The gross impact measures are a summary of data from ESD’s 2012-2014 Training Benefits Program annual reports. The aggregate data documents participant demographics, training completion rates, and pre-to-post-training occupational change in order to provide an overview of the Training Benefits Program outputs in those years.
JLARC staff collected employment data from ESD’s data systems and surveys of Program participants. ESD also obtained information on training programs from the State Board of Community and Technical Colleges (SBCTC). JLARC staff reviewed relevant data systems and controls and determined that quality control procedures are sufficient to allow JLARC staff to rely on ESD and SBCTC data for the gross impact analysis.
The gross impact analysis conducted by JLARC staff aggregated data on Program completion and occupational changes reported in ESD’s annual reports from 2012-2014. Earlier annual reports did not contain the same data elements that would allow comparison across all years.
JLARC staff coordinated with ESD to develop the net impact analysis report that analyzed the Program’s effect on participants’ employment rates, earnings, use of UI benefits, and the cost-benefit for participants and taxpayers.
JLARC staff contracted with Kevin Hollenbeck, an economist at the Upjohn Institute for Employment Research, to audit ESD’s methodology. Dr. Hollenbeck’s independent audit confirmed that ESD’s methodology was sound. Dr. Hollenbeck and JLARC staff reviewed the net impact analysis results through multiple drafts, and ESD incorporated suggestions to improve the validity of the method.
ESD used propensity score matching and regression to determine the net impact of the Training Benefits Program on participants’ employment and earnings. Propensity score matching is a statistical technique used to create a comparison group of individuals who did not participate in training benefits but who are most alike to the participants. Each participant is matched to an individual who became unemployed in the same year as the participant. ESD matched them based on previous earnings, occupation, previous periods of unemployment, workforce development area, age, and education. The propensity matching method provided a quasi-experimental approach to isolate the difference in outcomes for those that received Program benefits versus those that did not.
Participants were divided into groups (called cohorts) based on the first year they received a training benefits payment as well as gender. The analysis covers 11 years of participants, from 2002 to 2012. Since there are 11 years of participants divided into males and females, there are 22 total cohorts. The first cohort, those who entered the Program in 2002, has 11 years of data on employment, earnings, and unemployment insurance payments. The second cohort has 10 years of follow-on data. The final cohort, 2012, has only one year of follow-on data. Regression models for each follow-on year and cohort were averaged to create net impact estimates.
After the conclusion of JLARC staff’s field work, ESD published a second report focused on the effect of college coursework for participants. That more recent study employs a different methodology. Neither JLARC staff nor Dr. Hollenbeck reviewed it prior to publication and cannot confirm the validity of the findings or methods.
JLARC staff used three methods to evaluate ESD’s administration of the Training Benefits Program:
ESD should explore, in consultation with the State Board of Community and Technical Colleges and other relevant organizations, possible causes why the Program has not had more positive impacts on recent participants’ employment and earnings. ESD suggests poor labor market conditions during the recent recession may partly explain the lower earnings of some participants who entered the Program in later years. However, without additional years of data, it is unclear whether the influence of the recession explains the variation in performance between early and later years of participants. The Department should consider other variables such as whether differences between participants’ and non-participants’ occupations, industries, and employers may affect employment and earnings.
The Department should identify whether this can be accomplished with existing resources.
If the Legislature also wants updated information on the Program’s impact for future participants, this will require approximately $160,000 to replicate the impact study in 2020.
Plan due by November 1, 2016
|Agency Response:||Concur - see Agency Response tab|
The Employment Security Department should develop a plan and associated cost estimate to improve its administration of the Training Benefits Program, to include:
The Department should identify milestones and resources needed to implement the plan.
Plan due by November 1, 2016
|Agency Response:||Concur - see Agency Response tab|
The State Board for Community and Technical Colleges (SBCTC) and the Office of Financial Management (OFM) were given an opportunity to comment on this report. Both agencies responded that they did not have comments.
Ryan McCord, Research Analyst, 360-786-5186
Amanda Eadrick, Research Analyst, 360-786-5174
Zane Kelly, Research Analyst, 360-786-5193
Valerie Whitener, Audit Coordinator
Keenan Konopaski, Legislative Auditor
Eastside Plaza Building #4, 2nd Floor
1300 Quince Street SE
PO Box 40910
Olympia, WA 98504-0910
The Joint Legislative Audit and Review Committee (JLARC) works to make state government operations more efficient and effective. The Committee is comprised of an equal number of House members and Senators, Democrats and Republicans.
JLARC's non-partisan staff auditors, under the direction of the Legislative Auditor, conduct performance audits, program evaluations, sunset reviews, and other analyses assigned by the Legislature and the Committee.
The statutory authority for JLARC, established in Chapter 44.28 RCW, requires the Legislative Auditor to ensure that JLARC studies are conducted in accordance with Generally Accepted Government Auditing Standards, as applicable to the scope of the audit. This study was conducted in accordance with those applicable standards. Those standards require auditors to plan and perform audits to obtain sufficient, appropriate evidence to provide a reasonable basis for findings and conclusions based on the audit objectives. The evidence obtained for this JLARC report provides a reasonable basis for the enclosed findings and conclusions, and any exceptions to the application of audit standards have been explicitly disclosed in the body of this report.
On April 20, 2016 this report was approved for distribution by the Joint Legislative Audit and Review Committee.
Action to distribute this report does not imply the Committee agrees or disagrees with Legislative Auditor recommendations.
John Braun, Chair
Mark Mullet, Assistant Secretary
Ed Orcutt, Secretary
Derek Stanford, Vice Chair
With passage of EHB 1091 (2011), the Legislature directed JLARC to evaluate the effectiveness and efficiency of the Employment Security Department’s (ESD) Training Benefits Program.
The Training Benefits Program provides extended unemployment payments for certain unemployed workers who need training for a new career. The training must be for a high-demand occupation. During training, the requirement that participants look for work is waived. The Legislature created the program in 2000.
The program does not pay directly for the costs of training. If needed, participants must find other sources to pay for training, such as federal training programs, Pell Grants, loans, or personal resources.
The Training Benefits Program is administered by the Employment Security Department and is funded through employer payroll taxes. Expenditures totaled $23 million in Fiscal Year 2014, including $1 million to administer the program.
Originally, the Legislature targeted the program to dislocated workers, who were workers terminated from an occupation in decline. In 2009, the Legislature expanded eligibility for training benefits to also include unemployed workers who are low-income, disabled due to injury or illness, or served in the military or National Guard.
A 2011 law removed certain deadlines for applying for the program and broadened the definition of dislocated worker.
As directed by statute, JLARC will assess whether the Training Benefits Program is complying with legislative intent, whether the program is effective in re-employing people in high-demand occupations, and whether the program is operating in an efficient manner.
JLARC’s study has three components.
The study will address the following five questions:
Staff will present its preliminary and final reports at the JLARC meetings in December 2015 and January 2016.