Training Benefits Program participants successfully complete training for new occupations

The Training Benefits Program, operated by the Employment Security Department (ESD), provides additional unemployment benefits to eligible workers while they train for new, high-demand occupations.  The Program does not pay for training costs such as tuition, fees, and books.  More information about the Program is in Appendix 1.

A first step in assessing whether unemployed workers benefit from the Program is to determine whether they complete their training and change occupations.  The majority do.  Sixty-five percent of participants who entered the Program between 2008 and 2012 reported completing their selected training (Exhibit 1).

Training limited to certain degrees and occupations

Participants may train only through programs that teach skills needed for occupations identified as high demand in the participants’ geographic locations.  Participants must receive training from community or technical colleges or other vendors on the state Eligible Training Provider list.  These vocational training programs generally lead to a certificate or an associate degree.  Typically, bachelor degree programs are not eligible.

Program participants change occupations after training

Seventy-two percent of the participants who completed training and were working at the time of the survey indicated that they were working in the occupations for which they trained.

• The top occupations were in the healthcare and computer fields.  High-demand healthcare occupations include medical assistants, respiratory therapists, and medical technicians.  Programmers and web developers are examples of high-demand computer occupations.
• Fewer reported working in administrative support, management, and construction occupations.

To date, only participants who entered the Program in its early years have experienced gains in employment and earnings

A legislative goal for the Program is to enhance workers’ earning power.  The Employment Security Department (ESD) compared estimated outcomes for Program participants to outcomes for a comparison group of non-participants who had similar characteristics and were also unemployed.  Characteristics included prior earnings, occupation, and age (see Methodology tab).  JLARC staff reviewed the analysis and learned that outcomes varied based on when participants entered the Program.

Impacts vary after the training period

In 2002 and 2003, 4,015 participants entered the Program, compared to 17,018 participants entering between 2004 and 2012.

Program participants experienced a drop in employment and earnings during the first three years, as compared to non-participants.  During this period, many participants were in training instead of working.  In contrast, non-participants may have entered the job market and earned income.

After the first three years, employment outcomes varied based on when participants entered the Program (Exhibit 2).

Exhibit 3 shows the average difference in earnings, compared to non-participants, for years after the three-year initial training period.  Again, outcomes varied based on when participants entered the Program.  The 2002 and 2003 participants had gains in wages over the comparison group beginning one to two years after the training period.  Program participants who began in later years had lower earnings than the comparison group.

Characteristics of early participants

The 2002 and 2003 participants had some characteristics that were different from participants in later years.  It is unclear whether these different characteristics explain the different outcomes for the first two years.

• More of the early participants returned to their former employer: Between 39 and 43 percent of the 2002 and 2003 participants returned to their former employer of record, compared to 18 percent of the participants from all other years.  The data does not indicate whether participants returned to the same job or to a different one with the former employer.  Returning to one’s employer of record probably creates an earnings advantage compared to starting with a new employer.
• One industry was prevalent in early years: Fifty-three percent of the 2002 and 2003 participants were previously employed in the durable goods manufacturing industry versus 14 percent of the 2004-2012 participants.  The durable goods manufacturing industry includes aerospace manufacturing.  Since occupations in aerospace tend to pay high wages, it is possible that participants who returned to this industry increased the average of earnings for all 2002 and 2003 participants.

Legislative Auditor Recommendation #1

The Employment Security Department should prepare a plan to identify reasons why outcomes improve for some training benefits participants and not others, and determine whether there are opportunities to change the Program to improve outcomes for all participants.

Additional payments participants received during their training period resulted in a greater use of unemployment insurance trust fund benefits

Training benefits, like regular unemployment insurance claims, are paid from a federally-managed trust fund.  The amount individuals receive in unemployment insurance payments is affected by how often they become unemployed and their earnings.

As shown in Exhibit 4, the trust fund paid more unemployment insurance to Program participants than to non-participants.  These results are based on unemployment insurance payments through 2013.

• While participants were in training, the trust fund paid more in unemployment insurance compared to the group of non-participants.
• After the training period, the trust fund paid slightly less in unemployment insurance over time because of the Program.  However, this future reduction was not enough to offset the initial increase in benefits payments.
• Through 2013, the trust fund paid an average of $10,000 more for each participant compared to non-participants.  ESD projected unemployment insurance payments through the end of the participants’ careers, and expects that the trust fund would still pay more to participants than to non-participants over time.

This use of the trust fund may impact Washington employers because they pay taxes that support the fund.  However, individual employers do not pay an increased tax rate if former employees participate in the Program.

While Employment Security Department’s administration of the Program complies with legal requirements, there are opportunities to improve customer service and performance

JLARC staff evaluated ESD’s administration of the Program by reviewing data and documents, and conducting site visits and interviews with Program staff, partners, and participants.  The Training Benefits Program is run by a specialized unit within the Employment Security Department (ESD).  Since FY 2005, the unit’s size and spending has expanded and contracted:

• Between FY 2005 and FY 2010, the unit’s spending grew from $1.5 million to $2.5 million.  The number of staff in the unit reached a high of 21 in FY 2010.
• Beginning in FY 2010, ESD reduced the size and spending of the unit to seven staff and expenditures of $839,000.  This represents a spending decrease of 57 percent.  The unit experienced a 68 percent drop in caseload during that time.

ESD complies with legal requirements when determining eligibility

The evaluation found that Training Benefits Program staff were thorough in their review of training benefits applications and attentive to eligibility factors set in statute.  Factors include whether:

• The applicant’s occupation is in decline
• The applicant needs training to find suitable employment
• The training will lead to a high-demand occupation
• The training meets Workforce Board performance criteria

One indicator that eligibility decisions comply with the law is the percent of the unit’s decisions that are upheld following an appeal by participants or employers.  Appeals are heard by administrative law judges with the Office of Administrative Hearings.

The Training Benefits unit’s decisions were upheld more often than decisions on regular unemployment claims.  In 2014, 511 decisions (16 percent) were appealed.  Of these, judges upheld 78 percent.  In the same year, 65 percent of regular unemployment insurance claims in Washington were upheld on appeal.

Opportunities exist to improve Program administration

While the Program complied with legal requirements, JLARC staff’s site visits, surveys, and discussion with partners revealed opportunities for improvements.  ESD can improve its application, guidance to partners, decision timeliness, quality assurance, and performance measurement system.

Improve the application form

JLARC surveyed Program participants and interviewed WorkSource employees who assist them.  Fifty-four percent of participants said they found the questions on the application confusing.  Employees at all 11 WorkSource centers that JLARC staff visited reported that applicants had difficulty with the application.

• The training benefits application does not satisfy state executive branch plain talk rules and guidelines, which require agencies to use simple and clear language when communicating with the public.
• The guidelines also recommend user testing for applications.  ESD has not conducted user testing.  The Department reports that planned testing in 2014 was cancelled due to budget reductions.
• The eight-page application requires applicants to provide six or more attachments.  There are seven states with similar programs and all have applications that are at least five pages shorter and required fewer attachments.

Improve guidance for partners who help unemployed workers apply for the Program

ESD has two main partners in helping applicants for the Training Benefits Program: the state’s 58 WorkSource employment centers, and the state’s system of community and technical colleges.  ESD previously provided guidance and support for these partners but no longer does (Exhibit 5).

WorkSource center staff interviewed for this study indicated they are not informed when ESD changes policies or procedures for the Training Benefits Program.  The amount and quality of assistance WorkSource staff provide to Program applicants varies across the state.

Improve the timeliness of the ESD’s decisions on applications

ESD has not met the federal guideline for benefit eligibility decisions (21 days) since June 2013.  Performance before that date was mixed (Exhibit 6).  If applicants do not receive an eligibility decision for an extended period, the uncertainty can impact their ability to make decisions on educational plans.

Establish quality assurance review of its training benefits decisions

ESD uses two quality assurance programs to review its regular unemployment insurance decision, but it does not use these programs to review its training benefits decisions.  ESD could make use of this existing structure for quality assurance review of Program decisions.

• Benefits Timeliness and Quality review: ESD auditors review 100 recent claims decisions each quarter to determine if the decisions are correct and benefits are paid on time.
• Washington Quality Standards review: ESD auditors review a random selection of claims and judge them against a set of standards that define a quality decision.

Improve performance measures for the Training Benefits Program

The unit tracks application volume and decision timeliness to determine staffing levels.  Unit staff have proposed but not implemented additional measures including: training completion rates, pre- and post-training occupation and wage information, and application questions where applicants frequently made errors.  Program completion and occupation information is collected by ESD and included in annual reports to the Legislature, but it is not actively used to manage Program performance.

Legislative Auditor Recommendation #2

The Employment Security Department should develop a plan and associated cost estimate to improve its administration of the Training Benefits Program, to include: improving the application form, improving guidance to its Program partners, improving the timeliness of its decisions, establishing quality assurance review, and improving its performance measures.

Future reporting on earnings and employment

The 2011 legislation that modified the Training Benefits Program also called for additional reporting by Employment Security Department and additional study by JLARC staff.

Additional reporting by ESD and JLARC staff

• ESD will continue providing annual reports to the Legislature about basic Program information such as participant demographics, completion rates, former occupations, and administrative costs.  These annual reports do not include the statistical analysis of the impact of the Program.
• The Legislature directed JLARC staff to review the Training Benefits Program every five years.

Resources needed if the Legislature wants updated information on Program’s impact on participant employment and earnings in future reports

The information presented in this report on the statistical impact of the Program on participant employment and earnings ended with 2013 data.

Program participants who began in later years had lower rates of employment earnings than the comparison group.  Adding more years of data may provide insight into whether this pattern continues.  More resources would be required for ESD and JLARC staff to collect the necessary information and update the impact analysis in five years.

• ESD estimates a cost of $126,000 to update the analysis.
• JLARC staff estimate a cost of $34,000 to hire a labor economist to review ESD’s methodology and results.

The Training Benefits Program provides income support to unemployed workers while they train for new occupations

The Training Benefits Program provides additional unemployment benefits to eligible workers while they participate in training for a new occupation.  The Program is intended to help workers move from occupations that are in low demand to occupations in high demand.  A legislative goal for the Program is for the training to enhance workers’ earning power.  The state spent $38.5 million in total on benefits and administration of the Program in 2013-2015.

Unemployed workers who may be eligible for the Program include workers who:

• Lost their job in a low-demand occupation;
• Are disabled due to injury or illness and who need training to find employment;
• Are low-income and whose earning potential can be enhanced through vocational training; or
• Are military personnel who need training to find suitable employment.

Program benefits:

• Up to 26 weeks of training benefits unemployment insurance payments in addition to 26 weeks of regular unemployment insurance payments
• No requirement to look for work while in training

Program limits:

• Participants must train for high-demand occupations
• Participants receive benefits only while enrolled in training and making satisfactory progress
• There are no payments for the direct costs of training, such as tuition, fees, and books

Training benefits paid from a federally-managed trust fund

Each year, $20 million of the federally-managed unemployment insurance trust fund is set aside to pay training benefits in Washington.  Any unspent money rolls over to the next year.  All Washington employers pay the same “social” tax rate to support the Training Benefits Program.  Individual employers do not pay an increased tax rate if former employees participate in the Program.

As shown in Exhibit 7, the Program has spent less than $20 million in all but three fiscal years.  An unspent balance of $134 million has accumulated.

Program administered by the Employment Security Department

The Program is run by a seven-person unit within the Employment Security Department (ESD).  Program staff review applications, determine eligibility, and monitor participants’ progress in training.  Expenditures to administer the program were $839,000 in FY 2015.

Legislature requested Program review

In 2011, the Legislature directed JLARC to evaluate the efficiency and effectiveness of the Program.  The study mandate was part of a bill that made significant changes to the unemployment insurance system, including training benefits (EHB 1091).

Three key components

The methodology for this study included a gross impact analysis, a net impact analysis, and a review of Training Benefits Program administration.  The net impact analysis section included an independent review of Employment Security Department’s (ESD) use of “propensity score matching” to create a comparison group of individuals who did not participate in training benefits but who are most alike to the participants.

Gross Impact Analysis: Training rates and other characteristics of Program participants

The gross impact measures are a summary of data from ESD’s 2012-2014 Training Benefits Program annual reports.  The aggregate data documents participant demographics, training completion rates, and pre-to-post-training occupational change in order to provide an overview of the Training Benefits Program outputs in those years.

JLARC staff collected employment data from ESD’s data systems and surveys of Program participants.  ESD also obtained information on training programs from the State Board of Community and Technical Colleges (SBCTC).  JLARC staff reviewed relevant data systems and controls and determined that quality control procedures are sufficient to allow JLARC staff to rely on ESD and SBCTC data for the gross impact analysis.

The gross impact analysis conducted by JLARC staff aggregated data on Program completion and occupational changes reported in ESD’s annual reports from 2012-2014.  Earlier annual reports did not contain the same data elements that would allow comparison across all years.

Net Impact Analysis: Participants’ employment and earnings

JLARC staff coordinated with ESD to develop the net impact analysis report that analyzed the Program’s effect on participants’ employment rates, earnings, use of UI benefits, and the cost-benefit for participants and taxpayers.

JLARC staff contracted with Kevin Hollenbeck, an economist at the Upjohn Institute for Employment Research, to audit ESD’s methodology.  Dr. Hollenbeck’s independent audit confirmed that ESD’s methodology was sound.  Dr. Hollenbeck and JLARC staff reviewed the net impact analysis results through multiple drafts, and ESD incorporated suggestions to improve the validity of the method.

ESD used propensity score matching to create the comparison group

ESD used propensity score matching and regression to determine the net impact of the Training Benefits Program on participants’ employment and earnings.  Propensity score matching is a statistical technique used to create a comparison group of individuals who did not participate in training benefits but who are most alike to the participants.  Each participant is matched to an individual who became unemployed in the same year as the participant.  ESD matched them based on previous earnings, occupation, previous periods of unemployment, workforce development area, age, and education.  The propensity matching method provided a quasi-experimental approach to isolate the difference in outcomes for those that received Program benefits versus those that did not.

Participants were divided into groups (called cohorts) based on the first year they received a training benefits payment as well as gender.  The analysis covers 11 years of participants, from 2002 to 2012.  Since there are 11 years of participants divided into males and females, there are 22 total cohorts.  The first cohort, those who entered the Program in 2002, has 11 years of data on employment, earnings, and unemployment insurance payments.  The second cohort has 10 years of follow-on data.  The final cohort, 2012, has only one year of follow-on data.  Regression models for each follow-on year and cohort were averaged to create net impact estimates.

After the conclusion of JLARC staff’s field work, ESD published a second report focused on the effect of college coursework for participants.  That more recent study employs a different methodology.  Neither JLARC staff nor Dr. Hollenbeck reviewed it prior to publication and cannot confirm the validity of the findings or methods.

Program Administration

JLARC staff used three methods to evaluate ESD’s administration of the Training Benefits Program:

• ESD data and documents: JLARC staff reviewed and analyzed Program data as well as administrative materials, and training materials regarding the Program.
• Participant survey and focus groups: JLARC staff contracted with the Social & Economic Sciences Research Center at Washington State University to conduct a survey of recent Training Benefits participants and to conduct three focus groups of current participants.
• Site visits: JLARC staff visited 11 out of 58 WorkSource sites across Washington to learn how WorkSource assists unemployed workers with the Training Benefits Program.

Legislative Auditor makes two recommendations to improve Training Benefits Program outcomes and administration

Recommendation #1:

The Employment Security Department should prepare a plan to identify reasons why outcomes improve for some training benefits participants and not others, and determine whether there are opportunities to change the Program to improve outcomes for all participants.

ESD should explore, in consultation with the State Board of Community and Technical Colleges and other relevant organizations, possible causes why the Program has not had more positive impacts on recent participants’ employment and earnings.  ESD suggests poor labor market conditions during the recent recession may partly explain the lower earnings of some participants who entered the Program in later years.  However, without additional years of data, it is unclear whether the influence of the recession explains the variation in performance between early and later years of participants.  The Department should consider other variables such as whether differences between participants’ and non-participants’ occupations, industries, and employers may affect employment and earnings. 

Recommendation #2:

The Employment Security Department should develop a plan and associated cost estimate to improve its administration of the Training Benefits Program, to include: improving the application form, improving guidance to its Program partners, improving the timeliness of its decisions, establishing quality assurance review, and improving its performance measures.

The Employment Security Department should develop a plan and associated cost estimate to improve its administration of the Training Benefits Program, to include:

• Improving the application form
• Improving guidance to its partners (WorkSource and community college staff) who help unemployed workers apply for the Program
• Improving the timeliness of its decisions on the applications
• Establishing quality assurance review of its decisions
• Improving performance measures for the Program

The State Board for Community and Technical Colleges (SBCTC) and the Office of Financial Management (OFM) were given an opportunity to comment on this report. Both agencies responded that they did not have comments.

Contact

Authors of this Study

Ryan McCord, Research Analyst, 360-786-5186

Amanda Eadrick, Research Analyst, 360-786-5174

Zane Kelly, Research Analyst, 360-786-5193

Valerie Whitener, Audit Coordinator

Keenan Konopaski, Legislative Auditor

Joint Legislative Audit and Review Committee

Eastside Plaza Building #4, 2nd Floor

1300 Quince Street SE

PO Box 40910

Olympia, WA 98504-0910

Phone: 360-786-5171

FAX: 360-786-5180

Email: JLARC@leg.wa.gov

Audit Authority

The Joint Legislative Audit and Review Committee (JLARC) works to make state government operations more efficient and effective. The Committee is comprised of an equal number of House members and Senators, Democrats and Republicans.

JLARC's non-partisan staff auditors, under the direction of the Legislative Auditor, conduct performance audits, program evaluations, sunset reviews, and other analyses assigned by the Legislature and the Committee.

The statutory authority for JLARC, established in Chapter 44.28 RCW, requires the Legislative Auditor to ensure that JLARC studies are conducted in accordance with Generally Accepted Government Auditing Standards, as applicable to the scope of the audit. This study was conducted in accordance with those applicable standards. Those standards require auditors to plan and perform audits to obtain sufficient, appropriate evidence to provide a reasonable basis for findings and conclusions based on the audit objectives. The evidence obtained for this JLARC report provides a reasonable basis for the enclosed findings and conclusions, and any exceptions to the application of audit standards have been explicitly disclosed in the body of this report.

Committee Action to Distribute Report

On April 20, 2016 this report was approved for distribution by the Joint Legislative Audit and Review Committee.

Action to distribute this report does not imply the Committee agrees or disagrees with Legislative Auditor recommendations.

JLARC Members on Publication Date

Scope & Objectives

Why a JLARC Study of the Training Benefits Program?

With passage of EHB 1091 (2011), the Legislature directed JLARC to evaluate the effectiveness and efficiency of the Employment Security Department’s (ESD) Training Benefits Program.

Training Benefits Provides Income Support to Unemployed Workers Who Enroll in Training

The Training Benefits Program provides extended unemployment payments for certain unemployed workers who need training for a new career. The training must be for a high-demand occupation. During training, the requirement that participants look for work is waived. The Legislature created the program in 2000.

The program does not pay directly for the costs of training. If needed, participants must find other sources to pay for training, such as federal training programs, Pell Grants, loans, or personal resources.

The Training Benefits Program is administered by the Employment Security Department and is funded through employer payroll taxes. Expenditures totaled $23 million in Fiscal Year 2014, including $1 million to administer the program.

Legislative Changes in 2009 and 2011 Expanded Access to the Program

Originally, the Legislature targeted the program to dislocated workers, who were workers terminated from an occupation in decline. In 2009, the Legislature expanded eligibility for training benefits to also include unemployed workers who are low-income, disabled due to injury or illness, or served in the military or National Guard.

A 2011 law removed certain deadlines for applying for the program and broadened the definition of dislocated worker.

Study Scope

As directed by statute, JLARC will assess whether the Training Benefits Program is complying with legislative intent, whether the program is effective in re-employing people in high-demand occupations, and whether the program is operating in an efficient manner.

JLARC’s study has three components.

• One component will examine the implementation and operation of the program including how claimants apply for the program and how ESD determines eligibility.
• Another component will focus on effectiveness. ESD is performing analysis of whether the program has helped people get jobs and earn higher wages compared to a group of similar people who did not participate in the program. JLARC has contracted with an independent expert to review ESD’s analysis.
• The third component will review data on program participants, including their prior occupation and the type of training they take.

Study Objectives

The study will address the following five questions:

1. Has the program been implemented according to legislative intent?
2. Are the parties that play a role in the program working together effectively?
3. Are the administrative costs of the program consistent with federal guidelines and comparable to similar programs in other states?
4. Are people getting jobs and earning higher wages compared to people who did not participate in the program?
5. What are characteristics and occupations of training benefit participants, and what types of jobs are they getting after training?

Timeframe for the Study

Staff will present its preliminary and final reports at the JLARC meetings in December 2015 and January 2016.