Introduction
Most organizations agree that employee training programs are a valuable part of human capital strategy, yet few have successfully quantified the ROI of that investment. A framework for measuring training impact isn’t a new idea however. The Kirkpatrick-Phillips model of evaluating training was published in the US Training and Development Journal in 1959, and consists of five levels: 1) Reaction and Planned Action, 2) Learning, 3) Job Applications, 4) Business Results, and 5) ROI. So why have organizations struggled with the practical implementation of this model?
While some training benefits are intuitive, the key is focusing on how these benefits ultimately add value for the organization. When HR fails to communicate financial impact, the real language of business, it can unfortunately result in a lack of investment in training and budget cuts. However, before organizations can move to levels four and five of the Kirkpatrick-Phillips model, they must track, gather and integrate the right data. The workforce is complex, but it can be measured. The end-game for truly measuring training effectiveness starts with linking training outcomes, such as improving retention and controlling workforce cost growth, to the bottom-line.
For the National Cancer Institute (NCI), a component of the National Institutes of Health (NIH), it’s critical to develop a dynamic workforce empowered to solve cancer’s greatest research challenges. NCI’s Office of Workforce Planning and Development (OWPD) is responsible for training and developing a high performing workforce that optimizes organizational capacity. Understanding and illustrating the impact of OWPD training programs allows NCI to better execute on their mission and implement continuous improvements.
Background
In partnership with Human Capital Management Institute (HCMI), NCI’s OWPD conducted a detailed analysis to quantify the outcomes and ROI of its supervisory and leadership development programs for both individuals and the organization. We looked specifically at the impact of participation in these programs had on: 1) individual participants, 2) participant workgroups, and 3) the larger organization.
The analysis focused on long-term, supervisory and leadership development courses offered to employees at various career stages, including future, newly appointed, experienced, and senior leaders. Specific training courses included: Senior Executive Enrichment and Development (SEED), Leadership Education and Action Program (LEAP), Knowledge Management (KM) Mentoring Program, Executive Coaching (EC), and The Empowered Supervisor (TES). This article will refer to these programs as the Academy and to its participants as Alumni. The Academy focuses on the non-technical skills needed to be an effective leader and supervisor. Examples of topics covered include conflict resolution, communicating for results, delegating work, leading change, and influence. The curriculum for the Academy is informed by the NIH Leadership and Management competencies, benchmarking other similar programs, and best practices in the field of leadership and management science.
SEED is a yearlong program for supervisory GS 14-15 and higher level employees. Program components include: half- and whole-day workshops facilitated by internal and external experts, individual coaching, and other experiential activities. SEED Alumni are invited to continue learning as a community of practice and to build networks, foster collaboration, and improve communication across the Institute.
LEAP is a yearlong program for non-supervisory GS 13 and 14 level employees. Program components include: half- and whole-day workshops, group coaching and practical application sessions. LEAP Alumni are also encouraged to participate in a post-program community of practice.
KM is a yearlong mentoring program open to any employee in good standing. KM promotes sharing and teaching of critical skills and institutional knowledge, and nurtures professional growth. Program components include: monthly one-hour mentoring sessions, monthly two-hour program workshops, and individual and team presentations.
EC is open to employees at the GS 13 level or higher and consists of twelve, 1-hour individual sessions with a trained, certified Executive Coach. Coaching focuses on improving workplace effectiveness by minimizing unproductive behavior and maximizing productive behavior.
TES is a five-month program for new and experienced supervisors. Program components include eight 4-hour workshops, six 2.5-hour group coaching sessions, and book and article readings and discussions. This highly interactive program is designed to aid supervisors in maximizing their ability to manage and develop employees, manage workflow, influence the organization, and manage themselves.
Analysis Process
To build the analysis model, over six years of Alumni training results were integrated with other workforce and financial data. As part of the data gathering process, the first step was identifying the appropriate information to combine with the Alumni training data. The core project team focused on the following data categories to include in the model: workforce headcount and transactions, terminations, promotions and transfers, employee demographics, supervisory reporting relationships, performance and monetary awards, training costs and replacement costs. In addition, the team gathered training satisfaction surveys from the OWPD training courses, and also an extract from the organization-wide learning management system to evaluate the impact of the broader training offering across NCI.
Once the requirements were identified, the next hurdle was building the data extracts. While existing reports were already in place, the NCI team had to both modify and build new ones to pull the relevant data. Not surprisingly, all the required data were not maintained in a single system or data warehouse. The OWPD team leveraged data sources including NCI’s core HRIS, HR data warehouse, secondary HR systems, historical databases, surveys, and financial reporting.
The data gathering was primarily led and conducted by the OWPD team. Following some initial challenges of data access and data extract validation, the team thoroughly reviewed and anonymized the data before passing it along to HCMI for positioning and analysis. Throughout this process, data gaps were identified, issues corrected, and updates were made as required. As the data model came together, positioning was done to create categories for analysis that included custom workforce groupings, training cohorts, tenure groups, retirement and age categories, manager and department groupings and more.
As the analysis process continued, the team also focused on identifying the primary evaluation measures that would define success for this project. These included retention, monetary awards, performance ratings, and promotions, which were investigated for both training Alumni and the employees they managed. Post-training results for Alumni were trended and compared to similar demographic cohorts across the organization that didn’t receive training, as well as the Alumni population prior to receiving training. Ultimately, after months of data gathering and analysis, the results were reviewed, validated and presented to key stakeholders across NCI.
Analysis Findings
The analysis showed that Academy training has a significant positive impact across all workforce measures evaluated. Academy Alumni have lower turnover, higher performance, more frequent and higher value monetary awards, and higher promotion rates than Non-Alumni. In addition, these trends are not limited only to Alumni, but also extend to employees they manage.
Highlights include:
Alumni are more than twice as likely to be retained as Non-Alumni, and Alumni high performers are almost half as likely to turnover.
Alumni are more successful at developing and retaining talent. The employees they manage are more than twice as likely to be promoted and approximately 35% less likely to turnover.
Alumni are 35% more likely to be high performers than Non-Alumni, and also receive almost 40% more value in monetary awards than Non-Alumni.
Projected ROI of OWPD training is between $3.9 and 5.5 Million annually over the next 5 years.
Figure 1 shows the turnover trend of training Alumni as compared to Non-Alumni between 2009 and 2014. On average, turnover is approximately 50% lower for training Alumni than Non-Alumni. This trend also holds true for high performer turnover, and is consistent across job groups and organization units. With organization-wide retirements projected to increase significantly over the next 5 years, OWPD training programs will be an integral component to mitigate turnover and knowledge loss going forward.
Perhaps more impressive, Alumni leaders are more successful at developing and retaining talent; the employees they manage are more than twice as likely to be promoted, and approximately
35% less likely to turn over compared to those managed by Non-Alumni. Figure 2 shows the managed promotion rate trend between 2009 and 2014, highlighting significantly higher promotion rates for employees managed by Alumni graduates. This means that Alumni graduates are creating value for NCI by participating in training. Furthermore, Alumni leaders do a better job of promoting participation in NCI’s broader training offering, and are also more frequently selected for larger roles across NIH.
Due to significantly higher rates of retention and promotion, the projected ROI of Academy training programs is between $3.9 and $5.5 million annually for NCI through 2019. In addition to cost savings from improved retention, employees that are promoted have lower salary and total compensation costs than their externally hired counterparts. Figure 3 shows the ROI projections over the next 5 years, and includes both direct expense reductions (hard costs) and future efficiency savings (soft costs), less Academy training program costs and unpaid salaries due to vacancies. The largest portion of the savings estimates are anticipated from reductions in replacement hire costs and salary savings from internal promotions, with additional savings projected due to reduced transaction processing, severance, vacation payout and benefits continuation costs.
Looking Ahead
Based on these findings, OWPD is eager to expand NCI’s training offerings, with an emphasis on high potentials and mission- critical job roles. As part of a comprehensive workforce planning strategy, training programs will be leveraged to help mitigate the risk of future knowledge loss for job groups and departments with the highest percentage of employees approaching retirement eligibility. In addition, increased focus on leveraging training to foster opportunities for internal mobility will help build defined career paths across NCI.
Ultimately, Academy training programs will help NCI and NIH build management bench strength for the future, increasing the percentage of internal promotions into management roles over the next five years. Internal promotions also result in lower salary costs, which help control Total Cost of Workforce, and mitigate future budget constraints. Currently, NCI and the OWPD team are evaluating additional analytics project opportunities and implementing HCMI’s training effectiveness dashboard to monitor and optimize future results.
