Problem and Setting
One of the largest banks in the U.S. wanted a solution to implement online pre-hire selection solutions to evaluate applicants for the teller position as part of their automated hiring process. On an annual basis the bank administrators evaluate over 100,000 individuals.
The client’s decision to replace their current staffing vendor was motivated by a lack of confidence in that vendor’s test and its ability to identify high performers, the duration of the test (about one hour), and the overall cost. Ingelix® SmartSystem was brought in.
The number of “unsatisfactory” employees reduced by 60% saving the client approximately $4 Million. Additional savings include interview time, managing unsatisfactory performers, cost of error corrections, lost opportunities for up-selling, cost of poor customer service, lower unemployment compensation costs, plus, recruiters’ interview burden reduction will be an additional savings in excess of $400,000 per year.
The client also engaged an independent consulting firm to evaluate the teller position and determine the competencies required for the positions. The consulting firm helped to match the appropriate analytics to the competencies identified. The client and consultant wanted a study to establish validity and local norms.
A Concurrent, Criterion-referenced validation study was conducted. This case study outlines the processes used and the results. Although this study relates to bank tellers, the principles are applicable to all industries and job positions.
Purpose of the Study
The study had multiple purposes:
- Identify the most predictive combination of employee metrics and analytics for the selection of bank tellers.
- Determine if the number of employee metrics could be reduced, to reduce the time applicants spent during evaluation.
- To identify the most effective and efficient cut scores.
Analytic Scales Studied
Eight scales incorporated into employee analytics, with brief descriptions of what each measure, are as follows:
- Reliability – Gets the job done with little follow-up or supervision.
- Stress Management – Demonstrates patience and calmness during times of stress with customers and/or coworkers.
- Flexibility – Takes on new tasks as necessary. Can re-prioritize.
- Helping Disposition – Willingly goes out of his/her way to assist coworkers and/or customers.
- Team Player – Is liked by others. Works well with others to achieve the team’s/department’s goal.
- Problem Solving Ability – Resolves problems using logic and reasoning skills.
- Math Skills – Ability to perform basic computations including adding, subtracting, multiplying, dividing, using decimals and percentages.
- Attention Detail – Makes few mistakes that involve visual perception such as verifying or checking information involving numbers and/or names or letters.
Each scale was assigned a weight relating to job importance as determined by the Client’s consultant in conjunction with the Client’s SMEs.
Validation Method
To accomplish the goals set out for the study, a concurrent, criterion-referenced validation strategy was chosen. The concurrent, criterion-referenced validation method requires that the solution be administered to current employees and (concurrently) performance data is collected for those employees.
The Client provided a data extract from its HRIS system that included all employees. Employees for whom there was no performance rating were removed. Employees with performance ratings were then selected to insure a range of performance rankings (high to low), and geographical representation.
Employees were scheduled to take the experimental analytics battery. The employees’ managers were then asked to rate the employee group on eight important competencies using a seven point Like the performance rating format.
The eight areas rated were: reliability, stress management, flexibility, helping disposition, team player, problem solving ability, math skills and attention to detail skills.
The final sample of employees for which both employee analytics and performance ratings were collected consisted of 97 Tellers.
Data Analysis and Results
Various statistical methods were used during the data analysis process. The following is a brief summary with associated results.
Factor Analysis
In order to determine an overall measure of job performance, the performance ratings were factor analyzed. This “overall performance measure” was then used as the performance criteria for this study.
Correlation Analysis (Original Battery)
To determine the extent to which employee analytic scores were related to performance, correlation analyses were performed at various levels.
The first set of correlation analyses were performed to determine the degree to which the original employee analytic battery chosen to match the consultant’s identified competencies was predictive of the overall job performance rating for Tellers. In addition, the correlations for the individual scales were also calculated.
The original employee analytic Battery had a correlation of .32 with overall performance and this correlation was statistically significant (p<.001). The scales differed in terms of predictability, with Math, Flexibility, Stress Management and Attention to Detail having the highest correlations.
Item Reduction
While the overall battery correlation was statistically significant (r=.32, p<.001), the main goal of the study was to create the most predictive and efficient battery possible. This meant creating a employee analytic battery that only includes the most predictive components, thus increasing validity while simultaneously reducing administration time.
After reviewing the individual employee analytic correlations, both the Math Test and Attention to Detail Test were retained.
To evaluate the most predictive of the behavioral analytic items across the eight behavioral scales, the Teller group was split into two groups: a validation sample and a cross-validation sample.
Using the validation sample, an item analysis was conducted by correlating the individual item responses to the overall job performance rating. This analysis resulted in reducing the original 75 behavioral employee analytic items to the most predictive 22 items. The responses to these 22 employee analytic questions were then summed for each incumbent in the validation sample. This summed score was then correlated with the overall job performance rating. The resulting correlation was .35, p<.012. This is a high, statistically significant correlation.
The responses to these same employee analytic questions were then summed for each employee in the cross validation sample. These summed responses were then correlated with the overall job performance ratings. The resulting correlation was a very high .40, p<.005.
These findings suggest that this combination of behavioral employee analytic questions offers a high level of validity for the Teller job group.
Multiple Regression Analysis
The correlation results revealed that the three tests (Math, Attention to Detail, and the 22 Behavioral items) all positively correlated with overall job performance. To determine whether prediction of performance can be improved by combining these three tests (and how should each be weighed), a forced entry multiple regression procedure was used. This statistical method produces a weighted equation of tests showing their combined correlation with overall job performance. The regression analysis (with the scales weighted) indicated a statistically significant multiple correlation of .48. This very high correlation suggests that the test battery is highly predictive of job success.
Cut Score Analysis
In determining how applicants’ test scores should be used in selection, consideration needs to be given to the following: (1) meeting the projected human resource needs, (2) the effects of different cut scores on the operational validity of the employee analytic battery and (3) minimizing the number of selected applicants who would become “unsatisfactory” performers.
The impact of various cut scores were analyzed, along with their operational validities for the overall job performance ratings, the percentage of applicants who would be screened out and who would “pass” the test if the cut score were implemented and the resulting percentage of unsatisfactory and satisfactory performers based on the study sample.
The Cut Score Analysis showed that the number of applicants hired that would not perform at an acceptable level could be reduced from 11% to just 4%, by implementing a statistically significant cut score that would screen out 28% of applicants taking the test. If the cut score were increased, the percentage of unsatisfactory performers would be decreased even further. In this instance the cut score could be increased without negatively impacting the correlation.
Practical Implications
The foregoing can be translated into practical application by the Client in the selection of Tellers by looking at the impact of using employ analytics to eliminate from consideration those that are not likely to perform satisfactorily. The elimination of unsatisfactory performers will eliminate the cost of replacing unsatisfactory performers and the cost associated with this process, the negative impact on customer service, costs associated with incorrect transactions and unemployment compensation and management time spent counseling unsatisfactory performers, etc.
The Results
Using the Client’s annualized year-to-date data, it was determined that the solution (using the 28th percentile cut score) will reduce the number of incumbent employees rated as “unsatisfactory” by sixty percent. This will save the client approximately $4 Million (assuming the cost-per-hire to be $5,000). Additional savings include time saved in interviewing applicants that will likely be poor performers (in this study, the number not interviewed would be 17,000), managing and counseling unsatisfactory performers, the cost of error corrections, lost opportunities for up-selling, the cost of poor customer service, lower unemployment compensation costs, etc.
In addition, the recommended refinements and revisions to the process will enable the Client to substantially reduce its recruiters’ interview burden by further reducing the number of candidates that are interviewed on the telephone by approximately an additional 20,000 applicants. Assuming a cost (based on recruiter time) of just $20.00 to $25.00 per interview there will be additional hard dollar savings in excess of $400,000 to $500 thousand per year.
