Note: Thanks to Dr. Vivian Woo and Dr. Daniel Meltzer for their helpful feedback
AI has been a hot topic in business over the last few years. The automation movement has been changing how we shop, work, and live our day-to-day lives, from self-checkout lanes to new phone apps. AI and automation also affect our hiring practices. In the human capital management world, AI/automation has become the topic du jour of many conferences and a variety of newspaper articles. Some recent Washington Post articles about the use of AI in hiring have caught the attention of many in the HR world, who may not be aware of how automated hiring practices are being implemented in organizations. The first article is about assessing nanny personality and fitness for hire based on social media posts and the second is a post about AI-based video interviewing tools that uses facial scanning technology to make determinations about a candidate’s fitness to be hired. Both articles seem to sound an alarm around AI and automated tools in hiring.
Among HR professionals, there’s growing concern about how to use these tools, and among applicants, there’s concern about whether an automated tool will fairly assess their skillset. Some larger organizations, such as Hilton, have already implemented AI and automated tools into their hiring practices. One of the most common questions I receive is how AI will change the workplace and whether these tools have value. While I am by no means an expert on the AI/automation tools that exist, I do know how to design and develop selection systems. Here are some tips that may be helpful to those interested in implementing or purchasing these types of products.
Preamble
Some important notes about algorithmic hiring processes that should be covered prior to any discussion of AI/automation. First, algorithmic hiring has existed for almost 100 years in the form of multiple regression and predictive hiring models using traditional pre-hire assessments (i.e., interviews, work sample tests, cognitive ability tests, etc.). Data-driven hiring techniques are the cornerstone of HR practice and can improve organizations in terms of financial performance, diversity, and a variety of other business-related outcomes. Overall, the process you would use to decide on an AI/automation tool is the same approach you would use for any other pre-hire assessment. I would recommend using the Uniform Guidelines on Employee Selection procedures as a reference for evaluating any pre-hire assessment.
The second note is one of nomenclature. I have been referring to these tools as AI/automation because many tools are not the type of artificial intelligence, we hear about in pop culture. Instead these tools mostly automate existing HR practices through technology. Automation has its own value to HR processes, and for the sake of brevity, I will not treat AI and automation as different things, I will merely address them as part of the same category. If you have questions beyond what is covered below, reach out to an HR consultant or an employment lawyer.
My recommendation to anyone considering a change in their human resources practice whether it involves AI or not, is to start with the needs assessment. The needs assessment process allows an HR professional to determine the business problem that must be solved by the potential change. Before beginning the process of vetting AI/automation vendors, conduct a needs analysis to determine what the real problem is. Once you have determined what the problem is then you can assess the viability of a potential AI/automation solution. This process is especially important given some of the claims that AI/automation tools make. Once you have a clear idea of what the problem is, you and your organizational stakeholders can make a clearer judgement about the AI/automation tools. Many HR professionals struggle when deciding about solutions because the problem has not been specificed.
Content
The simplest step to begin your evaluation of any potential AI/automation tool is to determine whether the content of the assessment matches the content of the job itself. Review your job description and job analysis or competency model to determine whether the AI/automation tool assesses a relevant knowledge, skill, or ability. In some cases, the tool may replicate on-the-job tasks. By establishing that the AI/automation tool is related to the job, this establishes that the content of the assessment matches the content of the job. This should be established for any pre-hire assessment. If the tool does not replicate on-the-job tasks, then you may ask if it assesses the necessary knowledge, skills, abilities, or other characteristics (KSAOs) of the job.
Predictions
While matching the content of the job to the job itself is a key element of any hiring practice, the proclaimed value of AI/automation by many organizations is in the tools’ ability to predict future job performance. This is referred to as predictive validity and is a correlation between the pre-hire assessment and future job performance. SHRM offers a toolkit on this topic that may be useful to those working in a selection practice. Ultimately, we must ask ourselves does the AI/automation tool make better predictions and allow your organization to hire more effectively.
Many HR professionals may wonder where to get this kind of information. First, any vendor of pre-hire assessments should be able to provide you with a technical report that explains how their tool helps organizations make better decisions and reduces bias in the hiring process. These reports should include information on the validation study that was conducted and the predictive capability of the assessment. This information is usually in the form of a research study showing the sample of employees used and the quality of hire and job performance. Be mindful of the outcome variables that are measured. If the potential vendor does not have that information, I would be wary of the services they provide.
In addition to accepting the predictive validity offered by the vendor, it may be in your best interest to conduct an internal validation study using your own employees. Give the AI/automation tool to your employees and correlate the tool’s scores/recommendations with the performance of your employees. This will allow you to assess the predictive capability of the tool itself. Even after you have implemented the AI/automation tool, it is in your best to interest to evaluate how much improvement you are seeing in decision quality. There are some existing approaches to assessing decision quality that may be valuable such as Taylor-Russell tables. If you have any concerns about assessing decision quality, your vendor maybe able to help or reach out to another data driven HR consultant.
Fit within Selection Battery
Your organization should use more than one pre-hire assessment in its selection battery (i.e., the group of pre-hire assessments given to job applicants). When adding an AI/automation tool to your selection battery, there are some important considerations. First, how does the AI/automation tool fit into your overall hiring strategy? In other words, does it make the hiring process faster, more effective in decision quality, or is it merely another way to add predictive power? Another important consideration is whether the AI/automation tool is an addition or a replacement for an existing part of your selection battery. Compare the predictive capability of your different tools, and determine whether the AI/automation tool adds any incremental validity (also known as increased predictive power). If the AI/automation does not add predictive power or does not fit your hiring strategy, then it may not be useful to add it. There may be other ways to improve your hiring processes.
When considering fit with your selection battery, an important consideration is employer brand. Review how your organization promotes itself as an employer and evaluate whether the AI/automation tool fits into your employer brand. The selection process is a two-way street. The organization is getting to know the applicant, but the applicant is also getting to know the organization. How an organization hire sends a clear message about the employer brand and about how the organization values people. If much of the process is automated, it may convey to an applicant that people aren’t so important to the organization. Human beings generally recognize we allocate time to things we deem important. If the organization can’t take enough time out of its day to speak to the candidate, it may leave job seekers with a negative impression. The Washington Post articles spend quite a bit of time on the concerns of applicants who experience these tools. Applicants should be told how the tools will be used and what the AI/automation tool assesses. One of the nannies who had been interviewed about her experience with the AI social media tool had not realized what the tool was evaluating her on. This can create later resentment. Clear communication can allay fears, but if your industry and potential job applicants are not amenable to the use of an AI/automation tool, then it may impact your recruitment efforts.
Applicant reactions may also determine the likelihood of legal action. New laws governing the use of AI/automation hiring tools are coming (see the recent law in Illinois) If applicants think the AI/automation tool is unfair or if the way the tool will be used in the hiring process is not clearly communicated, this may result in anger and legal action. Remember that as an employer, you are engaging in a psychological contract with job applicants who want a fair hiring system. If your organization communicates your process clearly, applicants know what to expect and will be less surprised by the AI/automation tool.
Technology Implementation
Implementation is also an important element of fitting the tool into your selection battery. Is the AI/automation tool able to fit it into your existing hiring process easily? Or does the tool not work with your applicant tracking software (ATS) or other technology? A technology solution should fit as smoothly as possible in your existing HR technology. The tool should be easy for you to use on the back end and appear seamless to applicants who are taking the AI/automation tool as part of the application process.
Cost
The reality of using any of pre-hire assessment is that they involve costs. Costs can be financial in terms of how much it costs to give the AI/automation tool per applicant. Cost can also be related to how the AI/automation tool fits into your system. There are important questions around this issue to consider such as the learning curve for your HR team in the use of the tool. Does it fit seamlessly with your ATS? Is it easier for your applicants to access? Does the tool truly make your system faster? As noted in a recent post about AI myths on the SHRM site, Hilton has increased the number of offers using an AI/automation tool, and that is a valuable outcome for Hilton. If something goes wrong with the AI tool will there be support for you to fix it. Finally, if there is bias that is found in the way the tool provides decisions, can you detect it before it becomes a problem.
If your organization does not have the budget for an AI/automation tool, there are less expensive ways of improving hiring that your organization may want to consider. Cost should be balanced against return on investment, which can be calculated based on the predictive power of the AI/automation tool. New tools can lead to changes in your HR strategy. An HR professionals must balance the cost of the hiring tools with the strategic needs of the organization.
Summary
The gist of all of this is to say that before adopting any pre-hire assessment (whether AI or otherwise) consider the following issues:
- Does the content of the assessment match the job?
- Does the assessment predict job performance effectively?
- Does the assessment fit your organization’s selection system?
- Is the tool cost effective?
- Does the tool solve the business problem?
Additional Resources:
EEOC Guidelines: https://www.eeoc.gov/policy/docs/factemployment_procedures.html
SHRM Pre-employment testing: https://www.shrm.org/ResourcesAndTools/Pages/Pre-Employment-Testing.aspx
SIOP Artificial Intelligence in Talent Assessment and Selection white paper: https://www.siop.org/Portals/84/docs/White%20Papers/visibility/AI.pdf
Dr. Richard Landers on AI: https://neshnikolic.com/podcast/richard-landers
MLK50 on bias in facial analysis: https://mlk50.com/facial-analysis-ai-is-being-used-in-job-interviews-and-thats-the-problem-d892216b912b
—
Sy Islam has over 10 years of experience in a variety of corporate, academic, and applied settings. He has served in management, consultant and research roles in a variety of organizations. He is currently an Assistant Professor of Industrial Organizational Psychology at Farmingdale State College. In addition to his role as a professor, he is a co-founder and a Principal Consultant with Talent Metrics. In his role at Talent Metrics, he collaborates with organizations through consulting engagements in his areas of expertise (training and development, selection, survey design, performance management, and team building).
