Case Study

Can Artificial
Intelligence Compete
With Talent Acquisition
Experts?
By: Randy Bailey, Talent Insights Consultant

*Opinions expressed are solely my own and do not express the views or opinions of my employer.
Case Study: Can Artificial Intelligence Compete With Talent Acquisition Experts? 2

Foreward
Talent acquisition specialist and 2017 SourceCon Grandmaster Randy
Bailey is among the best in the business with many years under his
belt to prove it. For a machine to even compete with the likes of him is
incredible on its own. While our HiringSolved software was not offi-
cially a competitor in this years Sourcecon Grandmaster Challenge,
our artificial intelligence managed to unofficially outscore every other
human and machine except one: a man named Randy Bailey.
Baileys inventiveness and creativity in solving the challenge ultimately
beat out the raw power and superior speed of our machine. While
matching patterns found in documents with a job description is some-
thing our AI can do in seconds, Bailey leveraged strengths on the hu-
man side by improvising with techniques that a machine cant perform
quite yet.

We at HiringSolved commend Randy on his win as the 2017 Source-

Con Grandmaster. It was an honor for us that he took the time to
document our softwares performance beside his own in the challenge.
We learned a lot about where our AI stands compared to an elite talent
acquisition specialist and look forward to the next round!

-Shon Burton, Founder and CEO of HiringSolved

Case Study: Can Artificial Intelligence Compete With Talent Acquisition Experts? 3

Executive Summary
This study was created to determine whether artificial intelligence
can perform at the same level of a human recruiter counterpart when
given the same starting data. The testing was framed within the context
of a recruiting contest which had clearly-defined parameters and
predetermined winning criteria. The sample population was a group
10 of 10 sourcing professionals and one AI. No tools were off limits, so if a
human contestant had access to their own AI (like I did at the end) they
SOURCING PROFESSIONALS
were free to use it. From a 5620-person database, contestants were
asked to determine the top 3 candidates for 3 given job requisites.

Job Descriptions: Scoring:

Ground Services Agent Hired (10 Points)
Product Manager Interviewed (5 Points)
Systems Administrator Sourced (3 Points)
5620 Chosen, but Misclassified (1
CANDIDATE DATABASE point)
Resources:
5620 Word (.docx) resumes

The primary motivation for this study was to determine whether AI

software could perform as reliably as a tenured sourcer when given the
same resources. The method for the study was to compare the ability
of an AIs algorithm with the ability of a sourcers search techniques
(boolean, etc). Documenting my methodology was crucial in the
evaluation of the competing recruiting AI tools. This was instrumental
in determining how new technology is performing

Congratulations to All
I also want to say congratulations to all the SourceCon Grand Master
challengers. Simply being a finalist is a huge accomplishment. Like
every year, this one brought forth some exceptional sourcers whom I
hope will become both colleagues and friends.
Case Study: Can Artificial Intelligence Compete With Talent Acquisition Experts? 4

Study Background
The 11 contestants in the competition were provided with 3
anonymized but real job descriptions. We were told that we were
supposed to find and figure out who actually got the job, who
was interviewed, and who was sourced with no other rules or
requirements.

The SourceCon Grand Master challenge is different every year. This

years novel iteration? Pitting man against machine.When I first saw that
we were up against an AI, my initial gut reaction was, Duh. No S#!% a
machine is going to be better at this than a human.

Reading Performance
For reference, humans are really slow at reading compared to a
computer. The average resume is more than 1000 words in length and
the average person reads at a speed of 200 words per minute with
60% comprehension. The fastest competitive readers in the world can
read nearly 1000 words per minute (about 16 per second) with 85%
comprehension, but that performance is only found in the top 1% of
all human readers. Machines can ingest huge information inputs much
faster than people.

Humans vs. 140 Million Words

Computers: per Second

Comparing Max
Reading Speeds
n HUMANS n COMPUTERS

16 Words
per Second
Case Study: Can Artificial Intelligence Compete With Talent Acquisition Experts? 5

Human Ingenuity
After seeing our AI competitor, I nearly resigned humanity to defeat
when suddenly, I was struck with a spark of defiance. I can defeat the
machines, I thought. I just needed to leverage the advantages of the
human brain in areas where machines fall short. After all, searching
through large databases is nothing new in my line of work where
professional networks often consist of millions of candidates.

Humans have highly evolved brains which excel at information

abstraction and flexible pattern matching. Machines have always
excelled in raw operational processing speed and have recently
become more flexible, but we humans still have a few tricks up our
sleeves.

Live Data in Competition

Utilizing tools like Rapportive I was able to verify that the data
provided was real and live. These were actual, real-life applicants. As
I discovered, once I started to dig into the data, the winning choices
were altered to make finding the real person much more difficult.

I verified two of the job descriptions by searching segments of them

on google to find matches. I found the actual companies for the
System Admin and the Ground Services Agent. Rain For Rent in
Bakersfield and Horizon Air in Seattle. The only one I couldnt find in
the wild initially was the Product Manager job description since it was
obscenely generic. Eventually, I was able to find it, but it took some
work.

Background on Sourcing
A sourcer typically chooses candidates from a database by performing
multiple searches and cross-referencing their results after optimizing
for specific terms. In a large database, this process can take many hours
and results are always beholden to a percentage of human error that
makes it easy for even seasoned professionals to overlook a prime
candidate.

In the wider scope of recruiting, database technical know-how is often

a separate and distinct discipline from other traditional methods of
Case Study: Can Artificial Intelligence Compete With Talent Acquisition Experts? 6

filling job roles. In recent years, this more technical group and their
specialized job duties have been given the title of sourcer as a
distinction from the traditional recruiter who relies on more low-tech
methods to match candidates.

Experiment Targets
3 JOB REQUISITES
3 TOP CANDIDATES

Background on AI
Industrial use of AI has developed a lot recently as more industries
are using it to solve problems that used to require humans. In some
extreme cases, AI has become a direct competitor to their human
counterpart, like how self-driving vehicles challenge truck drivers or
how Amazon brick-and-mortar stores challenge cashiers. In other
industries like recruiting, AI plays a more complementary role.
Recruiters are looking at AI and how it can collaborate with them to
improve their performance, rather than replace them.

Methodology (for Humans)

My methodology in this contest relied on human creativity, savvy,
imagination, exploits, advice from friends, and some friendly
technology.

Reverse Engineering the Contest

One advantage of having consciousness is that, unlike a machine,
you understand the metatextual context. Youre aware that youre in
a competition, that some other human built it, and that humans have
flaws.

The contest runners went out of their way to cover their tracks by
altering the documents provided, but I found the real information for
Case Study: Can Artificial Intelligence Compete With Talent Acquisition Experts? 7

both the obfuscated job descriptions and the actual candidates with
clues from the documents provided.

One major observation, which happened early on in the process,

was that I could see the altered document metadata. Documents that
the contest designers had tampered with were easily identified by
their irregular metadata compared to the rest. They stuck out like a
sore thumb. If a document had not been edited, I moved on, as I was
confident every answer would have been edited.

By cross-checking various data points from the altered documents, I

was able to find at least of their social media footprint - I found the real
person for every one of my answers.

Unlike a machine, I was able to actually reach out to the real candidates
and verify my findings directly from the source. Machines are
formidably powerful, but they lack the relentless human spirit and past
experience I had from previous Grand Master challenges that drove
me to leave no stone unturned.

Friends and Friendly AI

I have built
an incredible, Unlike machines, humans can make friends. By attending a couple
SourceCons I have built an incredible, irreplaceable community of
irreplaceable friends. Great people like talent analyst Andrew Gadomski introduced
community of me to the highly-resourceful Janet Dwyer from DataScava whose
veteran talent acquisition advice was invaluable. SourceCon is also
friends
where I met HiringSolved CEO Shon Burton, who helped amp up my
career, and sourcing thought-leader Jeremy Roberts, who now works
as HiringSolveds VP of Customer Experience.

My friendly relationship with HiringSolveds chief executives led to

a conversation about one of their latest AI technologies that was
coincidentally built to handle the exact problem I was working on.

HiringSolveds AI was built to match candidates to a job description

and performed well. By the time I got to use it, I had chosen most of
my prospects already, but it served me as an additional way to verify my
picks and ensure I hadnt missed something.
Case Study: Can Artificial Intelligence Compete With Talent Acquisition Experts? 8

How HiringSolved AI Helped

To verify my results, I loaded the 5620 resumes into a HiringSolved
instance. What took me 25 hours, the AI did in 90 seconds. It was fast,
but not as accurate as my final results. Human adaptability wins there.
Still, the SourceCon Challenge was a great demonstration of how
modern artificial neural network technology can be used to greatly
speed up processes that require a ton of manual labor.

For me, HiringSolveds AI helped confirm my own findings and gave

me the confidence to submit my choices for consideration in the
competition -- ultimately leading to my win. By the time I got to use it,
I had mostly solved or found my top prospects, so I used it as a tool to
mostly verify my answers and to make sure there were no resumes that
slipped through the cracks.

Tackling Search Challenges

Before the final hour when HiringSolved set me up with a search
instance that easily sifted through the data, I had to figure out how to
search a dump of 5620 Word documents all on my own.

Windows Native Search

Surprisingly, the crude method of using Windows native search and
searching everything within a single folder was super effective. I would
search for a variety of keywords or phrases, so that I wouldnt have too
many results, then copy the results into a new folder that I could then
do a second search. This was basically a simple way to do Boolean
(AND) statement when I didnt know or trust the search capabilities
of windows. For example: when I did a search of Bakersfield, only 10
resumes came up! Boom! If I found a few contenders, Id add those to
the folder for each job to view/stack rank later. I quickly found all the
prospects, which didnt change much over time.

Data Obfuscation
When I found out the contest designers had tried to make social
media searches useless by changing key identifying candidate details
to throw off a match, I immediately focused all my energy on getting
around it.
Case Study: Can Artificial Intelligence Compete With Talent Acquisition Experts? 9

System Administrator
Using the System Administrator job description, I found a company called Rain for Rent. I applied this
company name in a LinkedIn query and found they had hired singular System Administrator within the
last month. I had undoubtedly found the name of the person who got the job for a 10-point score! Or, so
I thought

As it turns out, that candidates name wasnt among the competition resumes. One of the 10 Bakersfield
resumes looked suspiciously close. Upon closer inspection I saw that they had merged two jobs from
the candidates employment history from 2005-2017 into one. I now realized what they were doing. As I
suspected, they had altered the winning candidates data.

Data: Real vs. Falsified

Real Data (LinkedIn) Falsified Data (Word Docs)
2005-Present
Systems Administrator, CITGO Oil & Gas
*Two jobs merged into one

2003-2005
Computer Specialist, GeekTech Unlimited
*Changed Boyle Engineering to Geek-
Teck Unlimited
2002-2003 Telecommunications Analyst, Super
Computer Services Inc.

*Changed Affiliated Computer Systems to

Super Computer Services, Inc.
Case Study: Can Artificial Intelligence Compete With Talent Acquisition Experts? 10

Metadata
By looking at metadata on some of my choice contest resumes, I could
see that some of them had been edited. I was able to confirm my
metadata theory when my pick for System Admin was one of the 25
out of 5620 documents that was edited according to the document
details.

Hint Found:
Total Editing Time: 4 Minutes

Product Manager
Unlike the System Administrator role, I did not find the job description
until after I found a suitable candidate. Instead, I used a technique
Janet Dwyer suggested and highlighted job description keywords to
try and match those with resumes.

I did a variety of searches with these keywords that were more telling
than the obvious product manager search. This led to a profile that
used many of the terms exactly like social/in-app/email from the job
description. The resume showed someone who went to the London
School of Economics & the University of San Francisco who had an
Advanced Social Media Certificate. I used this data to find a LinkedIn
profile in the wild that matched and confirmed the competition resume
was the right choice through this real-life matching profile.
Case Study: Can Artificial Intelligence Compete With Talent Acquisition Experts? 11

Product Manager Data: Real vs. Falsified

Real Data (LinkedIn) Falsified Data (Word Docs)
COO & Head of Product, Genome, San Francisco:
2013-Present
*Simplified the title, but dates are the same

Project Manager, Bank of Money London,

England: 2008-2012
*Clearly a bogus company, dates are the
same

Founder & CEO, GrabBag London, England:

2007-2010
*Company names are similar, dates are the
same

Consultant, Delight Consulting, London: 2004-

2007
*Went from Deloitte Consulting to
Delight Consulting -- same dates

Consultant, Luther Pendragon, London: 2003-

2004
*They got lazy and didnt bother changing
the last company name!

London School of Economics

BSC Business, Mathematics & Statistics

University of San Francisco

Advanced Social Media Certificate
*There were no dates on the resume and I
have a feeling the formatting got messed
up, but with all the above similarities, it ihad
to be the same person.
From that I learned that the real candidate worked at Cisco systems starting in November 2015. A telling
date...
Case Study: Can Artificial Intelligence Compete With Talent Acquisition Experts? 12

The Way Back Machine

You can remove whatever you want from the internet, but a beautiful
little web app called The Way Back Machine has been archiving almost
everything published on the web for years.

Sure enough, my earlier search for the Product Manager job

description online was in vain. Cisco had taken down the job
description more than a year prior. However, thanks to the Way Back
Machine and my newfound clue about the date of the job posting, I
found a matching job description for a Growth Product Manager
from October 2015. Another winning choice verified.

Ground Services Agent

This one was the most difficult because this type of candidate is very
unlikely to update or even have a professional profile on the web. Can-
didates from this professional category are notorious for having very
limited documentation of their career history. The job description was
very short and basic list of requirements like customer service experi-
ence preferred and ability to lift 50 pounds so it wasnt hard to find
matching descriptions. Its the candidates that were the issue.

For this search I wasnt able to find anyones professional information,

no matter how or where I searched, and I searched A LOT! I think half
of my 25 hours was spent trying to figure this out. As a result I was un-
able to definitively verify who got the job for this role. I tried to base all
of my resume answers on some sort of tangible facts vs a gut feeling or
instinct - but how could I do that if I didnt know who they were?

I tried searching every which way based on where they attended high
school. I looked at everyone that graduated from the same high school
in the same year. The problem was, I didnt have enough information
to match up the real person with the anonymized resume.

Way past the point of frustration, while venting to a colleague about

the challenge, a new idea popped into my mind! The answer was
staring me in the face...
Case Study: Can Artificial Intelligence Compete With Talent Acquisition Experts? 13

Metadata Again
In the document metadata of my top choice for Ground Services
Agent, there was an original author name. Bingo. His name was Phil.

Armed only with a first name, a high school, and a graduation year, I
did a hail-mary search on Facebook. By some incredible stroke of luck,
I found that only one Phil had graduated from that high school in that
particular year. Some may say thats a weak link, but it was the best
weak link I had.

After looking into his Facebook connections I found he had many

friends working as Ground Service Agents. Including (funnily enough)
his mother. All this, as well as him being my initial gut choice, led me
to choose him as the hired candidate.

Verification, Old School

One thing machines have yet to fully master is good old human rap-
port. Reaching out to people in a human fashion to see if they can
help is a nuanced ability that only comes with experience and instinct.

After having a good idea of who I thought was chosen, I actually used
their real-life email addresses to reach out and verify whether they were
hired or interviewed.

I sent 6 emails and got 2 responses confirming a runner-up System Ad-

ministrator was interviewed at Rain for Rent and a runner-up product
manager was interviewed at Cisco circa-2015. Two more categories in
the bag, as well as further verification that Cisco was in fact the compa-
ny for the Product Manager job description.
Case Study: Can Artificial Intelligence Compete With Talent Acquisition Experts? 14

Results
The experiment was a great success for us humans. Not only did I, a human (I promise), win the contest,
but I was able to confidently present my work after checking it against the matches provided by comple-
mentary AI, social web footprints, and old-fashioned human connection.

HiringSolved AIs performance was impressive on its own since its choices were present in all 3 jobs,
but ultimately it took the human touch to win. HiringSolveds AI picked a lot of the same top candidates
as me with some variation but it mostly complimented my abilities. Together, we outperformed 9 other
human contestants and an AI competitor by correctly identifying the candidates who were actually hired.

This case study showed how AI can help a professional sourcer process data and get quality results at a
much faster rate than standard methods allow. However, it is clear that human adaptability still trumps AI.

Contest Placements
1st Place: Randy Bailey (Me!) 40 points
2nd Place: Competitor 1 18 points
3rd Place: Competitor 2 18 points
4th Place: Brilent AI 14 points

Not competing: HiringSolved AI 19 points

Systems Administrator:
My Submissions - 15 points HiringSolved AI - 2 points
#1 Hired Candidate B (10 points) Candidate A (1 point)
#2 Interviewed Candidate A (5 points) Candidate B (1 point)
#3 Sourced Candidate C (0 points) Candidate C (0 points)
HiringSolved #1: 3203_6893 - Matched my #2/confirmed by the individual that they interviewed
HiringSolved #2: 5903_12683 - my #1, person who got the job confirmed by Linkedin
HiringSolved #3: 16503_35793 - My #3, so the same.
Case Study: Can Artificial Intelligence Compete With Talent Acquisition Experts? 15

Product Manager:

My Submissions - 15 points HiringSolved AI - 15 points

#1 Hired Candidate A (10 points) Candidate A (10 points)
#2 Interviewed Candidate B (5 points) Candidate B (5 points)
#3 Sourced Candidate X (0 points) Candidate C (0 points)
HiringSolved #1: 42788-99198 - Same as my #1
HiringSolved #2: 38533-89942 - Same as my #2
HiringSolved #3: 27452_59360 - Not on my radar/ (or any the 2nd place folks)

Ground Services Agent:

My Submissions - 10 points HiringSolved AI - 2 points

#1 Hired Candidate B (10 points) Candidate A (1 point)
#2 Interviewed Candidate X (0 points) Candidate B (1 point)
#3 Sourced Candidate Y (0 points) Candidate C (0 points)

HiringSolved #1: 1286-2926 - Not on my Radar (or 2nd place folks)

HiringSolved #2: 6181-14299 - My #1
HiringSolved #3: 119652_312970 - Not on my radar (or 2nd place folks)

Machine-Human Complement
Combining HiringSolveds AI technology with a talent acquisition expert was mostly harmonious. The
contest point totals really paint a picture of how AI combined with sourcing expertise is doubly better
than either an AI or a sourcer on their own.

Assumptions
We can assume HiringSolveds AI should perform on resume databases of any size. We can also assume
it would perform similarly with given requirements in any field beyond the three in-contest. The role se-
lection in this case study was determined by the parameters of the contest but the automated matching
should be capable of similar-quality results in any field.

This study addressed the important phenomena that in sourcing, even if the perfect fit is in your data-
Case Study: Can Artificial Intelligence Compete With Talent Acquisition Experts? 16

base, that doesnt mean you will find it. Often, great talent gets buried because of the way typical ap-
plicant systems handle items in a query. It was exciting to see how newer software solutions are trying to
handle this.

The Future
Recruiting AI technology could have a massive impact on the ability of recruiters to find quality can-
didates from large databases much faster. When it gets really good, it could also initiate some major
changes in professional recruiting culture.

Perhaps this AI could level the playing field for recruiters and make it easy for a technical amateur to
achieve the same results that a seasoned sourcer could. It could lead to a more homogenized culture,
where the distinction between sourcer and recruiter becomes less relevant. But, ultimately that time has
not yet arrived.

For now, this technology serves primarily as a compliment to talent acquisition experts. Humans are still
the best equipped to make informed decisions based on AI output. The results are getting more impres-
sive, but an amateur recruiter who chooses candidates based only on what an AI tells them will fail to
grasp great opportunities that a seasoned specialist can detect.

About the Author: Randy Bailey is a recruiting expert at the height of his career. His
skills as a sourcer are unmatched in the industry. Bailey started as a recruiting
manager at Robert Half in June 2005 and grew his ability and skill at several other firms
in the years following. In recent years he has worked directly as a talent sourcing spe-
cialist for major worldwide brands such as PepsiCo, Microsoft, and Walmart where he
currently resides. Bailey has been a finalist in the Sourcecon Grandmaster Challenge
for 4 years in a row before winning the title this year -- defeating all other competitors,
both human & machine.