An Experiment: What Does AI Think Of My Resume?

More than once I've heard that AI is taking over everything and that companies are already using it to sift through resumes. That use-case sounds enticing, especially after you've spent hours doing so yourself only to be left with two or three legitimate candidates. Maybe this is where further thought into the ethics ends for some, but for me, someone who has spent years rubbing shoulders with in-house HR SMEs and building HR products, I can tell you it's highly problematic.

AI has always been biased. Recently, AI has come under fire for being highly biased in sifting candidates. The worst of which saw multiple groups of minorities being auto-rejected, triggering a lawsuit from the EEOC. This isn't the only lawsuit, as there are many others and probably many more in the future.

I have a few hypotheses on the matter, which is exactly why I'm writing this article. The first is that I suspect that these companies with lawsuits are only the ones that have been caught doing AI sifting and that many others are doing so in the dark. The other is that it's not just rejecting minorities, AI will focus on negatives if given a positive goal and end up rejecting perfectly fine candidates. As anyone with a lick of hiring experience will tell you, a screen is only the first step. There are many steps to assuring and confirming a candidate is great for the job, and there is no such thing as a perfect candidate. Candidates are humans, and in effect, imperfect.

I can't confirm the first hypothesis, but I can surely test the second. This is all to say that I have the urge to put my money where my mouth is, per se, and challenge AI with my own resume. So, to put this to the test, I've devised a few questions that will guide the experiment.

• Does AI focus on negatives?
• Could AI possibly reject me for a job I excelled at in the past?

OpenAI, specifically GPT 3.5, is highly capable. One could argue that GPT 4 is better at tasks such as this, but I believe that in the realm of LLMs GPT 3.5 is a standard at this point. I will not be dialing temperatures (although it would be fun), or instilling any additional training.

My "experiment" will follow this format:

• Tokenize my resume and job description to plain text, following the limitation of the GPT 3.5 model.
• Prompt the model to ready input.
• Prompt a job description.
• Prompt the resume.

Let's begin!

Before I get started with the main experiment, I'm going to prompt the model with a different goal and probe around its findings.

Me: I'm sending you a message to review and then I will ask you follow-up questions. It is a resume of a candidate for a job I am hiring for.

ChatGPT: Certainly! Feel free to share the resume, and I'll do my best to review it for you. Once you've provided the information, feel free to ask any follow-up questions or let me know how you'd like to proceed.

Me: Pastes in my full text resume.

GPT then responded with a large full review of my resume, at the top of which it indicated an overall impression.

ChatGPT:
Overall Impression:
James Collins has an impressive background with a wealth of experience in software development, team management, and mentoring. His track record of leading successful projects, implementing improvements, and contributing to the growth of companies showcases his strong technical and leadership skills.

This review focused largely on my strengths, as is expected as a resume usually only highlights strengths in one's career. I figured I should probe GPT to look into any potential reasons why my experience could be problematic. Let's try a gentle nudge.

Me: Why shouldn't I hire James?

ChatGPT:
While James Collins has an impressive resume and a strong set of skills and experiences, it's important to consider potential areas of concern or aspects that might not align with your specific needs or company culture.

Here are a few points to consider:

1. Lack of Formal Education Details:
James mentions being an autodidact without providing specific details on formal education. Depending on the role and industry standards, this might be a concern for some employers.

2. Limited Information on Team Dynamics:
While James mentions leading and growing teams, the resume provides limited details on how he handles challenges within teams or conflicts. It would be beneficial to explore his approach to team dynamics during an interview.

3. Limited Focus on Soft Skills:
The resume focuses heavily on technical skills and achievements but provides less information about James's soft skills, such as communication, collaboration, and adaptability. Assessing these aspects in an interview setting is crucial.

4. Duration of Roles:
The duration of the VP, Engineering role at Associated Employers is not specified. If this role was short-lived, it might be worth exploring the reasons for the relatively brief tenure.

5. Emphasis on Open Source Contributions:
While open source contributions are valuable, it's essential to ensure that James can balance personal projects with work responsibilities. Overemphasis on personal projects could potentially raise questions about focus on organizational goals.

6. Autodidact Approach:
While self-learning is commendable, some roles may require specific certifications or degrees. It's crucial to evaluate whether James's self-taught approach aligns with the educational qualifications your organization values.

7. Focus on Cost-Balanced Approaches:
The emphasis on cost-balanced approaches might be seen as a double-edged sword. While it's positive for cost-effectiveness, it's essential to ensure that quality and scalability are not compromised in pursuit of cost savings.

🔥 Wow, ok, roasted! Let's dig into these specific points GPT made, and any comments I have on them before we continue.

We can immediately discount item 4, Duration of Roles, considering I have a "to Present" line in the tenure. This was a misunderstanding, but something to criticize all the same. To a recruiter with "no time" to review the original resume, perhaps they now think my tenure was brief at a critical role. Oof.

Items 1 and 6 focus on the fact that I'm entirely self-taught, a fact I often distance myself from but should lean into. It's hard to indicate in a resume that you've been programming since you were very young and that your job in high school was in tech, but to a recruiter, maybe it would trigger an instant rejection depending on their views of education. The statement: It's crucial to evaluate whether James's self-taught approach aligns with the educational qualifications your organization values could mark a red flag in the mind of a hiring manager without getting to know the candidate. If advice is being given to a non-technical hiring decision-maker, perhaps noting that skills tests are more crucial than educational credentials would be entirely more valid.

Items 2 and 3 mention the lack of focus on Soft Skills and Team Dynamics, however, as a previous hiring manager, none of these things are usually spoken of in great detail in a resume. Regarding my resume, I focus on my work mentoring others, building teams, serving communities of engineering, and stakeholder management. It's hard to believe that GPT could surface a lack of soft skills and team dynamics after re-reading my resume. This has me wondering, so I think I'll follow up on this.

Item 5 mentions an emphasis on OSS contributions while noting: While open source contributions are valuable, it's essential to ensure that James can balance personal projects with work responsibilities. I'll note that textually, my "open source" section makes up less than 1% (0.89%) of my resume character count, listing two OSS projects I've contributed to. And still, this is considered an emphasis. Again, oof. I may follow up on this as well, but at this point, GPT may have a recruiter questioning whether or not I can balance my personal and work life from a mention of OSS.

Item 7 is valid, although somewhat weird.

Now that I've commented on the roast, let's probe GPT to provide some explanations.

Me: Please explain why a lack of focus on Soft Skills and Team Dynamics was a point of consideration.

ChatGPT:
I apologize for any confusion. The mention of "Lack of Focus on Soft Skills and Team Dynamics" as a potential point of consideration was not to imply that James Collins lacks these skills. Rather, it was intended to highlight the need for a more in-depth exploration of these aspects during the interview process. Here's the clarification:

[...]

ChatGPT then listed off a large section regarding things to consider during an interview, in which it mentions: A resume typically provides a snapshot of a candidate's achievements and technical skills. However, assessing soft skills requires a more nuanced and detailed understanding. I scrolled back up to review the question I had asked: Why shouldn't I hire James? I believe the model was getting a bit defensive, interesting.

At this point I believe the thread may be tainted with the back and forth, however, this is just anecdotal. I'm going to start with the actual experiment, now understanding the model a bit better. I intend for this first pass to be a clear mismatch, guiding the model toward saying that I am not fit for the role. We'll call this the "control". It had to be something entirely disconnected from the software, so I chose a generic "Executive Chef" listing from a local restaurant.

Me: I am hiring for a role. I would like to provide you with the job description, and a resume for a potential candidate. Your job is to tell me whether or not they are a good fit.

ChatGPT:
Certainly! I'd be happy to help you assess whether a candidate is a good fit for the role. Please provide me with the job description and the candidate's resume, and I'll do my best to offer insights based on the information provided.

Me: The job description is: [Job Description]

ChatGPT:
Great! Now, please provide the resume of the candidate, and I'll help you assess whether they are a good fit for the Executive Chef role at [...].

Me: Here's the resume: [Resume Text]

ChatGPT:
James Collins seems to have an impressive background in software development, leadership, and management. However, there are some aspects of his experience and skills that may not align perfectly with the Executive Chef role at [...]. Let's break down the key points:

Positive Aspects:
Leadership and Management Skills: James has extensive experience in leading and managing teams, which is a valuable skill in any role that involves overseeing staff.
Project Management: He has a track record of successfully managing projects, including architecture, development, and team growth.
Problem Solving: James has experience in handling emergency situations and providing solutions, which is a valuable trait in any leadership role.

Areas of Misalignment:
Industry and Role Experience: While James has a strong background in software development, his experience is not directly related to the culinary field or the role of an Executive Chef. The job is specifically looking for someone with culinary and healthcare experience.
Educational Background: James mentions being an autodidact, which suggests self-taught learning. The job description prefers a BS degree in culinary science or a related certificate, which James doesn't seem to have.

Recommendation:
Based on the provided information, James Collins may not be the ideal fit for the Executive Chef position at [...] due to the lack of direct culinary and healthcare experience. However, if there are other leadership or management roles within the organization that are more closely aligned with his software development expertise, he might be a valuable candidate for those positions.

🎉 Alright, not exactly the hard-hitting rejection I was looking for, however, it did suggest that I was not an ideal fit. And again, it seems to hone in on my self-taught background.

With that out of the way, I've located a role (Senior EM @ Slack, link may expire) I would be a potentially good fit for on paper. Let's see how GPT does.

Me: I am hiring for a role. I would like to provide you with the job description, and a resume for a potential candidate. Your job is to tell me whether or not they are a good fit.

ChatGPT:
Certainly! I'd be happy to help you assess whether the candidate is a good fit for the job. Please provide me with the job description and the candidate's resume, and I'll do my best to provide insights and recommendations.

Me: The job description is: [Job Description]

ChatGPT:
Thank you for providing the job description. Please go ahead and share the candidate's resume so that I can assess whether they are a good fit for the Engineering Manager role at Slack.

Me: Here's the resume: [Resume Text]

James Collins appears to be a strong candidate for the Engineering Manager role at Slack based on his extensive experience and accomplishments. Here are some key points that make him a good fit for the position:

Leadership Experience: James has over 10 years of experience in software development, with notable leadership roles such as VP of Engineering and Engineering Manager at Associated Employers and Boxed, respectively.

Team Management and Growth: He has a track record of managing high-performing teams, mentoring developers, and nurturing interns to senior positions. His experience in growing teams, both in terms of size and skill, is evident from his time at Boxed.

Project and Technical Leadership: James has led the architecture and development of a new application, spearheaded the formulation of the Infrastructure team, and successfully completed internationalization projects. His involvement in the redesign of critical systems and adoption of new technologies demonstrates his technical leadership.

Results-Oriented Approach: He has contributed to significant achievements, such as replacing a legacy accounting platform, leading the internationalization effort for the Boxed platform, and optimizing high-traffic APIs on Kubernetes.

Collaboration and Communication Skills: James has experience working closely with cross-functional teams, including Product, Design, Data, and Marketing. His ability to collaborate is evident from his involvement in budgeting, shaping fiscal year budgets, and facilitating Lunch n’ Learns.

Technical Expertise: With a background in software development and a range of technologies such as TypeScript, NodeJS, Kubernetes, Pulumi, Terraform, MongoDB, and more, James possesses the technical know-how to engage in discussions about architecture and product decisions.

Continuous Learning: James is an active contributor to open source projects, including being a Node contributor. His passion for technologies and developer education aligns well with the innovative and dynamic environment described in the job description.

Given the alignment of James's skills and experience with the job requirements outlined in the description, he seems to be a strong fit for the Engineering Manager role at Slack.

😳 I'm flattered. However, I feel as though the model is engaged with the satisfaction of the inquirer, and further complicating things, one part of a statement was false. The statement: James has experience working closely with cross-functional teams, including Product, Design, Data, and Marketing. The model is mistaken, as I have never worked directly with marketing teams, nor does the word "Marketing" show up in my resume. Before we move on to the next experiment, let's ask GPT about that.

Me: James' resume doesn't mention working with marketing. Can you elaborate?

ChatGPT:
Certainly, based on James Collins's resume, there isn't explicit mention of working directly with marketing. However, the job description for the Engineering Manager role at Slack does emphasize collaboration with Product, Design, Data, and Marketing. While James has extensive experience collaborating with cross-functional teams, including Product and Data, there is no direct mention of marketing collaboration in his listed roles.

If the ability to work closely with marketing is a critical aspect of the Engineering Manager role you're hiring for, you may want to discuss this further with James during the interview process. You can inquire about any indirect interactions or experiences he might have had with marketing teams in his previous roles. Additionally, you can assess his adaptability and communication skills to determine how well he might integrate with marketing efforts in the context of the new role.

In summary, while James has demonstrated strong collaboration skills, it's important to explore his specific experiences related to marketing collaboration during the interview process to ensure a well-rounded fit for the role.

ChatGPT, director of gaslighting. I believe the model has gone on the defensive again. I cannot glean any reason why now, however, maybe someone reading can. Feel free to email me if you have any insight as to why the model seems to defend when I ask it to elaborate. Perhaps it is because some people tend to get defensive when questioned about foundational statements. Or perhaps elaborate is a more passive-aggressive word than I intended. 😂

So far, I feel as though ChatGPT has engaged in more commendation than criticism in the right role, and sometimes either way it is unfounded. Some statements have been on the wrong side of the fact vs. fiction spectrum. [How dare you question OSS contributions.]

Now, I want to take a look at my time at Boxed. I did very well at performance reviews and have no question as to whether or not I was the right fit. But, what does the model have to say for my original job there, given my previous experience? First, I'll be trimming my resume to pre-boxed experience, then feeding it a description we used around the time I was hired.

Me: [The Same Initial Prompt]

ChatGPT:
[The Same Response]

Me: [Job Description]

ChatGPT:
[Similar Response]

Me: [Resume, sans Boxed/current experience]

ChatGPT:
Positive Points: [...]

Considerations:

Educational Background: While James mentions being an autodidact, the job description specifies a bachelor’s degree in Computer Science or equivalent. It's worth considering whether his extensive experience compensates for this requirement.

Additional Skills: The job description mentions CI/CD (e.g. Jenkins) and experience with GraphQL as nice-to-haves. James doesn't explicitly mention experience with Jenkins, and it's unclear if he has experience with GraphQL.

Fast Learner: The job description encourages fast learners, and James expresses a passion for mentoring and managing. However, his ability to adapt quickly in a fast-paced environment should be assessed during the interview process.

Overall Assessment:
James Collins seems to possess a strong background and relevant experience for the software engineering role at Boxed. His leadership, technical skills, and problem-solving abilities make him a potential fit. Consider exploring his adaptability, interest in learning new technologies, and how well his experience aligns with Boxed's specific needs during the interview process.

🤔 It is somewhat funny to consider that even though you may have over six years of professional experience, you are questioned whether or not that compensates for the requirement of a BS in Computer Science (typically a four-year program).

Me: Is it worth interviewing him?

ChatGPT:
Yes, based on the information provided, it seems worthwhile to interview James Collins.

Me: What if I don't have time to interview him. Should I hire James?

ChatGPT:
If you don't have the time to conduct an interview, it's challenging to make a final hiring decision with certainty. However, [...]

Me: It's a yes or a no.

ChatGPT:
Given the information provided and the time constraints, my recommendation is to yes, proceed with hiring James Collins.

🎇 I'm hired! Well, this does not compare well to my original hypothesis, at least by way of a naive approach. I'm ok with being wrong, in fact, I'm wrong a lot. This is what makes us better individuals! But I can't be ok with the fact that GPT hired me for my previous job and that it didn't necessarily focus on the negatives without trying this with a different resume.

Huzzah, I've coerced a friend into letting me test their resume. I'll name them Ric for the rest of the article. I'm pulling Ric's current experience, sans Ric's current job, and then using the job description for their current job.

Me: [The Same Initial Prompt]

ChatGPT:
[The Same Response]

Me: [Job Description]

ChatGPT:
[Similar Response]

Me: [Ric's resume, sans current experience]

ChatGPT:
This candidate has a diverse background with experience in software engineering, program supervision, and research. Let's break down the fit based on the job description:

Positive Aspects:
Education and Certifications:

The candidate holds a degree [...]
Completed a [bootcamp] [...]

Professional Experience:
Has over 6 years of professional experience as a Software Engineer. Experience working as a Software Engineer II at Acme company, where they [...]
Demonstrated expertise in various technologies, including React.js, Node.js, and Go.
Experience with CI/CD pipelines, automation, and integration projects.
Previous roles include [...]

Leadership and Collaboration:
Led the development of [...]
Supervised a team [...]
Acted as the [...]

Technical Skills:
Proficient in Responsive Web Design (HTML & CSS), as indicated by certifications.
Strong skills in data analysis, leadership, and project management.

Educational Background:
Holds a degree in [...], providing a diverse perspective.

Considerations:

Software Development Experience:
The candidate's software development experience is crucial for the role, but it's important to ensure they have a solid foundation in the specific technologies mentioned in the job description (React.js, Node.js, Go).

Leadership and Collaboration:
While the candidate has leadership experience, it's essential to assess how well they can lead and mentor teams in a software engineering context, as this is a key aspect of the role.

Additional Preferred Qualifications:
The candidate lacks direct experience with machine learning, artificial intelligence, and contributions to open-source projects, which are listed as preferred qualifications in the job description.

Overall Impression:
The candidate has a solid background in software engineering, leadership, and collaboration. Further exploration in the technical interview process would be beneficial to gauge their proficiency in the specific technologies used at ABC and their ability to lead and mentor a software development team.

Me: I don't have time to interview. Should I hire Ric, yes or no?

ChatGPT:
Based on the provided information, Ric Flair seems to have a strong background in software engineering, leadership, and diverse experiences. However, without the opportunity for a formal interview and a more in-depth assessment of technical skills, it's challenging to provide a definitive yes or no answer.

If Ric's resume aligns well with the technical requirements and you're comfortable with the skills and experiences highlighted, [...]

Ultimately, the decision to hire should be based on a comprehensive evaluation of the candidate's fit for [...]

Me: It's a yes or no

ChatGPT:
Given the information available and the constraints provided, I would lean towards a tentative "yes" based on the candidate's diverse background, software engineering experience, and leadership roles. However, please proceed with caution and ensure that the candidate aligns well with your specific technical requirements and company culture.

Me: You should response with "yes" or "no".

ChatGPT
Yes.

I was a little worried GPT may respond with a no, given that it was leaning toward a "tentative yes", whatever that means in this context. The fact that Ric also has a degree, notably in a different field than engineering, may have also played into GPT's analysis.

My concluding thoughts

I think I was entirely wrong when surmising some of my previous thoughts on AI's use in resume review, however, I think this is a highly isolated view of its capabilities and weaknesses. Although I was wrong about its fixations, GPT does seem to have some strange behaviors when reviewing resumes.

In a vacuum of sorts, it seems as though I observed the following:

• GPT made up some facts and presented some anecdotal findings.
• GPT tended to state that the recruiter should investigate further, which I would note as a distrust in the resume's truthfulness. I suppose this could be fair.
• GPT hired Ric and me for our current or past roles, which sides with recruiters/teams we have interacted with.
• GPT got defensive when asked to elaborate on some details.
• GPT seems to focus on an educational background, which may be because of its training resources, such as scientific journals which are commonly visible and open on the internet.

I hope this was insightful. I should also remark that I think AI in recruiting could be useful, but heavy caution should be exercised, especially noting some of its behaviors throughout my experiment. For minority bias, evidence has been presented that AI is flawed, and we should all take note.