How to Hire an AI Engineer Without Getting Burned
Hire for the judgment that keeps showing up
The demo works. The résumé is impressive. Everybody leaves the interview excited.
Then somebody asks what happens if the system issues the same refund twice.
Silence is an expensive answer.
AI hiring is hard because the visible part of the work finishes early. A chat window that answers in fluent paragraphs looks done. Nothing about it tells you whether the system respects permissions, survives a timeout, or costs more per ticket than the human it was supposed to help.
You do not need to win an argument about attention heads. You need enough evidence to decide who gets to own those decisions on your behalf.
Hire for the judgment behind the demo. Make that judgment observable before you make the offer.
Here is the process I would run for an engineer shipping AI into a product: write the outcome, open one piece of real work, pay for a short working session, and score what you actually saw. Research and infrastructure roles need different exercises. Start with the job.
Write the job before you buy the résumé
“We need an AI engineer” is about as useful as “we need someone good with money.” Accountant? CFO? The person who tells the founder to stop buying domains?
Pick the problem you are hiring someone to own.
| The work you need | Evidence to look for |
|---|---|
| Research or model development | Experiments, baselines, data choices, and an honest account of what did not work |
| AI application engineering | A useful workflow, integrations, evaluation, and failure handling |
| AI infrastructure | Deployment, capacity, monitoring, cost control, and recovery under load |
| Evaluation and quality | Representative test cases, defensible scoring, and diagnosis of regressions |
| AI product engineering | User research, workflow design, adoption, and proof the feature improved the job |
One person can cover several rows. Expecting equal depth in all five is how a job description turns into a wish list with a salary attached.
Write the first-90-days outcome before you open interviews. For example:
Establish whether a support-drafting assistant reduces handling time without increasing policy errors. Deliver a measured pilot, a human review path, and a recommendation to expand, revise, or stop.
That gives a candidate something to push back on, which is the point. A strong one will ask how handling time is measured, who owns the policy, and whether anybody has checked how good the current human answers are. A weak one will say it sounds exciting.
If nobody on your team can judge the technical evidence, bring in an outside practitioner for the assessment — and ask whether they hope to sell you the implementation afterward. Otherwise the candidate ends up serving as their own technical reference, which is a conflict of interest with better posture.
Ask them to open the hood
A famous employer tells you where someone worked. A demo tells you something worked once, on a laptop, in a good mood. Neither tells you what this person can own on your team.
Ask for one project they can talk about all the way down:
“Walk me through something you personally shipped. What did you own, what broke, and what changed because of the evidence?”
Then follow one decision through the whole arc. What was the first approach? What did they measure? Which alternative did they reject, and why? What did a teammate contribute? What would they do differently now?
Ask for an artifact: a sanitized trace from a failed run, an eval report, a design doc, a test, a short code walkthrough. A trace is just the record of what the system did on the way to its answer — every tool call, every retry, every silent swallow. It is the difference between reading the essay and seeing the work.
A candidate who refuses to hand over a former employer’s customer data is passing the test, not failing it.
Take a reconstructed example instead, or use the shared exercise below. “Show me evidence” must never become “bring us somebody else’s secrets.”
For an earlier-career hire, the evidence is smaller and that is fine. Scale the expected scope and supervision to the role. You are testing understanding and ownership, not access to famous logos.
Five questions worth the interview time
These are prompts for investigation, not trivia. If memorizing the answer is enough to pass, the question is doing nothing.
1. “How would you tell whether this agent improved?”
Listen for success defined in the language of the job: tickets resolved correctly, drafts an agent actually sends, escalations that did not need to happen. Then ask which failures an average score would hide, and what they would compare the new version against.
A good answer makes the measurement inspectable. Ask them to sketch three test cases on the spot and say who decides whether each one passed. If a model grades the answers, ask how they check the grader. “It scored 94%” is not a measurement if the same run scores 82% on Tuesday.
Anthropic’s guide to agent evaluations draws the distinction that belongs in your interview: the record of what an agent did is not the same as the outcome. An agent reporting “I issued the refund” is a sentence, not a refund.
2. “The tool timed out after submitting a refund. Now what?”
“Retry” is the wrong reflex. The money may already be gone.
Listen for checking the transaction’s status before acting, an idempotency key so the second attempt lands on the first one, and an escalation path for when the state is genuinely unknown. Ask who sees the failure and how the work resumes afterward. The vocabulary matters less than whether their design can charge a customer twice for one mistake.
3. “What can this system read, change, and spend?”
Ask for the boundary: which records it can read, which actions it can take, where a human has to approve, and what stops a loop from running all night on your credit card.
Then ask where that boundary is enforced. A prompt telling the model to be careful is a firewall made of policy text — the intent is there, the enforcement is not. Have them draw the boundary and propose a test that tries to cross it.
Include data exposure while you are there: what leaves for the model provider, what gets written to logs, and who can read those logs. “We log everything” is a compliance conversation waiting to happen.
4. “Which part would you build without an LLM?”
A capable engineer can take the AI out of part of their own proposal. Eligibility rules, arithmetic, and permission checks have boring implementations that never hallucinate. Interpreting what a frustrated customer meant does not.
Ask what the model buys you in this specific workflow and what evidence would justify the extra failure surface. If every box in the diagram needs an agent, ask for a smaller diagram.
5. “Tell me about an approach you abandoned.”
Listen for the observation that changed their mind. Users wanted search, not chat. The pricier model cut total handling cost. The feature was not worth shipping and they said so.
A candid negative result beats a polished success story, because a success story rarely reveals a decision rule. Ask what they stopped doing, and how long it took them to stop.
Pay for a small working session
Use a bounded, paid exercise on synthetic data. Send the brief and the scoring criteria in advance — you are hiring for judgment, not for the ability to be ambushed. Let people use the tools they would use on the job, AI included, and then ask them to explain and verify whatever came out.
An illustrative 90-minute session for an application engineer:
You inherit a support assistant that drafts replies and proposes refunds. Here are twelve synthetic tickets, a short policy document, and four recorded runs. One answer cites a policy we retired in March. One refund request times out. One ticket asks for another customer’s information. Recommend whether we expand the pilot, and show me one small improvement or test.
Fifteen minutes to clarify the objective, forty-five to dig in, thirty to explain the recommendation. Hand them a prepared environment so the exercise is not secretly a test of npm install. Accommodate access needs, and keep conditions equivalent across candidates.
You are watching which questions they ask, which evidence they open, and which risk they reach for first. Do they notice that twelve tickets cannot establish reliability? Can they ship one narrow fix without claiming the system is now fine? Can they say what should happen next week?
The candidate who adds a failing test for the duplicate refund may have told you more than the one who shipped a beautiful chat interface.
Use the same core questions and the same rating criteria for everyone in the role — that is the basic structure behind the U.S. Office of Personnel Management’s structured interview guidance, and it exists so your panel compares candidates instead of vibes. Keep the exercise close to the real job. The line between a work sample and free consulting is thinner than most hiring managers think, and candidates can see it from across the room.
The hiring scorecard
Copy this into the interview doc. Agree on the required level for every dimension before you meet anyone, because the bar moves once you like somebody. Each interviewer scores independently before the debrief and attaches a concrete observation to every rating.
Use 1 = unsupported or materially flawed, 2 = workable with substantial guidance, 3 = sound within the role’s scope, 4 = sound judgment plus demonstrated verification. Use N/O = not observed when the interview never produced evidence. N/O is a gap to go fill, not a zero to average away.
| Dimension | Evidence that earns a 3 | Score / observed evidence |
|---|---|---|
| Technical judgment | Chooses a proportionate design and explains a rejected alternative | ___ / ___ |
| Product judgment | Defines a user outcome, a baseline, and a reason to stop | ___ / ___ |
| Evaluation | Proposes representative cases and checks outcomes, not just fluent answers | ___ / ___ |
| Production discipline | Handles partial failure, recovery, monitoring, cost, and latency | ___ / ___ |
| Security | Identifies sensitive data and explains enforceable access and spending limits | ___ / ___ |
| Communication | States uncertainty plainly and explains the consequence to a decision-maker | ___ / ___ |
| Ownership | Separates their work from the team’s and follows failures to resolution | ___ / ___ |
This is a decision aid, not a validated predictor of job performance. Calibrate it to your role, and check it against what actually happens after people join — otherwise you are tuning a judge you never scored.
For someone who will own production alone, I want sound evidence in every essential dimension. A strong total should never paper over an unresolved weakness in permissions or recovery; those are the two that bill you later. For a developing engineer, write down the support they will need and the name of the person providing it.
Close the debrief with three sentences: What can this person own? What support will they need? What are we still unsure about? A panel that cannot answer those is about to have a forty-minute conversation about executive presence.
Red flags deserve one more question
Be careful when a candidate cannot separate their contribution from the team’s, treats every past project as an unbroken success, or answers measurement questions with adjectives. “Highly accurate” needs a denominator.
Other smoke: agents appear in the design before the problem is understood; operating cost has no ceiling; failure recovery belongs to some other team; security lives entirely in the prompt.
Probe once with a concrete scenario before you conclude anything. An unfamiliar term is not a missing concept, and plenty of strong engineers learned the ideas under different names. Give credit when someone catches their own error mid-answer. Refusing to update after seeing contradictory evidence is the disqualifying move — needing a quiet moment to think is not.
Already worried about the hire? Audit the work first
A struggling AI project does not prove you hired the wrong engineer. The brief may have been impossible, the data unusable, or leadership may have promised full autonomy in a keynote before anyone measured quality.
Before you commission a rewrite, preserve the code, configuration, eval results, and relevant logs under proper access controls. Then establish which accounts, services, and API keys the company actually controls — this is where teams discover the whole pipeline runs on one person’s personal billing account.
Get an independent read on a few representative workflows. What works? What fails? Which claims reproduce? Restrict the risky actions while the uncertain behavior is under investigation, and sort the work into keep, repair, and replace.
Ask for a short recovery plan with acceptance tests, named owners, and a decision date. “We need a new framework” is a proposal to examine, not a diagnosis.
Give the hire permission to disappoint the roadmap
None of this works if your company punishes the judgment it just spent six weeks selecting for.
The engineer who says “human approval stays on this step” or “the pilot does not justify expansion yet” needs a leader who can hear it in front of other people. Hire for evidence and then bury the inconvenient findings, and you have built an expensive machine for producing the answers you already wanted.
So for the next AI role: write the outcome, use the scorecard, and watch the candidate dig into something imperfect.
You want the person who can show you why the system is ready — and who will tell you, out loud, on the day it isn’t.