Should we run a pilot first?

Yes. A short, scoped pilot on one real role tells you more than any reference call — provided the success criteria are agreed up front.

What is a fair pilot success criterion?

Something observable: a shortlist you would genuinely interview, delivered inside an agreed window, with evidence you can read.

Two or three, on the same role, with the same brief. Comparing different roles produces noise, not signal.

Ask for a methodology walkthrough instead — and treat any unverifiable statistic as marketing rather than evidence.

ComparisonsUpdated 10 February 20263 min read

Judge partners on what they can show you, not what they promise. Evidence, ownership and escalation beat any pitch deck.

Written for: Buyers running a recruiting vendor selection

Term	What to pin down
Cost shape	Fixed, percentage, or hybrid — and what triggers extra cost
Capacity	How many roles run concurrently before quality drops
Data ownership	Who keeps candidates and evidence at the end of term
Exit	Notice period, export format, and what you take with you
Attribution	How duplicate candidates across sources are resolved

1
Escalation path
Name the human who fixes a stalled role, and the response window.
2
Calibration cadence
Agree when the bar gets reviewed, not just when candidates arrive.
3
Reporting
Agree the two or three numbers you will both look at weekly.

Red flags

Unverifiable metrics, no candidate-facing transparency, no data export, and a shortlist you cannot interrogate.

Should we run a pilot first?: Yes. A short, scoped pilot on one real role tells you more than any reference call — provided the success criteria are agreed up front.
What is a fair pilot success criterion?: Something observable: a shortlist you would genuinely interview, delivered inside an agreed window, with evidence you can read.
How many vendors should we compare?: Two or three, on the same role, with the same brief. Comparing different roles produces noise, not signal.
What if the partner cannot show real outcomes?: Ask for a methodology walkthrough instead — and treat any unverifiable statistic as marketing rather than evidence.

Where to go next