How do I check ATS vendor references properly?

Ask for a customer who resembles you on three dimensions: hiring volume, team structure and complexity. A hundred-person company hiring steadily has almost nothing in common with an enterprise running campus recruitment across five countries, and a reference from the wrong profile tells you little. Ask specifically for someone who implemented within the last eighteen months, since the product, the onboarding process and the support team all change. It is also worth asking for a customer who switched from the system you currently use, because they can speak directly to the migration you are contemplating. If a vendor cannot produce a reference matching your profile, that is worth noting, though it may reflect confidentiality policy rather than absence, so ask which it is.

Specific ones about events, not opinions about quality. Ask how long implementation actually took compared with the original estimate. Ask what surprised them in the first ninety days. Ask how their hiring managers reacted and how long adoption took. Ask for the last support ticket they raised, what it concerned, and how long the resolution took. Ask what they had to build, buy or work around that they assumed would be included. Ask what they would do differently. That last question does more work than any other, because satisfied customers still answer it candidly, and the answer usually reveals the real cost of the decision. Avoid asking whether they would recommend the product, since the social pressure on a reference call makes the answer almost meaningless.

Look for customers rather than testimonials. Careers pages often reveal which system a company uses from the application flow or the URL structure, which gives you a list of real users in your industry. Professional networks and recruiting communities are the usual route to a conversation, and most practitioners will answer a short, specific question from a peer. Review sites are useful mainly for spotting recurring complaint themes rather than for scores, since the sample is skewed by review campaigns. When you do reach an independent user, keep the ask small: three questions by message is more likely to get answered than a request for a call. Two independent conversations alongside two vendor-supplied ones give you a balanced picture.

Weight them toward operational risk rather than functionality. You can verify features yourself in a trial, so use references for the things you cannot test in four weeks: support quality over time, how the vendor behaves at renewal, whether the roadmap they described materialised, and how disruptive upgrades are. Look for patterns across calls rather than reacting to a single strong opinion. One frustrated customer may have had an unusual implementation; three mentions of slow support responses is a finding. Record the answers against the same scoring sheet you used for demos so the input is comparable, and revisit any requirement you marked mandatory where a reference reported difficulty. Reference checking is where an evaluation catches the problems that appear in year two.