ChatGPT
Claude
Perplexity
Gemini
GrokFive measures are enough, if each has a written definition
To judge an AI admissions agent, track five things: resolution rate, handoff quality, qualified enquiries, time to first human contact, and what happens to applicants through to enrolment. Everything else helps you diagnose problems but should not drive decisions.
No credible public benchmark tells a UK university or college what a "good" resolution rate is, so this article does not offer one. It offers definitions, formulas and reading traps. The thresholds should come from your own conversations after a few weeks of observation.
| KPI | Formula | Trap to avoid |
|---|---|---|
| Resolution rate | Questions genuinely answered ÷ real questions asked | Counting abandonment as success |
| Handoff quality | Complete, accepted handoffs ÷ handoffs requested | Tracking only how many handoffs happen |
| Qualified enquiries | Valid, consented contacts with a stated course interest ÷ conversations showing real intent | Counting every email address as an applicant |
| Time to human contact | Median and 90th percentile from request to first adviser reply | Reporting the average only |
| Path to enrolment | Applicants by stage, grouped by week of first contact | Crediting the chatbot with everything that follows |
Resolution rate: what was settled, not what went unchallenged
Resolution rate is the share of real questions that received a correct and sufficient answer. Leave greetings, internal tests and empty sessions out of the denominator.
The common trap is "deflection" or "containment", which treats any conversation without a handoff as solved. A prospect who leaves irritated after a vague reply then scores as a win.
The honest method is sampling. Each week, read a random set of conversations and sort them: resolved, partly resolved, unresolved, wrong. Add a direct signal ("Did this help?") while remembering that only the very happy and very unhappy respond.
Track accuracy separately for critical facts: tuition fees, entry requirements, UCAS deadlines, Clearing availability, accommodation terms. A wrong answer about an entry requirement is worse than a handoff. The NIST AI Risk Management Framework treats measurement as a core function of governing any AI system, which is a useful framing for this weekly review.
Handoff quality: smoothness before frequency
A good handoff passes the adviser a summary, spares the applicant from repeating the question, and ends in a human answer. Measure those three things, then break handoffs down by reason: explicit request for a person, out-of-scope topic, sensitive situation (disability support, visa questions, financial hardship), or agent failure.
A dominant reason points to a content gap. Visa and immigration questions are a typical example for international applicants: write down what the institution can say and route the rest to the right team.
A low handoff rate is not automatically good. The agent may be holding people who needed a person. The aim is not to replace advisers but to keep their time for conversations where judgement matters.
Qualified enquiries, not collected addresses
The number of emails captured is the most flattering and least informative figure. Define a qualified enquiry with four conditions: valid contact details, consent recorded, a course or project identified, and intent to act (apply, attend an open day, book a call).
Calculate capture against conversations showing real intent, then pair it with the share of those contacts an adviser actually reached. An unreachable contact is a cost, not a result.
UK data protection law and the Information Commissioner's Office (ICO) guidance expect data minimisation and clear privacy information. Collect only what the next step needs, and be careful about how much free text your dashboard retains.
Time to first human contact
Measure the gap between the request and the first adviser reply, as a median and a 90th percentile. The median describes the usual experience; the 90th percentile describes the unlucky applicant, often the one writing in the evening or during a holiday.
Separate office hours from closed hours, otherwise the figure blends two realities. The promise the agent makes ("an adviser will email you tomorrow morning") must be kept, so measure that.
From chat to enrolment
The real test is what becomes of applicants: enquiry, application, offer, firm acceptance, enrolment. Track them in cohorts by week of first contact and keep the same denominator from one cohort to the next.
Be cautious about attribution. An applicant who used the chatbot and enrolled might have done so anyway. Where possible, compare periods or pages with and without the agent, and state the uncertainty. Link the method to your student acquisition ROI calculation.
The KPIs that mislead
- Conversation volume: it follows site traffic, not value delivered.
- Engagement rate and average duration: a confusing answer also makes a conversation last longer.
- Satisfaction on its own: few people respond, and not at random.
- Raw lead counts: without qualification, they reward collection rather than recruitment.
- Deflection: it confuses no handoff with problem solved.
For satisfaction, prefer tracking across the whole journey, as in our piece on measuring prospect satisfaction.
Reading the numbers in a UK context
The UK cycle has sharp peaks: UCAS deadlines, results day, Clearing, and open days. Compare a period with the same period a year earlier, never with the preceding weeks.
Segment at least by applicant type (home undergraduate, international, postgraduate, mature or foundation-year entrant), by course and by channel. A global figure hides the fact that a Russell Group applicant and a mature student on a foundation year ask different questions and decide on different timelines. For the wider picture, see our guide to AI chatbots for student recruitment and the one on AI agents for student recruitment.
A simple review rhythm
- Weekly: read a random sample and classify errors.
- Monthly: review handoff reasons and unanswered questions.
- Termly: compare cohorts through to enrolment.
- After every content change: retest fees, dates and entry requirements.
Write the definitions in a shared document. Otherwise two teams compute two different "resolution rates" and the debate is about the number, not the applicants.
From numbers to decisions
A metric is only worth tracking if it triggers an action. Four simple rules are enough to start.
- Low resolution, few errors: content is missing. List the unresolved questions and write the answers with the relevant department.
- Errors on critical facts: fix the source before touching the agent. If the fees page is ambiguous, the agent will be too.
- Many contacts, few reached: the issue is follow-up, not capture. Check who calls back, when, and through which channel.
- Slow replies in the evening: adjust the promise made to applicants or the team's cover before adding technology.
Record each decision with its date and the metric that prompted it. A few months later you will know what actually changed the result.
