ChatGPT
Claude
Perplexity
Gemini
GrokTrack five numbers and define each one in writing
An AI admissions agent earns its place on five measures: resolution rate, handoff quality, qualified inquiries, time to first human reply, and what happens to applicants from inquiry to enrollment. Every other number is a diagnostic aid, not a goal.
There is no trustworthy public benchmark for a "good" resolution rate at a college or university, so this article gives none. It gives formulas and reading traps. Set thresholds from your own conversations after a few weeks of baseline.
| KPI | Formula | Trap to avoid |
|---|---|---|
| Resolution rate | Questions truly answered ÷ real questions asked | Counting a dropped chat as solved |
| Handoff quality | Complete, accepted handoffs ÷ handoffs requested | Measuring only how many handoffs occur |
| Qualified inquiries | Valid, consented contacts with a program interest ÷ chats showing real intent | Counting every email as a prospect |
| Time to human reply | Median and 90th percentile, request to first counselor response | Averaging away the slow cases |
| Inquiry-to-enrollment path | Students by stage, grouped by week of first contact | Crediting the bot with the whole funnel |
Resolution rate: settled, not merely unchallenged
Resolution rate is the share of real questions that got a correct, sufficient answer. Keep greetings, staff tests and empty sessions out of the denominator.
Watch out for "containment" or "deflection," which counts any chat without a handoff as a win. A student who leaves frustrated after a vague answer then looks like a success.
Sample instead. Every week, read a random set of conversations and label them resolved, partly resolved, unresolved, or wrong. Add a thumbs-up or thumbs-down prompt, remembering that only the happiest and unhappiest users click.
Track accuracy on high-stakes facts separately: cost of attendance, deadlines for early action and regular decision, test-optional rules, scholarship criteria, housing terms. A wrong answer on financial aid is worse than a handoff. The NIST AI Risk Management Framework lists measurement among its core functions, and a weekly transcript review is a practical way to apply it.
Handoff quality: smooth beats frequent
A good handoff gives the counselor a summary, spares the student from repeating the question, and ends in a human answer. Measure all three, then break handoffs down by reason: explicit request for a person, out-of-scope topic, sensitive situation (disability accommodations, financial hardship, visa questions), or agent failure.
A dominant reason signals a content gap. Financial aid appeals are a classic: if they dominate, say what the office can state publicly and route the rest.
A low handoff rate is not automatically good. The agent might be holding students who needed a person. The goal is to keep counselors' time for the conversations where judgment changes outcomes.
Qualified inquiries, not collected emails
Email count is the most flattering and least useful number. Define a qualified inquiry by four conditions: valid contact details, consent recorded, a program or major identified, and intent to act (apply, visit campus, book a call).
Compute capture against chats showing real intent, then pair it with the share of those contacts a counselor actually reached. An unreachable contact is a cost.
On privacy, FERPA protects education records of enrolled students, and the U.S. Department of Education's Student Privacy Policy Office explains how it works. Prospects who have not enrolled are generally outside FERPA, but state privacy laws, the FTC Act and rules on follow-up calls, texts and emails can apply to what you collect and how you contact people. Ask your counsel before storing free-text chats in an analytics tool, and collect only what the next step needs.
Time to first human reply
Measure the gap between a request and the first counselor reply as a median and a 90th percentile. The median shows the usual experience; the 90th percentile shows the student who wrote at night or over a break.
Split business hours from after-hours. The agent's promise ("a counselor will email you tomorrow morning") must be kept, so measure that promise rather than raw speed.
From inquiry to enrollment
The real test is what students do next: inquiry, application started, application submitted, admit, deposit, enrollment. Track them in cohorts by week of first contact and keep the denominator constant.
Be careful with attribution. A student who used the chatbot and deposited might have done so anyway. Compare periods or pages with and without the agent where you can, and state the uncertainty. Link this to your student acquisition ROI model.
Vanity metrics
- Chat volume: it follows website traffic, not value delivered.
- Engagement rate and session length: a confusing answer also keeps people chatting.
- Satisfaction alone: few respond, and not at random.
- Raw lead counts: without qualification they reward collection over recruitment.
- Deflection: no handoff does not mean problem solved.
For satisfaction, track the whole funnel, as in our article on measuring prospect satisfaction.
Reading the numbers in the US cycle
The US cycle runs on deadlines: early action and early decision, regular decision, financial aid filing, decision day and summer melt. Compare a week with the same week last year. Segment by student type (first-year, transfer, graduate, international, adult learner), by program and by channel.
A first-year applicant using the Common App and a transfer student ask different questions on different timelines, and a blended figure hides both. For the broader picture, read our guide to AI chatbots for student recruitment and our piece on AI agents for student recruitment.
A simple review rhythm
- Weekly: read a random sample and classify errors.
- Monthly: review handoff reasons and unanswered questions.
- Each term: compare cohorts through to enrollment.
- After any content change: retest costs, deadlines and aid rules.
Put the definitions in a shared document so two teams do not calculate two different "resolution rates."
From numbers to decisions
A metric is only worth tracking if it triggers an action. Four simple rules are enough to start.
- Low resolution, few errors: content is missing. List the unresolved questions and write the answers with the financial aid, housing or registrar teams.
- Errors on critical facts: fix the source before touching the agent. If the cost-of-attendance page is ambiguous, the agent will be too.
- Many contacts, few reached: the issue is follow-up, not capture. Check who calls back, when, and through which channel.
- Slow replies in the evening: adjust the promise made to students or the team's coverage before adding technology.
Record each decision with its date and the metric that prompted it. By the next cycle you will know what actually changed the result and what did not.
