ChatGPT
Claude
Perplexity
Gemini
GrokFive measures, each defined before launch
An AI admissions agent at an Australian university, TAFE or private college should be judged on five measures: resolution rate, handoff quality, qualified enquiries, time to first human reply, and the path from enquiry to enrolment. Other numbers help diagnose, but they should not set targets.
No dependable public benchmark states a good resolution rate for an Australian institution, so none appears here. The article offers formulas and reading traps; thresholds come from your own conversations once you have a baseline.
| KPI | Formula | Trap to avoid |
|---|---|---|
| Resolution rate | Questions genuinely answered ÷ real questions asked | Counting abandoned chats as solved |
| Handoff quality | Complete, accepted handoffs ÷ handoffs requested | Tracking only handoff volume |
| Qualified enquiries | Valid, consented contacts with a course interest ÷ chats with real intent | Counting every email address as an applicant |
| Time to human reply | Median and 90th percentile to first adviser response | Reporting only the average |
| Path to enrolment | Applicants by stage, grouped by week of first contact | Crediting the chatbot with the whole funnel |
Resolution rate: settled, not merely unchallenged
Resolution rate is the share of real questions that received a correct and sufficient answer. Leave greetings, staff tests and empty sessions out of the denominator.
Avoid "deflection", which counts any chat without a handoff as solved. A student who leaves frustrated after a vague answer would score as a success. Each week, read a random sample and classify it: resolved, partly resolved, unresolved, wrong.
Track accuracy on critical facts separately: course fees for domestic and international students, entry requirements, ATAR or alternative pathway rules, intake dates, scholarship criteria. A wrong answer about a prerequisite is worse than a handoff. The NIST AI Risk Management Framework treats measurement as a core function of AI governance, and a weekly transcript review is a practical way to apply it.
Handoff quality: smooth beats frequent
A good handoff gives the adviser a summary, spares the student from repeating the question, and ends in a human answer. Break handoffs down by reason: explicit request for a person, out-of-scope topic, sensitive situation (disability support, financial hardship, student visa questions), or agent failure.
Student visa questions are a common source of handoffs for institutions recruiting offshore. If they dominate, document what the institution can say and route the rest. The aim is to keep advisers for the conversations where judgement changes the outcome, not to remove them.
Qualified enquiries
Define a qualified enquiry by four conditions: valid contact details, consent recorded, a course identified, and intent to act (apply, attend an open day, book a call). Compute capture on chats showing real intent, then add the share of contacts an adviser actually reached.
The Privacy Act 1988 and the Australian Privacy Principles govern many providers, and the Office of the Australian Information Commissioner (OAIC) publishes guidance. Commercial follow-up by email or text is also subject to Australia's spam legislation. Check with your privacy officer which obligations apply to you, collect only what the next step needs, and be careful about how much free text your dashboard retains.
Time to first human reply
Measure the gap between request and first adviser reply, as a median and a 90th percentile. Separate office hours from after-hours. Offshore applicants write in other time zones, so the agent's promise ("an adviser will contact you within one business day") must be tracked as a promise kept or broken, not just as speed.
Peaks stretch response times: ATAR release and offer rounds, orientation periods, and the weeks before semester intakes. Compare with the same week a year earlier.
From enquiry to enrolment
Track applicants through enquiry, application, offer, acceptance and enrolment, in cohorts by week of first contact, with a constant denominator.
Be careful with attribution. A student who used the chatbot and enrolled might have done so anyway. Compare periods or pages with and without the agent where possible, and state the uncertainty. Link the approach to your student acquisition ROI model.
Metrics that mislead
- Chat volume: it follows site traffic, not value delivered.
- Engagement and session length: confusion also lengthens a chat.
- Satisfaction alone: few respond, and not at random.
- Raw lead counts: they reward collection over recruitment.
- Deflection: no handoff is not the same as solved.
Reading the numbers in the Australian cycle
Admissions run through state centres such as UAC in New South Wales and the ACT, alongside direct applications for many private providers and international students. Group of Eight universities, regional universities, TAFE and private colleges face different volumes and question mixes. Segment by applicant type (school leaver, mature age, VET pathway, international, postgraduate), by course and by channel.
TEQSA registration and quality standards shape what providers may say about courses, so wrong answers on accreditation or pathways carry more risk than on most topics. For the broader picture, see our guide to AI chatbots for student recruitment and our piece on AI agents for student recruitment.
A simple review rhythm
- Weekly: read a random sample and classify errors.
- Monthly: review handoff reasons and unanswered questions.
- Each trimester or semester: compare cohorts through to enrolment.
- After any content change: retest fees, dates and entry requirements.
From numbers to decisions
A metric is only worth tracking if it triggers an action. Four simple rules are enough to start.
- Low resolution, few errors: content is missing. List the unresolved questions and write the answers with the relevant faculty or student services team.
- Errors on critical facts: fix the source before touching the agent. If the fees page is ambiguous, the agent will be too.
- Many contacts, few reached: the issue is follow-up, not capture. Check who calls back, when, and through which channel.
- Slow replies outside business hours: adjust the promise made to students or the team's cover before adding technology, especially for offshore enquiries.
Record each decision with its date and the metric that prompted it. By the next intake you will know what actually changed the result.
Comparing domestic and international enquiries
Domestic school leavers and offshore applicants ask different questions, in different time zones, with different decision timelines. Compute every metric separately for the two groups before building a combined figure. A strong overall resolution rate can hide poor results for international applicants, whose questions about visas, English-language requirements and credit recognition are harder to answer from a single web page.
