ChatGPT
Claude
Perplexity
Gemini
GrokFive measures, each defined before launch
An AI admissions agent at an Irish university, technological university or private college should be judged on five measures: resolution rate, handoff quality, qualified enquiries, time to first human reply, and the path from enquiry to registration. Other figures help with diagnosis but should not set targets.
No dependable public benchmark gives a good resolution rate for an Irish institution, so this article does not invent one. It provides formulas and traps; your thresholds come from your own conversations once you have a baseline.
| KPI | Formula | Trap to avoid |
|---|---|---|
| Resolution rate | Questions genuinely answered ÷ real questions asked | Counting abandoned chats as solved |
| Handoff quality | Complete, accepted handoffs ÷ handoffs requested | Tracking only handoff volume |
| Qualified enquiries | Valid, consented contacts with a course interest ÷ chats with real intent | Counting every email address as an applicant |
| Time to human reply | Median and 90th percentile to first staff response | Reporting only the average |
| Path to registration | Applicants by stage, grouped by week of first contact | Crediting the chatbot with the whole funnel |
Resolution rate: settled, not merely unchallenged
Resolution rate is the share of real questions that received a correct and sufficient answer. Leave greetings, staff tests and empty sessions out of the denominator.
Avoid "deflection", which counts any chat without a handoff as solved. Each week, read a random sample and classify it: resolved, partly resolved, unresolved, wrong. A feedback prompt helps, but only the very pleased and very annoyed use it.
Track accuracy on critical facts separately: course entry points and Leaving Certificate subject requirements, fees and eligibility for free fees, CAO deadlines, QQI level of the award, accommodation terms. A wrong answer about a subject requirement is worse than a handoff. The NIST AI Risk Management Framework treats measurement as a core function of AI governance, and a weekly transcript review is a practical way to apply it.
Handoff quality: smooth beats frequent
A good handoff gives the staff member a summary, spares the applicant from repeating the question, and ends in a human answer. Break handoffs down by reason: explicit request for a person, out-of-scope topic, sensitive situation (disability support, grant eligibility, non-EU immigration), or agent failure.
Mature-student and further-education route questions are a common cause of handoffs because they fall outside the standard points route. If they dominate, document the alternative routes and signpost the right office.
The aim is to keep staff time for conversations where judgement matters, not to remove people from the process.
Qualified enquiries
Define a qualified enquiry by four conditions: valid contact details, consent recorded, a course identified, and intent to act (apply, attend an open day, book a call). Compute capture on chats showing real intent, then add the share of contacts staff actually reached.
Irish institutions work under EU data protection law, and the Data Protection Commission publishes guidance. Collect only what the next step needs, give clear privacy information, and think carefully about how long you keep free-text conversations. Your data protection officer should sign off the dashboard design.
Time to first human reply
Measure the gap between request and first staff reply, as a median and a 90th percentile. Separate office hours from after-hours. The agent's promise ("someone will contact you tomorrow morning") must be tracked as kept or broken, not only as speed.
Peaks stretch response times: the Leaving Certificate results week, CAO offer rounds, and the weeks before registration. Compare with the same week a year earlier.
From enquiry to registration
Track applicants through enquiry, application, offer, acceptance and registration, in cohorts by week of first contact, with a constant denominator.
Be careful with attribution. An applicant who used the chatbot and registered might have done so anyway. Compare periods or pages with and without the agent where possible, and state the uncertainty. Link the method to your student acquisition ROI model.
Metrics that mislead
- Chat volume: it follows site traffic, not value delivered.
- Engagement and session length: confusion also lengthens a chat.
- Satisfaction alone: few respond, and not at random.
- Raw lead counts: they reward collection over recruitment.
- Deflection: no handoff is not the same as solved.
Reading the numbers in the Irish cycle
Most undergraduate applicants apply through the CAO, with its points system and fixed calendar, while postgraduate, further education and many private college applications go direct. These populations ask different questions at different times, so segment by applicant type (school leaver, mature, international, postgraduate), by course and by channel.
Private colleges and universities also differ in how they present QQI-validated awards, so make sure the agent states the award and level exactly as the institution does. For the broader picture, see our guide to AI chatbots for student recruitment and our piece on AI agents for student recruitment.
A simple review rhythm
- Weekly: read a random sample and classify errors.
- Monthly: review handoff reasons and unanswered questions.
- Each semester: compare cohorts through to registration.
- After any content change: retest fees, dates and entry requirements.
From numbers to decisions
A metric is only worth tracking if it triggers an action. Four simple rules are enough to start.
- Low resolution, few errors: content is missing. List the unresolved questions and write the answers with the relevant department.
- Errors on critical facts: fix the source before touching the agent. If the fees page is ambiguous, the agent will be too.
- Many contacts, few reached: the issue is follow-up, not capture. Check who calls back, when, and through which channel.
- Slow replies in the evening: adjust the promise made to applicants or the team's cover before adding technology.
Record each decision with its date and the metric that prompted it. By the next admissions cycle you will know what actually changed the result.
Comparing school leavers and other routes
School leavers applying through the CAO, mature applicants, further education progression students and international applicants ask different questions on different timelines. Compute every metric separately for these groups before building a combined figure. A strong overall resolution rate can hide poor results for applicants on non-standard routes, whose questions are harder to answer from a single web page.
