ChatGPT
Claude
Perplexity
Gemini
GrokFive measures, each with a written definition
For a Canadian university or college, an AI admissions agent should be judged on five measures: resolution rate, handoff quality, qualified inquiries, time to first human reply, and the path from inquiry to enrolment. The rest is diagnostic.
No reliable public benchmark says what a good resolution rate looks like for a Canadian institution, so none is given here. The article offers formulas and traps; the thresholds come from your own conversations.
| KPI | Formula | Trap to avoid |
|---|---|---|
| Resolution rate | Questions truly answered ÷ real questions asked | Counting abandoned chats as solved |
| Handoff quality | Complete, accepted handoffs ÷ handoffs requested | Tracking only handoff volume |
| Qualified inquiries | Valid, consented contacts with a program interest ÷ chats with real intent | Counting every email as an applicant |
| Time to human reply | Median and 90th percentile to first advisor response | Reporting only the average |
| Path to enrolment | Applicants by stage, grouped by week of first contact | Crediting the bot with the whole funnel |
Resolution rate: settled, not merely unchallenged
Resolution rate is the share of real questions that received a correct and sufficient answer. Exclude greetings, staff tests and empty sessions.
Avoid "deflection," which counts any chat without a handoff as solved. Each week, read a random sample and classify it: resolved, partly resolved, unresolved, wrong. A direct feedback prompt helps, but only the very happy and very unhappy answer it.
Track accuracy on critical facts separately: tuition and fees for domestic and international students, admission averages and prerequisites, application deadlines, co-op requirements. A wrong answer on a prerequisite is worse than a handoff. The NIST AI Risk Management Framework names measurement as a core function of AI governance, and a weekly transcript review applies it in practice.
Handoff quality: smooth beats frequent
A good handoff gives the advisor a summary, spares the applicant from repeating the question, and ends in a human answer. Break handoffs down by reason: explicit request for a person, out-of-scope topic, sensitive situation (accessibility accommodations, student aid, study permits), or agent failure.
Study-permit and transfer-credit questions are a common source of repeated handoffs for institutions with international and transfer applicants. If they dominate, document what can be said publicly and route the rest.
The goal is not to replace advisors but to reserve their time for conversations where judgment matters.
Qualified inquiries
Define a qualified inquiry by four conditions: valid contact details, consent recorded, a program identified, and intent to act (apply, attend an open house, book a call). Compute capture on chats with real intent, then add the share of contacts an advisor actually reached.
Privacy rules depend on where you operate. The federal PIPEDA framework, overseen by the Office of the Privacy Commissioner of Canada, applies to many private organizations, while Quebec institutions deal with Law 25, and public institutions in several provinces fall under provincial public-sector legislation. Ask your privacy officer which regime governs your institution, collect only what the next step needs, and be cautious about retaining free-text conversations. Commercial follow-up by email or text also falls under Canada's anti-spam law.
Time to first human reply
Measure the gap between request and first advisor reply, as a median and a 90th percentile. Separate office hours from after-hours. The agent's promise ("an advisor will email you tomorrow morning") must be kept; measure that promise rather than raw speed.
Peaks matter: application deadlines, offer releases, and open house season stretch response times. Compare a week with the same week last year.
From inquiry to enrolment
Track applicants through inquiry, application, offer, acceptance and enrolment, in cohorts by week of first contact, with a constant denominator.
Be careful with attribution. An applicant who used the chatbot and enrolled might have done so anyway. Compare periods or pages with and without the agent where possible and state the uncertainty. Tie the method to your student acquisition ROI model.
Metrics that mislead
- Chat volume: it follows site traffic, not value delivered.
- Engagement and session length: confusion also lengthens a chat.
- Satisfaction alone: few respond, and not at random.
- Raw lead counts: they reward collection over recruitment.
- Deflection: no handoff is not the same as solved.
Canadian context for reading the numbers
Education is a provincial matter, so the funnel differs. In Ontario, many undergraduate applicants apply through OUAC; in Quebec, applicants move through the CEGEP system before university and the R score matters for contingented programs; other provinces run their own application services. Segment by province of origin, applicant type (high school, transfer, mature, international, graduate) and program.
Language is a further cut. Where you serve both English and French, measure resolution separately: thinner content in one language shows up quickly in samples. For the broader picture, see our guide to AI chatbots for student recruitment and our piece on AI agents for student recruitment.
A simple review rhythm
- Weekly: read a random sample and classify errors.
- Monthly: review handoff reasons and unanswered questions.
- Each term: compare cohorts through to enrolment.
- After any content change: retest fees, deadlines and prerequisites.
From numbers to decisions
A metric is only worth tracking if it triggers an action. Four simple rules are enough to start.
- Low resolution, few errors: content is missing. List the unresolved questions and write the answers with the relevant department.
- Errors on critical facts: fix the source before touching the agent. If the tuition page is ambiguous, the agent will be too.
- Many contacts, few reached: the issue is follow-up, not capture. Check who calls back, when, and through which channel.
- Slow replies in the evening: adjust the promise made to applicants or the team's coverage before adding technology.
Record each decision with its date and the metric that prompted it. A few months later you will know what actually changed the result.
Comparing across provinces and programs
An institution that recruits across several provinces, or runs very different programs (business, engineering, health, trades), will see resolution rates that cannot be compared directly. Compute each metric by program before building a single headline number, and compare only programs whose questions look alike. Otherwise a program with heavy demand for administrative information drags the average down without the agent being at fault.
