AI receptionist evaluation checklist

A demonstration shows a tool on its best day. A trial should show it on your worst one. This checklist is what I would test before an AI receptionist answers a single real patient.

Set up the trial

• Use a test number or a hidden web page, not your live channels.
• Ask two or three members of the team to play patients, using situations from your own enquiry log.
• Agree in advance what a pass looks like for each scenario, and record each result.
• Use invented patient details only. Nothing from real records goes into a trial.

Scenarios to test

Price questions matter twice over. The GDC expects clear prices in practice literature and on websites, so patients should not have to ask ^[3], which makes published fees a fair thing for a tool to repeat. But a treatment plan must include a realistic indication of cost ^[3], and that comes from a clinician, not from a chatbot guessing. Patients should also receive clear information about emergency and out-of-hours arrangements ^[3], so test the urgent scenario both in hours and out of hours.

For complaints, the GDC expects an effective written complaints procedure that is followed at all times ^[5]. The tool's job is to record the complaint and route it into that procedure, never to handle it.

Disclosure

Check that the first message says it is an AI, in plain words. A greeting I would accept: "Hello, I am the [Practice] virtual assistant. I am an AI, and I can help with appointments and general questions. You can ask for a member of the team at any time." Then ask it directly whether it is a person, halfway through a conversation, and check the answer is just as plain.

Data handling

• What it collects: personal data should be limited to what is necessary for the purpose ^[1]. Check it does not ask for medical history to book a routine appointment.
• The contract: a supplier processing data for you needs a binding contract covering documented instructions, confidentiality, security, sub-processors, and deletion or return at the end ^[2].
• Security: the GDC expects confidential information sent or stored electronically to be encrypted ^[4]. Ask how transcripts and recordings are stored, where, and for how long.
• Training: ask in writing whether your conversations are used to train or improve the supplier's models.

Logs and failure handling

• Can you read every conversation, with the time, channel and outcome?
• Are handovers marked, and can you see who picked each one up and when?
• Can you export the logs if you change supplier?
• What happens when the tool is down: does the call or message reach a person, or a dead end?
• What happens when it does not understand: does it offer a person, or loop?

Once it passes, set the rules for what it may book and when it must hand over, using human handover and booking-rule design, and weigh it against other candidates in the prioritisation worksheet.