Testing an AI Support Agent on Past Tickets Before Go-Live
Connecting an AI agent to your helpdesk takes minutes. Trusting it takes a test plan: replay past tickets, run it in Draft only mode, and set the confidence threshold from real answers.
Connecting an AI agent to your helpdesk takes minutes. Trusting it with customers takes longer, and it should. The safe path has three steps. Watch it answer tickets you have already solved. Then let it draft on live tickets without sending anything. Only then let it reply.
This guide walks through that path with Kelu's Support Ticket AI, and the checks worth making at each step. The ideas apply to any AI agent you put in front of customers.
What can go wrong on day one?
Most first-week problems are not the model being clever in the wrong way. They are ordinary gaps between what the AI was given and what customers ask. Each one is cheap to find before go-live and expensive to find after:
- Missing content. The knowledge base holds the docs site but not the refund policy, the status page or the answers your agents keep in their heads. A careful AI declines everything in those areas. A careless one guesses.
- The wrong tickets. "Why was I charged twice?" needs someone who can see the account. Docs can't answer it, however good the retrieval.
- A threshold set by feel. Too low, and confident-sounding wrong answers go out. Too high, and the AI answers almost nothing and the project looks like a failure.
- Workflow surprises. Replies in the wrong inbox, tags your routing rules don't expect, a customer who wants a person and can't reach one.
- Internal notes reaching customers. Text written for agents gets quoted in a public reply. Kelu avoids this by design: helpdesk replies use only the knowledge base's Public documents, never restricted ones.
How do simulations on past tickets work?
A simulation replays tickets you have already resolved through the AI, as if they had just arrived. In Kelu you start one from the helpdesk integration's card: pick the integration, set Max tickets (up to 100) and run it. Kelu answers your most recent resolved tickets from the knowledge base and scores each answer. Nothing is posted to your helpdesk, and the tickets are not touched.
Each row shows the ticket, Kelu's draft answer, the reply your agent actually sent, and a confidence score. Reviewers approve or reject each answer with a thumbs up or down. The side-by-side view is the useful part. You are not asking "does this sound plausible?" but "is this as good as what we sent?"
Simulations have limits worth knowing. One runs at a time per integration, at most once an hour. A workspace can rehearse up to 500 tickets a day, and a run stops after 15 minutes. Re-run one after big changes: a docs migration, a new source, a new voice for replies. See Simulations for the full feature.
Then sort the results into three piles:
- Approved, high confidence. These are the tickets the AI can take. Note the lowest confidence score in this pile. You will need it later.
- Rejected, high confidence. The pile that matters most. Open the cited sources. Usually a page is out of date, two pages disagree, or the question needed account data. Fix the content, or keep that kind of ticket out of scope.
- Low confidence. The AI wasn't sure. If your agent's reply quoted something that exists, the content is there but hard to find, so improve the page. If not, it's a gap. Write the page, or add a Custom Answer.
Why start in Draft only mode?
Past tickets are a good test, but they are not today's tickets. Draft only mode is the bridge. A new Kelu helpdesk integration starts in Draft Only (Safe). Answers at or above the confidence threshold become private notes on the ticket, labelled with their confidence, and the ticket is tagged kelu-draft. Nothing reaches the customer. Agents read the draft, edit it if needed, and send it themselves.
Run it for a week. You learn what a simulation can't show: how drafts fit your agents' workflow, which inboxes get useful drafts, and whether agents send them as written. When agents stop editing the drafts for a kind of ticket, the AI is ready for that kind of ticket.
Two details help here. First, Kelu answers nothing until you pick where it may answer under Answer only in: Zendesk groups, Intercom team inboxes, Front inboxes or Freshdesk groups. Only tickets created after you save count. Start with one group and widen it later. Second, Linear is different. It has no private notes, so a draft would be a comment the reporter can read, and Draft only isn't offered there. Kelu also starts answering new Linear issues as soon as it is connected. So a Linear rollout has no drafting week: check the knowledge base first, and rely on the threshold and your escalation label.
Confidence thresholds and "I don't know"
Every answer gets a confidence score. Above the threshold, Kelu replies, or drafts in Draft only. Below it, nothing is posted, not even a note, and the ticket gets your escalation tag (needs-review by default) for your team to pick up. A new integration set up in the dashboard starts at 80%.
Set the threshold from your simulation, not from a guess. Put it where the approved answers sit, just under the lowest score you would have been happy to send. During the Draft only week, watch Avg confidence on the integration's Connection & Analytics tab.
The threshold is not the only guard, and it shouldn't be. When retrieval decides your sources don't cover the question, Kelu sets the confidence to zero, so the ticket is escalated however low the threshold is. An answer that declines is capped low too. And an answer that isn't supported by its own sources fails a grounding check and is withheld. This is abstention: "I don't know" is enforced by the system, not just requested in a prompt.
For a support agent, a clean hand-off is a good outcome. It costs one ticket. A wrong answer costs the ticket plus the customer's trust. So if your deflection rate is low after go-live, check your content before you lower the threshold. A low rate usually means missing pages, and Coverage Gaps lists the questions that went unanswered.
Go-live checklist
Adding a helpdesk needs a Pro or Enterprise plan. Support Ticket AI works with Zendesk, Intercom, Freshdesk, Front and Linear for tickets, and Freshchat for live chat. For how it compares with AI built into helpdesks, see AI support platforms compared. Then work down the list:
- Fill the knowledge base: docs, help center articles, and the policies agents quote. You can also index solved Zendesk tickets as a source. In a public knowledge base they become public documents: helpdesk replies use them, and so do your widget and public API. In an internal knowledge base they stay restricted, and helpdesk replies don't use them. Turn on PII masking before the first sync.
- Run a simulation on up to 100 recent resolved tickets. Compare every answer with your agent's reply.
- Fix what the rejected answers point at: stale pages, contradictions, missing policies.
- Set the confidence threshold where the approved answers sit.
- Pick one group or inbox under Answer only in.
- Run Draft only for a week, and read the drafts with the agents who would have sent them.
- Test the hand-offs. A customer who replies to an automated answer gets an internal note and your escalation tag, and Kelu stops answering that ticket. A customer who asks for a person is handed over without an answer. Writing stop ai replies in an internal note stops Kelu on that ticket.
- Turn Draft only off. For the first weeks, watch the deflection rate, average confidence and how many customers replied after an automated answer.
- Add more groups or inboxes once the numbers hold.
FAQ
Does a simulation post anything to my helpdesk? No. Kelu answers past resolved tickets from your knowledge base and scores each answer, but posts nothing and leaves the tickets as they are.
How many past tickets can I test on? Up to 100 of your most recent resolved tickets per run, one run per integration an hour, and up to 500 tickets a day per workspace.
What does the customer see in Draft only mode? Nothing. Confident answers become private notes for an agent to send, and tickets below the threshold only get the escalation tag. Linear is the exception: it has no private notes, so Draft only isn't offered there.
What happens to a ticket the AI isn't sure about? Nothing is posted. The ticket gets your escalation tag and stays with your team. The same happens when your sources don't cover the question at all.
Does Kelu close the tickets it answers? No. A confident answer is posted with links to its sources and the ticket is tagged kelu-answered, but it stays open in your team's queue.
Keep reading
What Is Ticket Deflection? Definition, Formula, and How to Improve It
Ticket deflection is the share of support questions resolved without creating a ticket. Here is how to define it, measure it honestly, and raise it.
How to Deflect Support Tickets Before Submission: The Form Deflector Guide
Post-submission auto-replies fail because tickets are already filed. Here is how form deflection answers the question when the user presses submit, before the ticket exists.
RAG vs Fine-Tuning for Documentation Q&A: Which One Do You Need?
For answering questions over your own documentation, retrieval-augmented generation beats fine-tuning on freshness, citations, and cost. Here is the decision framework.
Turn your knowledge into answers
Connect your docs, policies, or playbooks and see cited AI answers in minutes — free, no credit card required.
Free to start · No credit card required