Start with the knowledge base, not the prompt
Write down what the support team is allowed to answer. Every document needs an owner, a scope, an effective date, and a replacement rule. Delete or archive pages that conflict with current policy.
| Knowledge base field | Why the automation needs it |
|---|---|
| Title and product area | Helps retrieval and gives the agent a readable source |
| Effective date and expiry | Stops an old policy from winning by accident |
| Audience and channel | Separates an internal runbook from a customer answer |
| Source URL or document ID | Lets a person verify the passage |
| Owner and review date | Creates a clear update path when policy changes |
| Allowed answer scope | Tells the model when to escalate |
Chunk documents around a single procedure or decision. Keep the source heading with each chunk. A short paragraph with its conditions is easier to verify than a page fragment with no context.
A retrieval flow for Telegram and web chat
Use the same support service behind both channels. The Telegram adapter and web widget should pass a normalized question, user context, and conversation ID. They should not implement separate policy logic.
channel update
-> normalize question and permissions
-> retrieve approved passages
-> answer policy and model route
-> source check and confidence rule
-> reply, ask for detail, or hand off
The model should receive the question, the selected passages, and a clear instruction to say when the passages do not answer it. The retrieval layer should return source IDs so your UI can show where an answer came from.
Model routes to evaluate
The live catalog checked on 2026-10-03 contains these candidate IDs. The rows describe test roles, not benchmark results or guarantees.
| Workload | Candidate IDs | What to inspect |
|---|---|---|
| Short answer grounded in one or two passages | claude-sonnet-5, gpt-6-luna, deepseek-v4-flash | Citation selection and refusal when the source is silent |
| Long policy with exceptions | claude-opus-5-5, gpt-6.1-sol, gemini-3.8-flash | Conditions, source coverage, and answer length |
| Second-family comparison | claude-fable-5-1, gpt-6-sol, qwen3.8-flash | Same fixture, same output schema, separate review |
| Triage before retrieval | gpt-5.6-luna, gemini-3.7-flash, deepseek-v4.1-flash | Intent label, language, and escalation flag |
Choose a route by verified fixture results. Keep the selected model in configuration so a model catalog change does not silently alter every support answer.
Return structured support data
Ask the model for fields your application can validate. A response should carry the answer, source IDs, an escalation flag, and a short reason. Reject malformed output instead of repairing it into an action.
type SupportAnswer = {
answer: string;
sourceIds: string[];
needsHuman: boolean;
handoffReason: string | null;
};
function canSend(answer: SupportAnswer, sourceIds: Set<string>) {
const knownSources = answer.sourceIds.every((id) => sourceIds.has(id));
return knownSources && (answer.needsHuman || answer.sourceIds.length > 0);
}
The application still checks permissions, account state, and policy scope. A source ID is evidence for a draft answer. It is not permission to perform a refund or change an account.
Freshness and conflict handling
Use an update event or a bounded refresh job when a policy changes. Store the version that produced each answer. If two documents conflict, stop the automatic answer and send the conflict to the document owners.
Track these fields for each response:
- question and channel ID, with personal data removed where possible;
- model ID, knowledge base version, source IDs, and policy version;
- elapsed time, retrieval result count, parser status, and escalation reason;
- agent decision when a human reviews the draft.
Keep retention short enough for your support policy. Access to transcripts and source documents needs the same permissions as the support console.
An evaluation set that reflects real tickets
Build a small fixture set from resolved tickets after removing personal data. Include a correct answer, a missing-policy case, two conflicting policies, a multilingual question, a prompt injection, an account-specific request, and an explicit handoff request.
For each fixture, check factual support, source IDs, refusal behavior, schema validity, and escalation. Record false answers and false handoffs separately. A single pass rate hides the failure that matters to your team.
The Telegram support bot provider guide covers webhook idempotency, queues, and provider errors. The model selection guide helps turn the fixture results into a route configuration.
FAQ
Does retrieval remove hallucinations from support automation?
No. Retrieval gives the model source material, but the application still needs source checks, refusal rules, tests, and human escalation.
How many passages should I send to a model?
Start with the smallest set that answers the question and test it on your fixtures. More text can hide the condition that the answer needs.
Should internal runbooks be visible to customers?
Keep audience metadata on every document and filter it before generation. A support agent can see an internal runbook while a customer receives a safe summary.
Can the bot perform account actions?
Only through explicit tools with permission checks, validation, idempotency, and an audit record. A retrieved passage or model answer cannot authorize an action.
How do I keep policies current?
Assign an owner, effective date, expiry or review date, and replacement rule to every source. Send conflicting or expired material to a person instead of guessing.