Building a Support Agent That Knows When to Stop — My Agentic AI Capstone
A technical deep-dive into my IIT Madras Agentic AI capstone: a framework-free customer-support resolution agent with deterministic pre-LLM safety, grounded RAG, guarded tool use, memory — and the live date-arithmetic failure that reshaped its architecture.
Every capstone in an agentic AI program builds an agent. Mine ended up being about something narrower and, I think, more useful: teaching an agent when to stop — when to refuse, when to escalate, and when to admit it doesn’t know.
The project is a support-resolution agent for NovaMart, a fictional electronics retailer handling ~4,000 support contacts a day. Roughly 70% of those are questions whose answers exist verbatim in policy documents or the order database. The agent’s job is to resolve that 70% with grounded, cited answers — and convert the risky 30% (chargebacks, unauthorized orders, discretion calls) into well-formed, prioritised human tickets.
I built it framework-free (Track B): plain Python plus the OpenAI API, ~900 lines of owned code, five dependencies. Everything LangChain would have provided — RAG, tool orchestration, memory, planning — exists in the codebase directly and inspectably, which mattered the day I had to answer “why did it say that?”
Three design decisions carry the project:
Safety runs before the LLM, deterministically. Six refusal patterns (fraud, verification bypass, cross-customer data…) and six escalation triggers (chargeback, unauthorized order, threats…) are regex checks that fire in 0.1ms — unsafe requests never reach the model at all. The system prompt repeats the rules only as a second layer.
Retrieval is unconditional, and silence beats improvisation. A numpy cosine store over 31 section-aware chunks grounds every answer with policy-id citations; below a 0.25 similarity floor the agent says “I don’t have that information” and offers a human, never a guess.
If a sub-task has a deterministic solution, route around the LLM. The best lesson came from a live failure: asked about returning a laptop delivered 8 days earlier — inside the 14-day window — the agent retrieved the right policy, fetched the right delivery date, did the subtraction itself, and got it wrong. A wrong verdict wrapped in a correct citation. The fix wasn’t a prompt tweak; it was architectural — return-intent queries now always pre-run a pure-Python eligibility tool whose verdict is injected as authoritative. The LLM explains the verdict; it no longer produces it.
The final build passes 10/10 behavioural eval cases (tool use, grounding, refusal, escalation) at 1.3s median latency, with zero fabricated policies observed and PII-redacted logs throughout.
This capstone concludes my Agentic AI program at IIT Madras. The full pipeline — mock mode included, so it runs without an API key — is reproducible from the repository, and every claim above is backed by a regenerable artefact in its evidence/ folder.
About the Author
Ajay Walia
AI {IT Architect} focusing on local-first multi-agent AI engineering, zero-data-egress systems. Ideator, Creator and Executor on Curious Bit.
Keep Reading
Aether, Rethought — The Shape Was Wrong All Along
Our first build mirrored the org chart. It was the wrong shape. Here's how five recognised agentic design patterns, scored against the same criteria, led to a hybrid recommendation — and what changes in v3.
I Built My Own RSS Reader in an Afternoon — With AI Doing the Typing
How I built LumenAI — a local-first, native macOS RSS reader with full-text search, offline reading, and pluggable AI summaries — from an empty folder to a signed DMG in about an hour, with Claude writing every line.
Aether, Grown Wild — The Implementation Journey (v2.6 → v2.8.2)
The second chapter of Aether: how a local-first team of IT architecture agents grew from a clean idea into a 13-agent, web-first, self-escalating system — and every bug that shaped it along the way.