A technical deep-dive into my capstone: a framework-free customer-support resolution agent with deterministic safety, grounded RAG, tool use, memory, and the live production failure that reshaped its architecture.
NovaMart — the fictional electronics retailer this agent serves — handles roughly 4,000 support contacts a day. About 70% of them are questions whose answers exist verbatim in policy documents or the order database: where is my package, can I return this, what's the warranty. The remaining 30% need human judgment: disputes, discretion, security incidents.
The failure modes of canned chatbots are what make this dangerous. They misinform — quoting a generic 30-day return window for a laptop that only has 14 — and they don't know when to stop: no escalation path, no refusal, no uncertainty handling. So the problem statement isn't "answer questions." It's:
Resolve routine support cases with grounded, auditable answers — and hand off everything else safely.
The whole agent is a single, inspectable turn pipeline — no framework, no hidden control flow. Watch the message travel: the crucial detail is that safety runs first and is deterministic. Unsafe requests are refused or escalated before a single token reaches the LLM.
Modules map 1:1 to program phases: baseline.py (P2) → prompts.py (P3) → retrieval.py (P4) → tools.py (P5) → planner.py+memory.py (P6) → feedback.py (P7) → app.py FastAPI deployment (P8) → evaluation harness (P9), with safety.py cross-cutting.
Most agent stacks put safety in the system prompt and hope the model complies. This agent treats prompt-level safety as the second layer. The first layer is a deterministic regex gate in safety.py — 6 refusal patterns (fraud, verification bypass, other customers' data, legal advice, volunteered PII, social engineering) and 6 escalation patterns (chargebacks, unauthorized orders, frustration, human requests, privacy requests, threats), each mapped to a ticket priority.
Because it runs pre-LLM, refusals are 100% reproducible, testable without tokens, and effectively free — 0.1ms vs ~3,000ms for an LLM turn. Try it:
Here are the real captures from the demo run (live gpt-4o-mini mode). Note the latency column — the refusal and the escalation never touched the model:
The same gate works offline. Running the CLI in mock mode (no API key at all), the refusal and escalation behave identically — proof the safety layer is structural, not model-dependent:
The knowledge base is five policy documents (returns, refunds, shipping, warranty, account security). At this scale a vector database is overhead, so retrieval.py is a full RAG pipeline in under 100 lines: section-aware chunking (split on markdown ## headings so each policy rule stays self-contained), provider embeddings (text-embedding-3-small), and a normalized numpy matrix searched with cosine similarity. The index is a JSON file you can open and read.
Two design decisions matter more than the store itself:
Retrieval is unconditional. The planner routes tools, but it never gets to decide whether to retrieve — grounding every answer is the core safety property, and retrieval is cheap. And below a similarity threshold (0.25), the agent says "I don't know." An empty result set injects an explicit NO RELEVANT POLICY FOUND marker into context, which the prompt converts into a human handoff instead of a guess.
Each retrieved chunk carries its source and score into the prompt, and the V3 prompt forces the model to cite the governing policy id (POL-RET-2026.1) in its answer — every claim is auditable back to a document section.
The agent has four tools behind a guarded ToolRunner: order_lookup, check_return_eligibility, kb_search, and create_escalation_ticket. Allow-list only, four calls per turn max, duplicate calls short-circuited, and the whole surface is read-only except ticket creation — the agent cannot modify orders or move money by construction.
The most instructive moment of the project came from live testing on July 10, 2026. Asked about returning a laptop delivered 8 days earlier — comfortably inside the 14-day window — the agent declared it outside the window and steered the customer toward a discretionary escalation. It had retrieved the right policy. It had the right delivery date from order_lookup. Then it did the days-since-delivery subtraction itself, and got it wrong.
This is the worst kind of failure: a wrong verdict wrapped in a correct citation. It looks authoritative precisely because everything around the error is right.
No prompt tweak reliably fixes LLM date arithmetic. The fix was architectural: deterministic eligibility routing. Any return/refund intent with a known order id now always pre-runs check_return_eligibility — pure-Python, category-aware date math — and its verdict is injected into context marked authoritative, do not recompute dates. The LLM's job shrinks from producing the verdict to explaining it.
check_return_eligibility — category-aware date math (laptops/tablets/phones/large TVs: 14 days; default: 30), correct by construction. *The harness scored this run 2/3 only because its verdict-extractor couldn't parse "the return window has closed" — the stated verdict itself matched ground truth. The BEFORE run's 1/3 failed on substance.The evaluation harness makes this before/after reproducible: it runs the same three return questions with routing off and on, grading each stated verdict against ground truth computed directly by the tool. The lesson generalizes far beyond this project: if a sub-task has a deterministic solution, route around the LLM — don't prompt it into pretending to be a calculator.
reset wipes it instantlyforget me deletes the profile (retention rule)The payoff shows in demo D5. The customer says "that first order" without repeating any id — the agent resolves it to NM-10077 from memory and reasons over the shipping-delay policy:
Adaptation is deliberately explainable rather than clever: feedback signals (/up, /down + comment) are stored PII-redacted, and a transparent rule — two or more "too wordy" complaints → concise style — writes a preference into the long-term profile that reshapes the system prompt on every later turn. An auditor can trace exactly why the agent's tone changed, which is not something you can say about fine-tuning.
Three system-prompt variants were run against a fixed 5-question test set with identical agent configuration (memory off, fresh session per variant):
"You are a support assistant… answer the question."
+ Fluent, far beyond the rule-based baseline
− Fabrication risk: uncited policy claims, over-promising, no escalation path
Adds "say so instead of guessing", length limits, no-PII rule.
+ Fewer fabrications, honest uncertainty, consistent tone
− No citations → not auditable; unknowns dead-end at "I'm not sure"
Answer ONLY from context, cite policy ids, explicit "I don't know → escalate" path, refusal rules, few-shot example.
+ Auditable citations, correct category exceptions, actionable escalation
− ~350 extra tokens/turn (≈ +15–20% cost); occasionally over-conservative
The most interesting result was a new failure mode the comparison surfaced: V3 initially cited policy ids even when retrieval returned nothing — the id was leaking from the few-shot example. That drove the NO RELEVANT POLICY FOUND marker and the unknown-answer guard. And on "Do you deliver to the UK?", V1 and V2 answered correctly while strict V3 deferred to a human despite having the policy in context. That over-refusal is logged as a known trade-off: for a support agent, failing safe beats fabricating — but it's a real cost, on the roadmap to fix.
The eval harness runs 10 scenario cases, each declaring expected behaviour flags — should the agent refuse? escalate? call a tool? retrieve? Pass/fail is graded automatically from the agent's own turn metadata, not by eyeballing prose. Safety cases are deterministic pre-LLM, so those results reproduce across runs regardless of temperature.
| Case | Category | Expected checks | Result | Latency |
|---|---|---|---|---|
| T1, T10 | tool_use | correct tool selected from natural language | 2/2 PASS | ~3.0–3.7s |
| T2, T6, T8 | policy_grounding | retrieval fired & grounded the answer | 3/3 PASS | ~2.5–3.3s |
| T3, T7 | safety_refusal | fraud & cross-customer data refused | 2/2 PASS | 0.1ms |
| T4, T5, T9 | safety_escalation | chargeback / unauthorized / human-request escalate | 3/3 PASS | 0.5–0.7ms |
Every log line passes redact_pii (emails, phones, cards, SSNs, addresses) before being written — the JSONL logs carry latency, tools, sources, safety action, and prompt version, enough for dashboards and audits with zero personal data.
Everything LangChain or CrewAI would provide here exists in the codebase, directly and inspectably: chunking → embeddings → cosine store (≈ vector store + retriever), OpenAI function calling + guarded ToolRunner (≈ agent executor), rolling window + persisted profile (≈ conversation memory), strict-JSON planner with heuristic fallback (≈ router / plan-and-execute).
At this scale, framework-free was the better engineering choice for four reasons. Debuggability — when an incident review asks "why did it say that?", every prompt, retrieval score, and tool decision is visible in code I own. Safety placement — frameworks route everything through the model loop; here, unsafe requests never reach the model at all. Dependency surface — five small dependencies instead of a framework's transitive tree. Cost of change — swapping the vector store for FAISS or the provider for Anthropic is a one-file change behind stable interfaces.
And an honest boundary: past ~50 KB documents, multi-agent orchestration, or a team standardized on LangSmith-style tracing, I'd reach for the framework.
The roadmap, in priority order: an LLM-as-judge grading pass for answer quality (the harness checks behaviour, not prose); an adversarial safety suite — paraphrase attacks against the refusal patterns plus prompt-injection probes; retrieval re-ranking and query rewriting once the KB grows; a feedback→prompt-review pipeline beyond style adaptation; and multi-order disambiguation ("which order do you mean?").
If the project has one thesis, it's this: an agent's value comes as much from what it refuses to do as from what it does. This one resolves the high-volume 70% of contacts with cited answers and deterministic policy math, converts the risky 30% into well-formed, prioritized human tickets — and every claim above is backed by a regenerable artefact in the repo's evidence/ folder.
Thanks for reading. This capstone concludes my Agentic AI program at IIT Madras — the full pipeline (mock mode included, no API key needed) is reproducible from the repository.