Four years after LLMs entered the mainstream, the single most common mistake I see architects make is spending most of their decision energy on the model. Which frontier model? GPT or Claude or Gemini? The model choice matters — but it is one decision out of roughly fifteen, and it is far from the most consequential one.

Building an LLM-powered system in 2026 is an architecture decision made across a stack of competing frameworks, each solving a well-posed problem at a specific layer. I spent several months mapping that landscape as a practitioner — the result is a 73-page whitepaper I call the Framework Atlas. This post distils it into the five things I think every architect, engineer, and senior IT leader should know before picking a single tool.

The Stack Has a Shape

There is no single AI stack, but there is a canonical shape. Every non-trivial production LLM system — whether a support chatbot, a document search engine, or a multi-agent workflow — is a composition of six layers:

Application layer. The surface your user interacts with. LangChain is the default; Semantic Kernel is the Microsoft-native choice; CrewAI leads when the app itself is agentic.

Agent layer. When a single LLM call is not enough — when the system needs to plan, call tools, or coordinate among multiple agents — this layer provides the loop. LangGraph is the most production-grounded option in 2026.

Data / retrieval layer. The memory of your system. LlamaIndex leads on orchestration; Weaviate, Pinecone, and Chroma compete at storage, each tuned for a different operational profile.

Model layer. The foundation models themselves. This layer is increasingly commoditised. The most important design decision here is not which model you start with — it is whether you can swap it without rewriting the layers above.

Serving / inference layer. How you turn a model into an endpoint. vLLM dominates throughput-bound workloads; BentoML packages models into clean APIs for teams that want to think about models, not infrastructure.

Infrastructure layer. Kubernetes, Docker, cloud, on-prem. Every framework choice depends on where you can actually deploy.

Wrapped around all six layers are three concerns that have become non-negotiable since 2024:observability and evaluation,fine-tuning and training, andguardrails and safety. If your system design has no answer for any of these three, it is under-designed. Ignoring them does not eliminate the risk — it just defers it until something breaks in production.

The Abstraction Trap

Every framework in the atlas is catalogued against eleven attributes, but the one architects under-weight most consistently isabstraction level — how much code you write versus how much the framework decides for you.

LangChain’s high abstraction makes the first demo fast and the tenth production fix slow, because you are debugging through someone else’s default decisions. FAISS’s low abstraction costs more lines but yields fewer surprises at 3am.

The operational signal:match abstraction to team seniority. Junior teams over-value high abstraction; senior teams over-value low. A mixed team benefits from a medium default — and from making the choice explicitly rather than defaulting to whatever has the best GitHub star count.

Decision Heuristics That Actually Hold

Rather than optimising at each layer independently, the atlas maps common requirements to preferred framework combinations. These are the ones I have found most durable in practice:

Two things stand out from this table. First, LangChain and LlamaIndex are not competitors — they compose cleanly, with LangChain at the application layer and LlamaIndex at the retrieval layer. Second, local inference is no longer an edge case. Ollama plus a Llama-3-class model is a realistic production option for regulated industries where data sovereignty is a hard constraint.

Agents Moved to Production — With Guardrails

In 2023, autonomous agents were mostly demos. By 2026, they are in targeted production use: triage, routing, research synthesis. What changed is not the models — it is the frameworks.

LangGraph’s state-machine model gives agents deterministic control flow: you declare states, transitions, and retry policies explicitly. AutoGen models multi-agent systems as conversations, which makes it remarkably expressive for critique-revise loops and planner-executor separations. The practitioner heuristic:for production agents, LangGraph. For multi-agent conversations, AutoGen. For lightweight document workflows, CrewAI.

The critical note:never deploy autonomous agents in production unless the failure cost is bounded. The agent should draft; a human should approve. The pattern that ships is almost always a hybrid — autonomy where the stakes are low, escalation where they are not.

Guardrails have crossed from afterthought to critical infrastructure in the same period. Prompt injection is the new SQL injection. Every production system needs an input guard, an output guard, and a policy layer between them. The minimum viable defense in 2026 is: input guard → LLM → output guard. Anything less is operating without a seat belt.

The 2026 Outlook: Three Trends Worth Designing For

Agents are becoming the compile target. LangGraph, AutoGen, and CrewAI are converging on a common abstraction — a loop over an LLM with tool use and state. Expect a future that looks like the deep learning layer in 2018: multiple frontend frameworks, one common runtime. Design your agent layer to be swappable.

Retrieval is eating search. Elasticsearch, Postgres, and OpenSearch all ship vector indexes now; Weaviate and Pinecone ship BM25. The primitives have converged. The differentiator is no longer features — it is operational maturity and the team’s ability to run the infrastructure. Hybrid retrieval (vector + keyword) is the production-safe default.

Guardrails are becoming infrastructure. Today they are a library you bolt on. In two years they will be a runtime — prompt injection detection, PII scrubbing, and policy enforcement applied by default to every model invocation, the way CORS and auth middleware is applied to every HTTP request today. Get ahead of this by treating your guardrails layer as critical infrastructure now, not as a compliance checkbox later.

A Practitioner’s Closing Note

Frameworks age faster than architectures. The stack shape you design today — application, agent, retrieval, model, serving, infrastructure — will still be valid in three years. The individual framework boxes you fill it with probably will not be. The single most important design invariant isswappability at each layer. Make the layer interfaces clean, keep the framework-specific code thin, and you will be able to move when the landscape shifts — and it will.

The full Framework Atlas (v4.0, April 2026) covers all ten framework categories in detail, including comparison tables, maturity radars, cost and latency envelopes, and four reference architectures with working code. It is available below.

Download the Framework Atlas — Building with LLMs v4.0 (PDF)

Vision models, language models, and most other generative systems are confident-but-wrong some non-trivial fraction of the time. The instinct is to fix that with better prompts, bigger models, or smarter agents. The cheaper move is usually to add a small structured review seam — a thirty-second checkpoint where a human can glance, correct, and move on.

This post is the case study for one such seam, dropped into a build I needed for myself. Of 35 garden photos handed to a vision model,74% came back with correct first-pass labels. After thirty seconds editing a CSV,97% were acceptable to publish. Total API cost:$0.18. Total inference time:~74 seconds at 2.1 sec/photo ongpt-4o-mini. The CSV was the highest-leverage code in the project — and it isn’t really code.

Here’s the story.

The annoyance

It was a Saturday afternoon in early March. I’d come back from a walk around the garden with thirty-five photos on my iPhone — bottlebrush in full red, honeysuckle dripping with rain, a lilly-pilly cluster doing its outrageous pink thing, and at least one inexplicable shot of an old railway station I’d passed on the way home.

I wanted to post a handful of them with consistent little hashtag labels —#bottlebrush,#honeysuckle,#flower — burned into the corner like a quiet caption. Not a watermark, not a filter, just a small readable pill that says “this is what you’re looking at.”

What I didn’t want was to open each HEIC in Preview, draw a text box, fiddle with the font, export, repeat thirty-five times. So I did the only reasonable thing: I wrote a small Python tool that does it for me.

The shape of the pipeline

┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐
│ Folder │──▶│ Vision │──▶│ CSV │──▶│ Apply │──▶│ Tagged │
│ photos │ │ provider │ │ review │ │ + pill │ │ output │
└──────────┘ └──────────┘ └──────────┘ └──────────┘ └──────────┘
↑
human-in-the-loop seam
--mode propose: folder ─▶ vision provider ─▶ CSV
--mode apply: CSV ─▶ render + pill ─▶ tagged output

The minimal product was easy to describe. Point the script at a folder. For each image — HEIC, JPG, PNG, whatever the iPhone or my camera roll throws at it — open it, figure out what’s in it, draw a small rounded hashtag pill into the bottom-right corner, save the result to atagged_output/ subfolder. No watermark across the centre of the image, no filter or colour grade, no destructive edit to the original, and no making me choose the label by hand when a vision model can have a decent first guess.

That last point is where the design got interesting.

The seam

You could write this as a single command: walk the folder, ask the model, render the tag, done. I tried that first. The first-pass run produced a folder of beautifully tagged images, about a quarter of which were wrong in some quietly maddening way — a daisy called#flower, a fern called#leaves, the railway station called, charitably,#station.

So the script runs in two passes.

--mode propose opens each image, hands it to the vision model, and writes a CSV with five columns:

image_path, label, score, suggested_tag, final_tag

final_tag is initialised tosuggested_tag, but the whole point of the column is that you can edit it. Open the CSV, glance down the list, fix anything obvious —flower becomesmorning_glory,leaves becomesbamboo — save, close. On this batch,9 of 35 rows needed editing (a daisy, the railway station, two ferns, the bamboo, and four generic-flower fallbacks). A thirty-second pass.

--mode apply then reads the CSV row by row and renders the tag using whatever’s infinal_tag. The CSV is the human-in-the-loop seam. It is much cheaper than re-running inference, and it catches the cases where the model was right about the genus but wrong about the species, or just wrong.

Three providers, one interface

I didn’t want to commit to one vision model — the price/quality trade-offs are too lively right now. The script supports three providers behind one interface, picked via--provider local|openai|xai.

Local CLIP. HuggingFace’sopenai/clip-vit-large-patch14 against a fixed candidate list. Free, offline,~0.4 sec/photo on an M3 Pro. The cost is breadth: anything outside the candidate list collapses to the nearest match. CLIP doesn’t know what a bottlebrush is unless I tell it the word.

OpenAI.gpt-4o-mini by default, with an opt-in--high-accuracy flag that retries low-confidence cases (under 0.72) ongpt-4o.~2.1 sec/photo, ~$0.18 for the 35-photo batch. Open-ended labels — howbottlebrush,honeysuckle,fern, andberries ended up in the CSV rather thanflower,flower,leaves,fruit.22% of the batch tripped the retry threshold and went togpt-4o.

xAI Grok. Same OpenAI-compatible client, pointed atapi.x.ai withgrok-2-vision-latest. Useful if you’re already on the x.ai stack or want a different model family’s vote.

The mental model: local CLIP for batch-of-a-hundred-photos-on-a-flight, OpenAI as the daily driver, and the high-accuracy retry for exactly the case where the model says “flower” with 0.55 confidence and I want it to look harder before I have to.

The blue morning glory below is what generic labels look like in practice — still a decent fallback, just unspecific. The model wasn’t wrong; it just wasn’t curious.

Two small touches

Two design choices are the difference between “the script works” and “the output looks intentional.”

Style-aware contrast. The pill needs to be readable on both a bright sky and dark foliage. The script crops the bottom-right region of the image, measures the mean luminance using the standard Rec. 709 weights, and flips the colour scheme above or below a threshold:

defstyle_aware_colors(img):w,h=img.sizecrop=img.crop((int(w*0.68),int(h*0.80),w,h))r,g,b=ImageStat.Stat(crop.convert("RGB")).mean[:3]luminance=0.2126*r+0.7152*g+0.0722*bifluminance<140:return(255,255,255,245),(0,0,0,95)# white text, dark pillreturn(0,0,0,245),(255,255,255,95)# black text, light pill

Eight lines of PIL. In this batch every photo sampled dark — gardens are mostly green and shadow in the corner — so every output got the dark pill. The bright-pill branch is still there, waiting for a photo with sky or a light wall in the corner.

Save with fallback. HEIC writes occasionally fail for reasons that aren’t worth diagnosing in a personal tool. The save function tries the original format first; if PIL throws, it quietly drops to JPEG with the same filename stem. Eight more lines. On this batch,3 of 35 fell back to JPEG. Without the fallback those three would have been a stack trace and a half-finished folder. With it, thirty-five of thirty-five made it through.

What I’d add next

Multi-tag support, so a photo can be#lorikeet #bottlebrush when the bird showed up in the bottlebrush. EXIF preservation through the round-trip — right now PIL strips most of the metadata, which I don’t love. A tiny review UI to replace the CSV step, either a Tkinter window or a one-page localhost app. Smarter candidate lists for the local provider, scoped by season or geography — Sydney summer has a different vocabulary than European spring.

None of these are urgent enough to displace “the script already does what I wanted.”

Closing observations

Three lessons that generalise beyond this script.

Human-in-the-loop is cheap and underrated. The CSV seam between propose and apply takes thirty seconds per batch and saves me from confidently wrong outputs. For any task where a model is confident-but-wrong some non-trivial fraction of the time — RAG, codegen, moderation, enterprise copilots, agentic workflows — a structured review step pays for itself almost immediately. The CSV doesn’t have to be elegant. It has to exist.

Pluggable providers are worth the small abstraction tax even on personal tools. I went from local CLIP togpt-4o-mini to Grok in the space of one afternoon without rewriting the rendering code. The interface is(client, model, image) → (label, score) and that’s it. Once you’ve paid that cost once, you can keep up with a fast-moving model market essentially for free.

Small touches decide whether a script feels finished. Luminance-aware contrast and a save-format fallback don’t change what the tool does; they change how the output reads.

The model wasn’t the product. The seam was.

A short reel of the tagged photos in the wild:Instagram story.

The model is what writes the email. The embedding is what finds the one you wrote last March.

Most modern AI systems are built from two fundamentally different mechanisms, and most confusion about what AI “is” comes from conflating them. LLMs aregenerative: tokens in, tokens out, with the output shaped by the prompt and the sampling settings, varying every time you ask. Embeddings aregeometric: a deterministic mapping from a word or sentence to a fixed vector, where comparisons are positional and identical input always produces identical output. Both are essential. Both are old enough to be uncontroversial. Most useful systems combine them.

What follows is theWeek 6 Graded Mini Project of theIITM Pravartak Professional Certificate Programme in Agentic AI and Applications, used here as a lens for both mechanisms across five hands-on exercises.

The two paths, side by side

The header image above shows the contrast in one frame. The LLM path is a loop with sampling — non-deterministic by design, behaviour controlled by temperature, top-p, and prompt structure. The embedding path is a one-shot lookup followed by a geometric comparison — deterministic, fast, stable.

That single distinction tells you which mechanism to reach for. If the answer needs to be written, generated, synthesized, or improvised, you want the LLM. If the answer needs to be found, ranked, deduplicated, clustered, or routed, you want embeddings. Most production systems use both because most real problems are some combination of “find the right context” and “say something useful about it.”

A quick decision table to anchor the rest of the article:

The exercises below show why each row works the way it does.

Exercise 1: Text generation reveals prompt and sampling sensitivity

Section A1 loadeddistilgpt2 through the Hugging Facepipeline API and generated three continuations of the same prompt:

generator=pipeline("text-generation",model="distilgpt2")generator("AI is transforming industries by",max_new_tokens=40,num_return_sequences=3,do_sample=True)

Three continuations came back from the same model, the same prompt, the same call:

“AI is transforming industries by using science to bring people together with a greater understanding of the importance of science. The new book takes an approach to both science and technology, allowing people to focus more more effectively on the basics and to…”

“AI is transforming industries by replacing the manufacturing sector with a manufacturing sector that can be turned into a manufacturing and IT sector by creating new jobs and creating new jobs. The new jobs and investment in the next decade will help spur growth…”

“AI is transforming industries by creating a new, faster, and more attractive way of generating capital and creating jobs for both the United States and Europe. This is an effective new way of doing this.”

Three different stories. None of which the model “knew” — it just produced plausible-sounding next tokens under stochastic sampling. Notice the repetitions (“manufacturing sector with a manufacturing sector”), the loops (“more more effectively”), the empty filler (“a new, faster, and more attractive way of generating capital”). DistilGPT-2 is a small model — these are the artefacts of a system that’s good at local fluency but doesn’t have a strong forward plan.

The headline insight: LLM outputs are statistical, prompt-sensitive, and unrepeatable unless you fix the seed. The same prompt can give you variety (a feature when brainstorming) or drift (a bug when consistency matters).

Exercise 2: Tokenization is where the abstraction begins

This is the section to slow down on. Take the sentence:

“LLMs are powerful tools for natural language understanding.”

A human reads eight words. The model sees ten tokens.

After BPE (Byte-Pair Encoding) with the DistilGPT-2 tokenizer:

['LL', 'Ms', 'Ġare', 'Ġpowerful', 'Ġtools', 'Ġfor', 'Ġnatural',
'Ġlanguage', 'Ġunderstanding', '.']

The stringLLMs doesn’t appear in the model’s vocabulary as a single unit, so it is split intoLL andMs. TheĠ prefix encodes “preceding space” — that’s how BPE preserves word boundaries without a separator character. The period gets its own token.

The mismatch betweenwhat a human reads andwhat the model processes has real consequences:

• Cost is per token, not per word. API billing, latency, and rate limits are all token-denominated. A 1,000-word prompt to a frontier model may bill at 1,300–1,500 tokens depending on language.
• Context windows are token windows. A 4,096-token context holds roughly 3,000 English words. Much less for code (whitespace and symbols inflate counts), much less again for languages with poor vocabulary coverage in the tokenizer.
• Rare strings behave oddly. Brand names, technical acronyms, foreign words, internal jargon — anything outside the trained vocabulary gets fractured. Model behaviour around those fractures is harder to predict, and prompt sensitivity often hides at this layer.
• The same string can tokenize differently with leading whitespace."king" and" king" are different token sequences. That’s why pasted prompts sometimes produce subtly different outputs than typed ones.

Tokenization is the lowest layer of the LLM stack and the one most engineering conversations skip. If you’re tuning prompts and getting unstable behaviour, the first place to look is what your input looks likeafter the tokenizer touches it, not what it looks like in your editor.

Exercise 3: Prompts shape what you get

Section B ran three task-shaped prompts through the same generator, withtemperature=0.8 andtop_p=0.95:

• Summarization — explicit instruction with a 30-word cap.
• Q&A — structured format withQ: andA: markers.
• Creative — open-ended request for a 4-line poem about AI.

The summarization output respected the spirit of the constraint but drifted past 30 words on most runs — DistilGPT-2 is small enough that hard length control isn’t reliable even with explicit instructions. The Q&A output, asked for the capital of Japan, returnedI believe... — the model hedged. A larger model would say Tokyo confidently; a small model produces statistically plausible Q&A-shaped text without strong factual grounding. The creative prompt produced varied and stylistic continuations, but with the lowest grounding: fluency over precision.

Structure compresses the output space the model is sampling from. Vagueness expands it. That single sentence is most of what “prompt engineering” actually is — the rest is technique.

Exercise 4: Word embeddings encode semantic geometry

Pivot to the other mechanism. Section C1 loadedGloVe vectors (glove-wiki-gigaword-50 — 50 dimensions, trained on Wikipedia and Gigaword) via Gensim, then asked for the five nearest neighbours of three words:

There is no generation here. Each word is mapped to a fixed 50-dimensional vector, and the “nearest neighbours” are the words whose vectors sit closest in that space by cosine similarity. The geometry was learned by training on co-occurrence — words that appear in similar contexts end up in similar positions. That’s whyking andprince are nearest neighbours, whyqueen pulls inelizabeth (the corpus has plenty of references to Queen Elizabeth), and whydiamond cleanly resolves to a jewellery cluster.

The classicking − man + woman ≈ queen analogy works in this same space; the lab didn’t run it, but the geometry is there. Embeddings don’twrite anything — theyplace things near other things. That single property is what makes them the backbone of semantic search, retrieval, deduplication, and recommendation.

Exercise 5: Sentence similarity from averaged word vectors

Section C2 extended the geometry to sentences. Five short sentences across two topics — AI/ML and jewellery — were averaged into sentence vectors (mean of their word vectors, with simple lowercase tokenization), then compared with cosine similarity.

Plotted in 2D via multidimensional scaling on the cosine distances, the clustering is unambiguous:

The numerical version:

Within-cluster pairs sit at 0.84–0.88. Cross-domain pairs sit at 0.50–0.62. The grouping is exactly what you’d want a retrieval system to do.

Three caveats worth naming, because they explain why modern retrieval doesn’t actually use GloVe averages:

• Averaging discards word order. “Dog bites man” and “man bites dog” produce identical sentence vectors. For most retrieval that’s tolerable; for anything where syntax carries the meaning, it isn’t.
• Transformer encoders fixed this. Models like BERT, RoBERTa, and their descendants producecontextual embeddings — each token’s vector depends on the tokens around it. Pool those across a sentence and you get a representation that respects word order and disambiguates polysemy.
• Sentence-BERT and friends made it production-grade. SBERT (and successors like OpenAI’stext-embedding-3, Cohere’s embeddings, Voyage, etc.) trained encoders specifically for sentence-level similarity. That’s the difference between “the demo works on five sentences” and “you can index a million documents and search them in milliseconds.”

GloVe averaging is a baseline. It’s the right baseline to start with, because it lets you see the geometry without the architecture getting in the way. Production systems start from this picture and replace the lookup step.

When both mechanisms meet

The final exercise sits at the intersection.distilbert-base-uncased-finetuned-sst-2-english is a transformer encoder (an embedding model under the hood) with a classification head fine-tuned for sentiment. Run it on three workplace-themed inputs:

The third row is the interesting one, and it’s worth unpacking because it points at a problem that turns up in every enterprise deployment of pretrained models.

“Acceptable but needs tuning” is, in workplace context, alukewarm-positive — closer to “approved with caveats” than “this is bad.” The classifier scored it NEGATIVE with 0.9898 confidence. Three things are happening at once:

• Domain mismatch. The model was fine-tuned on SST-2, which is movie reviews. “Needs tuning” reads negative there. In an engineering team’s language, “needs tuning” is constructive — the same words have different sentiment loadings in different domains.
• No calibration on workplace text. The score is 0.9898 — extreme confidence — for what should be a borderline case. Pretrained classifiers tend to be miscalibrated on out-of-distribution inputs: they’re not just wrong, they’re confidently wrong. Calibration techniques (temperature scaling, Platt scaling, conformal prediction) exist for exactly this.
• Weak supervision is the practical fix. When you can’t fine-tune (no labelled data, no budget, no time), the durable answer is to treat the classifier as one signal among several — combine it with rules, keyword filters, or a second model — rather than trusting any single number above the threshold.

Architecturally, the lesson generalises across all three Section D variants. Generation is “embedding + decoder loop.” Classification is “embedding + categorical head.” Retrieval is “embedding + cosine.” Same underlying mathematical object, different output shapes. The architectural choices around the embedding determine what the system does — and where it fails when you take it out of the domain it was trained on.

Closing observations

Three things that generalise beyond this lab.

Tokenization is where most LLM cost and quirks actually originate. It’s the lowest layer of the stack and the one most engineering conversations skip. If you’re tuning prompts and getting unstable behaviour, the first place to look is what your input looks like after the tokenizer touches it.

Embedding-based similarity is older, cheaper, and more deterministic than people remember. Before reaching for an LLM call to compare two pieces of text, embed them and compute cosine. It’s milliseconds, free, and stable. A surprising fraction of “AI features” are really embedding lookups with a confidence threshold.

Generation and similarity sit next to each other. They are not competitors. RAG is the obvious example — embeddings retrieve, the LLM generates the answer grounded in what was retrieved. TheWeek 15 RAG chatbot post is what these two mechanisms look like wired together for production.

One predicts words. One maps meaning. Knowing which one to reach for is most of the job.