AJAY WALIA
ONDEVICE.AI · BUILD NOTES · PART 2
PART 2 OF 2#local-first#architecture#llm#product#ollama

OnDevice 2.0 — The Shell Is the Product

Part 1 described a private AI workspace. This is what happened next: seventeen days, 240 catalogued changes, and one idea that kept surfacing until it reorganised everything — the model is a component, and the thing you actually build is the shell around it.

AW
by Ajay Walia
OnDevice.ai · six turning points, and what's still unfinished
The whole of Part 2 in one loop: the chassis never moves, the core is swappable, and the lines from outside still stop at the edge. Generated with Grok Imagine.
TL;DR

On 10 August, OnDevice.ai was a chat box with three document generators bolted to the side and a model name in an environment variable. By 26 August it was a shell: every message enters through one router, leaves through one of eleven named pipelines, and the model underneath is something you pick from a menu — local GGUF, local MLX, a cloud key you paste in, or your own vLLM endpoint. Nothing about the change was a rewrite. It was six decisions, each of which made the next one obvious.

01Latency was a UX problem, not a model problem 02The model became a socket, not a setting 03One door, eleven rooms — the request router 04Documents grew up in three stages 05Grounding had to earn its keep 06The chrome got out of the way 07What still isn't done

The shape of it

Seventeen days, eleven build days

Every material change went into a changelog entry and a catalogue row with a stable ID. That discipline was a note-to-self at the time; in hindsight it's the only reason this article can be written from evidence rather than memory.

240
Catalogued
entries
11
Build days
10→26 Aug
11
Router
categories
7
LLM
backends
4
Named
specialists
35
Test
files

Of the 240 entries, 238 are shipped, one is partial and one is pending. That ratio is less impressive than it sounds — it mostly means I didn't log things until they worked. The genuinely unfinished work is in the last section, and there is more of it than a status column admits.

01 10 → 12 AUGUST

Latency was a UX problem, not a model problem

A 27-billion-parameter model on a laptop is not slow in any absolute sense. It is slow compared to a web app, which is the only comparison a user actually makes. Ask for a deck and nothing visible happens for forty seconds. Every tester I put in front of the early build did the same thing at around the fifteen-second mark: they clicked again. Then they said it had crashed.

It had not crashed. It was working perfectly, in silence, which turns out to be indistinguishable from failure.

Problem

Local inference takes tens of seconds. With no feedback, users read silence as a crash — and clicking again queued a second job on an already-busy machine.

What changed

Anthropic-style SSE lifecycle, a phased progress panel with step N/M, an elapsed timer, keepalive frames every ~2s, Send-becomes-Stop with a real server-side cancel, and a browser notification when the tab is in the background.

Why it mattered

It stopped being a slow app and became a working app. Nothing about the model changed. The perceived product changed completely.

The OnDevice.ai progress panel showing 'Sharpening the request, then researching… (2/5)' with a 10 second elapsed timer
Phase 2 of 5, ten seconds in, with a live timer and a Stop button. The status text names the actual phase rather than showing a generic spinner. The base document pipeline runs preparing → drafting_spec → generating_bytes → validating → finalizing; this is the research-assisted deck path, which prepends its own refining_ask stage — hence "sharpening the request, then researching".

The detail that mattered most was the smallest: naming the phase. "Loading" tells you nothing. "Sharpening the request, then researching… (2/5)" tells you the machine has a plan, knows where it is in that plan, and expects to finish. The elapsed timer does the rest of the work — it converts an unbounded wait into a measurable one.

Explainer · Server-Sent Events

SSE is a one-way stream from server to browser over ordinary HTTP. The server holds the connection open and writes text frames as work completes; the browser reads them as they arrive. Unlike WebSockets it needs no protocol upgrade, no special proxy configuration, and it reconnects on its own.

Two things make it work in practice. First, keepalive frames: if nothing is sent for a while, intermediate proxies and the browser both start to assume the connection is dead — so the server emits a comment frame every couple of seconds purely to say "still here." Second, buffering headers: Cache-Control: no-cache, no-transform and X-Accel-Buffering: no, without which a reverse proxy will helpfully collect your tokens and deliver them in one lump at the end, defeating the entire exercise.

Cancellation is the other half. The browser's AbortController tears down the fetch, but that alone only stops the client listening — the server happily continues burning GPU on an answer nobody will see. A server-side cancel_check polled between pipeline steps is what actually stops the work.

Catalogue: C-003, C-007 to C-012, C-015 to C-020 · Tests: Q-004, Q-006

02 10 → 23 AUGUST

The model became a socket, not a setting

This is the turning point that gave the article its title, and it arrived by accident.

The first local build had OLLAMA_MODEL in a .env file. Changing models meant editing a file and restarting the server. That was tolerable while there was one model. Then I wanted to compare Qwen against Glimmer. Then I needed a vision model for images, and a coder model for code, and the 27B was too slow for a one-line question that an 8B would answer instantly.

The obvious fix was a dropdown. The interesting consequence was what the dropdown implied: if the model can change between turns, then nothing in the rest of the system is allowed to assume which model it is talking to. Every capability has to be probed, not assumed. That constraint is what turned a chat app into a shell.

The moment the model can change, the model stops being the product. What's left — the router, the pipelines, the validation, the boundary — is the thing you actually built.
THE SHELL — FIXED Request router Eleven pipelines Four specialists Intermediate specs Structural gate Workspace + files Trust boundary · nothing leaves unless you ask it to THE SOCKET — SWAPPABLE Ollama GGUF · :11434 LM Studio MLX · :1234 Cloud keys 4 providers Custom URL vLLM · SGLang one OpenAI-compatible interface · everything above the socket is unaware of which core is in it

Seven backends now feed the same socket. Two are local runtimes on this machine, four are cloud providers unlocked by pasting a key, and one is whatever OpenAI-compatible server you point it at. The list of models is discovered live from whichever backend is active, merged with a preset catalogue, and tagged by format.

The OnDevice.ai settings popover showing temperature, reasoning effort, context window, search provider, cloud API key fields and a custom OpenAI-compatible URL
Everything that makes the model a component, in one popover: sampling temperature, extended-thinking effort, context window, search provider, four cloud key slots, and a custom vLLM/SGLang endpoint. The keys are written to disk at mode 0600 and never returned to the browser in full.

The policy that keeps it honest

One decision here looks like a limitation and is actually the thing that makes it work: Ollama serves GGUF, LM Studio serves MLX, and they do not share a model directory. It is tempting to point OLLAMA_MODELS at the LM Studio library and have one pool of weights. It doesn't work — the formats and runtimes are genuinely different — and pretending otherwise produces failures that look like model quality problems rather than plumbing problems.

LayerLocalCloudWhat the shell does about it
RuntimeOllama (GGUF) · LM Studio (MLX)OpenAI · xAI · Anthropic · OpenRouterSame OpenAI-compatible client for all seven
VisionProbed via /api/show capabilitiesAssumed from provider docsNever sends pixels to a model that reports text-only
Contextnum_ctx, 4k → 128kProvider default1M YaRN needs vLLM — routed to Custom URL
ThinkingOllama native think: true|falseProvider-specificSeparate reasoning_delta stream, collapsible panel
Cachingkeep_alive=30m + prefix cachecache_control: ephemeralCache hit/written shown under the reply
KeysNone neededllm_secrets.json, mode 0600Server-side only — never NEXT_PUBLIC_*
Explainer · GGUF, MLX and quantisation

GGUF is the file format used by llama.cpp and everything built on it, including Ollama. MLX is Apple's array framework for Apple Silicon, with its own weight format that uses unified memory more directly. Same model, different container, different runtime — and no, you cannot point one at the other's folder.

Quantisation is what makes a 27B model fit on a laptop at all. Full weights ship at 16 bits per parameter — about 54 GB for 27B. Q4_K_M stores most weights at roughly 4 bits with the more sensitive layers kept at higher precision, landing somewhere near 16 GB. You lose a little accuracy. You gain the ability to run it.

The reason everything speaks OpenAI-compatible HTTP is not deference to OpenAI — it's that /v1/chat/completions became the de facto lingua franca. Ollama, LM Studio, vLLM and SGLang all expose it. One client, seven backends, and a new runtime is a URL rather than an integration.

Catalogue: L-002 to L-036 · Decisions: hybrid GGUF/MLX split (11 Aug), cloud keys on disk (23 Aug)

03 17 → 23 AUGUST

One door, eleven rooms — the request router

By mid-August the chat handler had grown a thicket of if-statements. Is this a document request? Does it mention a PDF? Are there images attached? Does it look like a summarise? Each check had been reasonable when added, and collectively they were unmaintainable and untestable.

The fashionable answer in 2026 is an agent: give the model tools and let it plan. I deliberately did not do that, and the reason is the same reason the documents pipeline works — an intermediate you can inspect beats a decision you have to trust.

Problem

Intent detection had become nested conditionals with overlapping conditions. Adding a capability risked silently breaking an unrelated one, and no test could assert "this message goes here".

What changed

Every message hits one router that classifies it into exactly one of eleven closed categories, then hands off to exactly one named pipeline. Rules run first; a short local-model JSON classify runs only when the rules return ambiguous.

Why it mattered

The route is recorded on the message as extra.route — category, source and pipeline. When something goes to the wrong place you can see exactly where the decision was made, and write a test that pins it.

MESSAGE IN User message Router rules first local classify if ambiguous ambiguous → one JSON classify (local model, not cloud) ELEVEN CLOSED CATEGORIES · ONE PIPELINE EACH complete_interview document_gen pdf_then_answer summarize fact_check research vision csv_analyze browser file_edit chat SPECIALISTS vision · pixels only research · 2 hops + facts pack browser · Firefox, allow-listed text editor · workspace files a pipeline may call one specialist a specialist may not call another no free-form planner · no sub-agents · no dynamic tools · the route is recorded on the message

The specialists deserve a note, because "specialist" in this codebase means something narrower than the industry usually means by it. A specialist is not an agent. It has no planner, no tool discovery, and no ability to decide it needs a different specialist. The browser specialist visits only http(s) hosts the user named in the message — plus an optional environment allow-list — in an isolated Firefox profile, with downloads off and a Stop button that kills the job. It returns a screenshot and the page text. That is the entire contract.

The same restraint governs model selection. After routing, the shell may swap in a specialist model for that category — LLaVA for vision, a coder model for code — but only from a fixed allow-list, only if it's already installed, and only for that turn. Then it restores the home model. The status line says so out loud: "Using llava for vision; then back to qwen3.8-27b."

Explainer · Why routing beats planning here

An agentic planner decides at runtime what steps to take. That's powerful when the task space is genuinely open, and it's a liability when it isn't. The cost is that every failure is a reasoning failure — the model chose badly, and reproducing the choice requires reproducing the exact context.

A router with a closed category set converts that into a classification failure, which is a completely different animal. It's inspectable (the category is written on the message), testable (you can assert message → category), and cheap to fix (add a rule). You give up the ability to handle requests nobody anticipated. For a workspace with eleven kinds of work, that trade is almost free.

The hedge is the ambiguous escape hatch: when rules can't decide, one short JSON classification runs on the local model. Even the fallback stays inside the boundary.

Catalogue: C-046 to C-048, C-057, A-011, A-012, L-035 · Tests: Q-019, Q-020, Q-024

04 14 → 23 AUGUST

Documents grew up in three stages

Part 1 argued that document reliability comes from schema-validated specs and deterministic generators, never from asking a model to emit binary Office files. That bet held. What it did not do was make the documents any good.

On 14 August, "Create a 5-slide presentation on Q1 infrastructure goals" produced two empty slides that cloned the command string back at me. The file was structurally perfect. It opened in PowerPoint. It was worthless.

v1SkeletonDefault PowerPoint placeholders, one bullet per topic, no arc. Opens reliably, says nothing.
v2Usable draftDesigned layouts — navy header, accent rail, card grids, two-column panels. The model authors 4–5 concrete bullets per section before generation. Looks like a deck.
v3Credible executive draftA forced narrative arc, quality gates that reject placeholder language, anti-leakage rules, optional web research, and a reflection pass that reads rendered thumbnails and patches the spec.

The narrative contract

The single highest-leverage change was ensure_narrative_arc: whatever the model produces, the deck must contain an Agenda, the user's own topics as sections, a Plan or Timeline, Success Measures, and Next Steps. Not an infrastructure template — a shape, domain-agnostic, with the domain inferred from the request.

You can watch it happen. I asked for five slides on a datacentre migration and got nine, because the arc requires more than five to be complete:

OnDevice.ai showing a generated presentation: the chat reply with slide count and quality score, the Files drawer with the PPTX and slide thumbnails, and the Canvas pane listing the nine-section spec
Left: the reply reports 9 slides, 44,588 bytes, and an advisory quality score of 0.93. Centre: the Files drawer, with the generated PPTX and its slide thumbnails. Right: Canvas, showing the spec that produced it — Agenda, Context & objectives, Priorities & workstreams, Risks & dependencies, Outcomes & measures, Phased Plan, Success Metrics, Next Steps. Total time from prompt to file: about 35 seconds on a local model.

Canvas is the part I underestimated. Because the intermediate spec is a real object rather than a hidden prompt, it can be shown — and if it can be shown it can be edited. "Change slide 3 title to Risks" patches the spec and regenerates the file. No re-prompting, no regenerating from scratch, no drift in the other eight slides.

The intermediate spec started as an implementation detail for reliability. It turned out to be the most useful thing in the interface.
SPECS BEFORE BYTES — AND A LOOP THAT CHECKS THE RESULT Request + 12-msg history Outline LLM · arc enforced JSON spec quality gate Generator deterministic Structural gate hard · blocks Download expiring token reflect: render thumbnails → typed spec patches → regenerate (max_repairs) LibreOffice · advisory never blocks a download

Two words on that diagram carry all the weight. Hard: structural validation is the only thing that can block a download — if the OOXML isn't right, no file. Advisory: LibreOffice rendering and the quality score are opinions. They get reported (0.93 in the screenshot above) and they never stop you getting your file. Conflating those two would mean a slow or missing LibreOffice install could silently break document generation, which is exactly the class of failure this architecture exists to prevent.

Explainer · The reflection loop

Once a PPTX exists you can render its slides to images and look at them. That's what reflection does: render thumbnails, check for the failure modes that survive structural validation — an empty title, twelve bullets crammed on one slide, a body that repeats the title — and emit typed patches against the spec rather than free-text corrections.

Typed patches are the important bit. The model isn't asked to rewrite the deck; it's asked to fill fields in a known repair schema. The patch is applied to the spec, the deterministic generator runs again, and the whole thing is bounded by max_repairs so it cannot loop forever.

It's the same principle as the router: give the model a small, well-typed decision instead of a large, open one, and check the intermediate.

One branch of this deserves a mention because it complicates the story. If a model called deeppresenter-9b is installed, decks are generated by that model's JSON spec rather than the general path — a purpose-built 9B specialist for one task, with an optional Docker host that bakes in LibreOffice so PPTX pages can be rendered back for reflection. It's the one place the project runs an MCP tool list at all, and even there the tools belong to the host, not to the chat model. If the host is down, the normal pipeline still produces a deck.

The honest note: this is a good draft generator, not a good designer. Against Grok on depth of content, layout polish and language, local output is still thinner. That gap is tracked, not hidden — see below.

Catalogue: D-016 to D-028, D-030 to D-056, F-006 · Tests: Q-010 to Q-012, Q-025

05 11 → 17 AUGUST

Grounding had to earn its keep

A local-first product has an awkward relationship with web search. The whole promise is that your data stays on your machine; search is the one feature that deliberately sends part of your question to somebody else's server. It can't be on by default, and it can't be missing.

The resolution was WEB_SEARCH_MODE=auto: search runs only when the question looks live — news, weather, "latest", a URL, a fact-check phrasing. Ask it to explain a concept and nothing leaves the machine. Ask it who won yesterday and the query goes out, with the privacy cost documented rather than buried.

Problem

Searching on every message leaks the user's questions and pollutes answers with irrelevant snippets. Never searching makes the model confidently wrong about anything recent.

What changed

Auto mode gated on query shape; five providers behind one interface; authority ranking; page fetch to expand thin snippets; two-hop research; an explicit fact-check pipeline; and citations verified against the source file.

Why it mattered

Search became a capability with a stated cost rather than an always-on default — and the offline status strip tells you why a search was skipped, so silence is explained rather than mysterious.

Citations got the same treatment as documents: don't trust, verify. When a model cites an attached file, the pipeline records the quoted text, the document index, the title, and the start and end page numbers — then checks the quote against the actual file. Verified citations are marked in the UI. A quote the model invented doesn't get the badge.

Explainer · Retrieve-then-generate, and prefix caching

Retrieve-then-generate inverts the usual order. Instead of letting the model answer from memory and hoping it's right, you fetch sources first, inject them into the prompt as numbered documents, and instruct the model to answer only from those sources or explicitly abstain. Hallucination doesn't disappear, but it becomes visible: an unsupported claim has no [n] next to it.

That makes prompts long, which is where prefix caching earns its place. Turn two in a conversation usually repeats most of turn one — same system prompt, same PDF analysis, same citation pack — and only the last few hundred tokens differ. Two caches cooperate: an application-level prefix cache that avoids recomputing the analysis at all, and the runtime's own KV cache, kept warm by sending keep_alive=30m so Ollama doesn't unload the model between turns. Cloud providers get cache_control: ephemeral on the system message for the same reason. The UI reports hit or written under the reply, because a cache you can't observe is a cache you can't debug.

Catalogue: W-001 to W-020, C-045, L-032, F-001 to F-005 · Tests: Q-014 to Q-018

06 16 → 26 AUGUST

The chrome got out of the way

Sixty-odd features had accumulated in two weeks, and every one of them had asked for somewhere to live. The header had become a shelf: backend, model, online dot, temperature, reasoning, theme, files, canvas, prefs, memory, search status, account. All of it visible, all of the time, competing with the thing people actually came to do.

The last two changes in the log are the ones that made it feel finished, and neither adds a feature. C-065 moved every file — pending attachments, files already sent in this chat, and generated artifacts — into a Files drawer, and replaced the composer's row of attachment chips with a count badge. C-066 moved theme, panels, preferences, memory and status into a Tools drawer. The header kept four things: chat, backend and model, Tools, account.

The OnDevice.ai Tools drawer, showing status, four theme presets, Files and Canvas panels, custom instructions, a prompt library, and a local memory toggle set to Off
Everything that used to crowd the header, now one click away: status, four themes, the two side panels, global custom instructions, a prompt library, and local memory — off by default, stored only on this machine.

There's a smaller entry from 16 August that I find more instructive than either drawer. I'd been trying to fix the type scale by nudging it a point at a time, and the log records why that failed: "a 1pt tweak was ~1px and did not show." The eventual fix was to double the scale outright, look at it, then come down one visible step. It's a reminder that when you can't see a change, you're not tuning — you're guessing, expensively.

⌘K

Command palette

New chat, canvas, incognito, annotate — without hunting for a control.

◐

Four themes

Graphite, Midnight, Forest, Amber. Also five deck themes for generated PPTX.

◯

Incognito

Chats stay out of the sidebar list and are deleted on reset.

▣

Offline status

Active model, why a search was skipped, whether the prompt cache hit.

Local memory is worth calling out because of what it doesn't do. It's opt-in, default off, written to plain files on disk, and editable by hand. You add to it by saying "Remember: …" in chat. No embeddings, no cloud sync, no silent accumulation of a profile. For a product whose entire argument is about where data goes, memory that quietly builds itself would have contradicted the premise.

Catalogue: P-007, P-008, C-039 to C-041, C-051, C-060 to C-066

07 AS OF 26 AUGUST

What still isn't done

The catalogue says 238 shipped, 1 partial, 1 pending. That's a flattering way to count, for two reasons. Things I decided not to do yet never get a row at all. And "shipped" only ever meant this landed — never this is finished. Two of the rows below are marked shipped in the catalogue and are still, honestly, gaps. Here's the ledger.

Pending · W-019
No provider fallback chainIf Tavily is down, search doesn't quietly fall back to Brave and then DuckDuckGo — it just fails for that turn. The plumbing is there; the chain isn't wired.
Partial · P-003
The rename is half-doneThe product is OnDevice.ai everywhere a user can see. Underneath, the repo folder is still Muse_Contributer, the database is muse_workspace, the demo user is demo@muse.local, and a model preset is still called muse-glimmer-30b. Cosmetic, and it will stay that way until the name has proven it's sticking.
Shipped at 128k · L-027
1M context isn't real yetQwen3.8's million-token context needs YaRN rope scaling under vLLM or SGLang. Ollama GGUF can't do it. What shipped on 23 August is num_ctx up to 128k — a lot, and not the number on the model card. The custom-URL backend is the honest route to the real thing, and nobody has walked it yet.
Shipped, gap open · D-051
Decks are still thinner than Grok'sThe richness pass is marked shipped, and the narrative arc, metric callouts and reflection loop did close a lot of the gap. They did not close it. On depth of content, layout polish and language, a frontier model still writes a better deck — the project's own status file calls local output "an executive draft." What it has instead is a floor: it opens, it costs nothing per document, and it never left the laptop.
Deferred
Hardening is deferredAUTH_ALLOW_DEMO is still true by default, refresh tokens don't exist, and Redis is scaffolded rather than used. Correct for a single-user local tool; not correct for anything shared, and I'd rather say so than let the health endpoint imply otherwise.
Deferred
It isn't in version controlNo git remote, no CI. A deliberate choice while the shape was still moving, and one with a real cost — everything in this article is reconstructed from a hand-maintained changelog rather than from history. That's the next thing to fix.

There's a category the list doesn't capture: things that got ruled out. No general agent. No sub-agents. No dynamic tool discovery. No MCP handed to the chat model — the DeepPresenter host has its own tool list, but the model you are talking to never sees it. No desktop control — the browser specialist visits named hosts and nothing else. Each of those was considered and declined, and the declining is what kept the eleven pipelines checkable.

The non-goals did more architectural work than most of the features.

Where it lands

The model is the part that ages fastest

The lesson I'd take from these seventeen days is not about local inference. It's about which part of an AI product is worth building carefully.

The model in the socket today will be embarrassing in six months. That's not pessimism, it's the whole trajectory — I swapped the default twice in this window alone, and the second swap took a dropdown selection rather than a deployment. Everything I'd point at as real engineering sits around the socket: a router whose decisions you can read, an intermediate spec you can edit, a validation gate that can't be talked out of blocking a broken file, a boundary that holds regardless of which core is plugged in.

That's the argument for the shell. Not that local models are as good as frontier ones — Part 1 was clear that they aren't. It's that the ceiling belongs to the model and rises on somebody else's schedule, while the floor belongs to the shell and is entirely yours to build.

← Part 1 · The AI That Never Leaves My Laptop All projects