The AI That Never Leaves My Laptop

The question that stops the demo

Show almost any enterprise a useful AI workflow and the demo goes well right up until someone from security asks the only question that matters: where does the data go?

It is not an unreasonable question. The answer for every mainstream assistant is some version of "to a datacentre we operate, under a policy you should read." For a great many teams that answer is fine. For the ones I work with — regulated industries, client-confidential material, an internal audit function with opinions — it ends the conversation. Not because the tool is bad, but because the paperwork to say yes costs more than the productivity it would buy.

The second question is quieter and arrives later: what happens when the price changes? A workflow that costs fractions of a cent per document today is a line item you don't control tomorrow. Anything built on metered inference carries that risk permanently.

OnDevice.ai is the answer to that question, built as a working product rather than a thought experiment. It is a private AI workspace: conversational chat, project-based context, file analysis, and structured document generation — running primarily on the user's own machine.

A Claude-shaped workspace, pointed at your own machine

The shape is deliberately familiar. There is a sidebar with projects and conversations, a chat pane that streams tokens with a Stop button, file upload, and an artifacts panel. If you have used Claude or ChatGPT, there is nothing to learn.

What's different sits in the header: a Backend selector and a Model selector. In the screenshot below it is running Glimmer 30B in GGUF format through Ollama, on a MacBook. No account, no quota, no request leaving the machine — and a temperature slider, prefill and stop sequences exposed directly, because when the model is yours there is no reason to hide the controls.

Specs before bytes

This is the decision the whole system is built around, and it's the one I'd defend hardest.

There is an obvious way to make an AI produce a PowerPoint file: ask the model for one. It is also a bad way. OOXML is a zip archive of interdependent XML parts with relationship graphs and content-type manifests. A model emitting that directly is generating a binary format token by token, where a single malformed relationship produces a file that simply refuses to open — and you find out when your client double-clicks it.

So the model never touches the bytes. It produces a JSON intermediate spec — a SlideDeckSpec, WorkbookSpec or DocumentSpec — which is schema-validated before anything else happens. A fixed, deterministic generator built on python-pptx, openpyxl or python-docx turns that spec into the actual file. The architecture document puts it plainly:

Then comes the part I think is genuinely unusual: structural validation is a hard gate on download. The artifact is checked as a zip, as OOXML parts, as a relationship graph, as well-formed XML. Fail, and the file is never offered. LibreOffice rendering checks also run — but they are explicitly advisory, in an isolated process, on temporary copies that never touch the final artifact. A LibreOffice hiccup can't block a structurally valid file from reaching you.

The payoff is a property the commercial tools don't offer: a file that downloads is a file that opens. Not "usually." The gate is structural and it is binary.

Everything that matters is inside one boundary

The stack is deliberately ordinary — Next.js and TypeScript at the front, FastAPI and Python behind it, PostgreSQL for durable state, Redis optional, local filesystem for binaries. The interesting part isn't the components. It's where the boundary sits.

Ten principles govern what may be added. Five are worth quoting because they explain the refusals as much as the features:

Two local runtimes, honestly separated

A detail that sounds pedantic and turns out to matter: Ollama and LM Studio are not interchangeable. Ollama runs GGUF; LM Studio on a Mac runs MLX. They use different model libraries and different formats, and pointing one at the other's model directory produces confusing failures rather than a working setup.

A lot of local-AI tooling papers over this. Here it's a stated principle — "do not pretend one library is shared" — with the two backends kept explicitly separate, the model catalogue filtered by format, and a documented bridge script for the one case where you genuinely want a GGUF from your LM Studio library registered into Ollama.

Switching backend in the UI persists to disk and survives an API restart. Switching provider deliberately clears the model override, because model IDs rarely match across backends and silently keeping a stale one is how you end up debugging a phantom.

What you can actually run today

This is the part that has changed fastest, and it's the part most people's mental model is a year out of date on. The catalogue shipped with the app is not aspirational — these are presets pointing at open-weight models you download and run on a workstation:

Eleven presets, spanning roughly 12B to 80B parameters, across two runtimes, all swapped from a dropdown against the same pipeline. The hardware underneath is a MacBook with 64 GB of unified memory — a good laptop, not a rack.

That is the single most important thing to understand about this whole category, and it's why "a local model can't compete" ages badly as an argument. It's true on the day you say it and less true every quarter. The capability that needed a datacentre, an API key and a data-processing agreement a year ago is now a download. Not the newest capability — but the one that was state of the art recently enough that most production work never needed anything better.

What the numbers actually say

That's the argument. Here's the evidence, and I want to be careful with it because this is exactly the kind of claim that gets overstated. Qwen3.8-27B was released on 14 August 2026 under Apache 2.0 — 27.78 billion parameters, multimodal, 262K native context. I downloaded it the following day. These are the figures from Qwen's own launch table:

A 27-billion-parameter model that fits on a laptop is within 2.1 points on GPQA Diamond and ahead on instruction-following and LiveCodeBench. Two years ago that sentence would have been nonsense.

Now the caveats, which matter as much as the numbers. Every one of those figures is vendor-run — they come from Qwen's own launch card, and at the time of writing there is no independent reproduction. Several use in-house benchmarks. The comparison against current frontier models is looser still, because the published GPT-5.6 and Opus results use different harnesses, effort settings and context limits. The most useful phrasing I've seen for this is Kingy.ai's:

Against the current frontier the leads are real and hold on the hardest long-horizon work — GPT-5.6 Sol reaches 88.8 on Terminal-Bench and 94.6 on GPQA where this model manages 73.0 and 89.2. So the honest summary isn't "local has caught up." It's that the useful floor has risen far enough that the gap stops mattering for a large share of real work — and it did so in about eighteen months.

Which points at the architecture most people will actually end up with, and it isn't the one either camp is selling: route routine and confidential work to a local model, escalate the genuinely hard or high-value cases to a frontier endpoint. This system is built for the first half of that sentence, and doesn't pretend to be the second.

Search that stays off unless it's needed

A local model with a training cutoff will confidently invent this morning's news. The fix is retrieve-then-generate: search first, put the sources in the prompt, and let the model cite them or abstain.

The design decision worth noticing is when it fires. Mode auto — the default — only searches when the message actually looks live: news, weather, prices, "latest", a URL. Ask it to restructure a paragraph and nothing leaves the machine. Ask it who won last night and it goes and looks.

You can force it to always or off. DuckDuckGo is the default provider; Bing, Brave and Google CSE are supported if you'd rather. And the architecture document is candid in a way product marketing usually isn't — it states outright that when search is enabled, queries leave the host. That's the honest caveat on the privacy claim, written down by the person making the claim.

Where it wins, and where it plainly doesn't

This is the section where these write-ups usually go wrong, so let me be direct about the conclusion first: on raw capability, Claude and ChatGPT win, and it isn't close. A frontier model has more knowledge, better reasoning, and far better taste. Anyone claiming a 30B model on a laptop matches them is selling something.

That's not the axis this was built to compete on.

Read the table honestly and a pattern falls out. The commercial services win every row about capability and convenience. This wins every row about control. Which column matters depends entirely on the document in front of you — and for a lot of the work I see, the second column is the only one that gets past legal.

The ceiling and the floor

One distinction is worth drawing carefully, because it's the most useful thing in this whole comparison.

A frontier assistant generating a document has a high ceiling and no floor. It will usually produce something impressive, and occasionally hand you a file that's broken or subtly wrong, with nothing in the path to catch it.

This has a lower ceiling and a hard floor. The deck won't win a design award today. It will open, every time, because a structural gate says so.

And the two move independently. The floor is set by the pipeline and never drops. The ceiling is set by whatever model you loaded this morning — which means it rises without me writing a line of code. Swapping Glimmer 30B for Qwen3.8 27B, or for the 80B build, is a dropdown. The generators, the specs, the validation gate: unchanged.

That's the actual bet, and it's a bet on a trend rather than on my code: own the floor, and let the ceiling arrive by download. Every few months the open-weight models absorb another year of frontier progress, and every one of those releases upgrades this system for free. The frontier labs have to keep out-running that. I just have to keep the disk space.

The work that can't leave the machine

And the honest inverse: if your work isn't sensitive, your volume is modest, and you want the best possible answer, use Claude or ChatGPT. They're better at that job. This exists for the cases where "better answer" isn't the binding constraint.

The non-goals are the architecture

Most AI products advertise what they added. This one keeps a written list of what it refuses, and the list is a mirror image of where the frontier labs are racing:

Every one of those is a real capability I'd enjoy having. Every one also widens the blast radius of a model that is, at the end of the day, a very confident text predictor. The whole product is aimed at people who must answer for what a system did — and each refusal is one fewer question you can't answer.

Which is also why file uploads are labelled untrusted in the prompt. A model that treats an uploaded document as instructions is one malicious PDF away from doing something it shouldn't. Here, files are data. Only the project instructions are trusted, because only you wrote those.

Work in progress, and honest about it

This is a live build, not a launch. Roughly 19,000 lines, 120 catalogued features, 19 test modules, and a 961-line architecture document that is genuinely current — which for a personal project is the part I'm most pleased about, because it's the part that usually rots first.

The document generation is the weakest link right now, and I'd rather say so than let a screenshot imply otherwise. Decks generate, validate and download — but a Grok or Claude deck still looks better, because the content planning behind it comes from a far stronger model. That gap is the ceiling I described earlier, and it moves when the model does.

What already works is the part that was hard to be sure about at the start: you can run a real AI workspace — projects, streaming chat, files, artifacts, search when you want it — on a laptop, with nothing leaving the machine, for the price of electricity. Two years ago that wasn't true. It's true now, and most people haven't noticed yet.

← More build notes

Sources

Benchmark figures for Qwen3.8-27B are reproduced from Qwen's launch model card as compiled and audited by Kingy.ai — Qwen3.8-27B: Specs, Benchmarks & Verdict (14 August 2026), with the official Qwen3.8-27B checkpoint on Hugging Face. All launch scores are vendor-run and were without independent reproduction at the time of writing; frontier comparisons use different harnesses and are directional rather than controlled. Everything about OnDevice.ai's own architecture comes from its Software Architecture Document and source.