There is a constraint in medical AI that shapes everything I build: a lot of the data I work with cannot leave the building. Not "should not" — contractually, legally cannot. For years that meant either running inference on hardware inside the hospital network or not using the interesting models at all.
So I have been paying close attention to small language models, not because they are a nicer product story, but because for a meaningful class of problems they are the only option that exists.
What Counts as Small
"Small" is relative and the boundary keeps moving. Practically, a small language model is one that fits and runs acceptably on hardware a person already owns — a laptop, a phone, a workstation GPU — rather than a rack of accelerators. Today that means roughly one to twelve billion parameters, quantised down to 4 or 8 bits so the weights fit in available memory.
The frontier models are much larger. Nobody outside the labs knows exactly how large — parameter counts for GPT-4 class models have not been published, and the confident numbers circulating online are guesses. What is known is the practical consequence: they run in data centres, they are reached over a network, and every query is a round trip.
| Frontier hosted models | Small local models | |
|---|---|---|
| Typical size | Not disclosed; data-centre scale | ~1B-12B parameters |
| Where inference runs | Provider's servers | Your device |
| Network required | Yes | No |
| Latency floor | Network round trip plus generation | Generation only |
| Data handling | Governed by provider terms | Never leaves the device |
| Marginal cost | Per token | Electricity and hardware you already bought |
That last row is worth pausing on, because "free" is the claim people make and it is not quite true. Local inference costs you memory, battery, thermal headroom and the capital cost of adequate hardware. What it does not cost you is a per-token bill that scales with usage, which is a different and often better cost structure.
Why This Became Practical
Three things changed at roughly the same time.
Quantisation got good. Running weights at 4-bit precision used to cost noticeable quality; with modern quantisation schemes the degradation on most tasks is small enough that the memory savings are clearly worth it. This is what took a 7B model from "needs a serious GPU" to "runs on a laptop".
Training got better at small scale. The insight behind Microsoft's Phi series was that data quality could substitute for parameter count — train on carefully filtered and synthetically generated material rather than scraped bulk text, and a small model punches well above its size. The Phi-3 technical report reports that the 3.8B model matches or beats considerably larger contemporaries on standard benchmarks. Treat that as the authors' benchmark result rather than a guarantee about your task — small models trained on heavily curated data tend to look better on benchmarks than they feel in open-ended use — but the direction is real, and it has been confirmed by everyone else building in this space since.
Consumer hardware acquired dedicated inference silicon. Phone SoCs and laptop chips now ship with neural accelerators and, critically, with enough unified memory bandwidth to make token generation tolerable. Memory bandwidth, not compute, is usually what limits generation speed on device.
Where Local Models Are Genuinely Better
Privacy is the real one, and it is not a preference — it is an unlock. Clinical data, legal documents, unpublished research, internal financial records: these are categories where sending the text to a third party is either prohibited or requires a procurement process measured in months. A model running on local hardware sidesteps the question entirely. Note the limit, though: running a model locally does not make you HIPAA-compliant or GDPR-compliant on its own. It removes one class of disclosure risk. Access control, audit logging, encryption at rest and everything else still apply, and I have seen teams conflate "on-prem" with "compliant" in a way that would not survive an audit.
Latency for interactive features. For anything that responds as you type — autocomplete, inline correction, a suggestion that appears while your fingers are still moving — the network round trip is disqualifying regardless of how fast the model is. A small local model starts producing tokens immediately. It will not out-reason a frontier model, but for the class of task where the answer needs to arrive before you finish thinking, it wins by default.
Working with no connection. Aircraft, rural clinics, basements, field sites, anywhere with unreliable mobile coverage. When I built a personal finance app for Android, the fact that it worked with the radio off was a feature users noticed. The same applies to anything that wants to be genuinely dependable.
Predictable cost at volume. If you are classifying a million documents a month, per-token pricing dominates your budget in a way that a one-time hardware purchase does not.
Where They Are Not Better
Small models are worse at the things that need breadth and depth: long multi-step reasoning, obscure factual recall, code that spans many files, nuanced writing. They hallucinate more, and they are less good at recognising when a question is outside their competence. If your task genuinely requires frontier reasoning, a 3B model will not get there and no amount of prompting will fix it.
They also have shorter usable context. The advertised context length and the length over which a small model actually maintains coherence are different numbers, and the gap is wider than it is for frontier models.
The Models Worth Knowing
Microsoft's Phi family is the most interesting for genuinely small deployments — the smallest variants run on phones and were designed for that from the start. Meta's smallest Llama releases are explicitly targeted at on-device use and have the widest tooling support. Google's Gemma models are open-weight and well documented. Mistral's 7B-class models remain a strong default for laptop-scale use; they were the release that convinced a lot of people this was viable at all. Apple's approach is a hybrid — small models on device for routine requests, with an escalation path to larger models running in an attested server environment for harder ones.
Rankings between these change constantly. Pick two, run your own task through both, and re-check in six months.
Trying It
The lowest-friction path is Ollama, which handles downloading, quantisation and serving:
curl -fsSL https://ollama.com/install.sh | sh
ollama run phi3That is the whole setup. LM Studio does the same thing with a graphical interface and a local OpenAI-compatible endpoint, which makes it easy to point existing code at a local model without rewriting anything. On mobile, the practical routes are Core ML on Apple platforms, the MediaPipe LLM inference API on Android, and ONNX Runtime if you need both.
Start with a 3B model before a 7B one. If the small one is adequate, the larger one is a waste of memory you will want for something else.
The Architecture That Is Actually Winning
The framing of "local versus cloud" is already obsolete in practice. What is being built is routing: a small model on device handles the routine majority of requests, and anything it cannot handle escalates to a larger model — with the rule that data classified as sensitive never escalates at all.
This is harder than it sounds, because the routing decision is itself a judgement call, and a small model deciding whether it is competent to answer is exactly the kind of self-assessment small models are bad at. Getting that gate right is, in my view, the interesting open problem in this space. It matters more than another benchmark point.
The useful question is not whether small models will replace hosted ones. It is which of your requests genuinely need a data centre — and for most applications, honestly assessed, the answer is fewer than you would assume.
Tags
Taresh Sharan