TL;DR: A small language model (SLM) is an AI text model with far fewer parameters than giant chatbots, typically from a few hundred million up to roughly 10 billion. That makes it cheap and fast enough to run on a laptop, a phone or a single server, often without an internet connection. SLMs such as Microsoft’s Phi, Google’s Gemma and Apple’s on-device model trade some broad knowledge and reasoning for speed, privacy and lower cost. They shine at focused tasks like summarizing, classifying and drafting, and they are often built using distillation, quantization and fine-tuning.
What is a small language model? It is a compact version of the same technology that powers ChatGPT and other large language models: a neural network trained to predict the next word, just with far fewer parameters. “Small” is relative. Some SLMs have billions of parameters, but they are tiny next to frontier models that are estimated to run into the hundreds of billions or more. This guide explains how SLMs work, how they compare with LLMs, real examples and when you should use one.

Small language model, defined
There is no official cut-off, but in practice the label is used for models that are small enough to run on modest hardware. IBM’s explainer lists examples ranging from scaled-down BERT models with a few million parameters up to models such as IBM Granite and Mistral’s Ministral 8B with about 8 billion.
Parameters are the internal numbers a model learns during training. More parameters generally mean more stored knowledge and better reasoning, but also more memory and compute every time the model answers a question, which is called inference.
SLM vs LLM: the key differences
| Small language model | Large language model | |
|---|---|---|
| Typical size | Hundreds of millions to ~10 billion parameters | Tens of billions to hundreds of billions or more |
| Where it runs | Phone, laptop, single GPU, edge devices | Data-center GPU clusters, usually via the cloud |
| Cost per request | Low | Higher |
| Speed | Fast, low latency | Slower, depends on server load |
| Privacy | Data can stay on your device | Data is usually sent to a provider |
| Broad knowledge and hard reasoning | Limited | Strong |
| Best for | Focused, repeatable tasks | Open-ended, complex tasks |
How small language models are made
An SLM is built on the same transformer architecture as its larger cousins. What changes is how developers squeeze quality into fewer parameters. The most common techniques:
Better training data
Microsoft’s Phi team argued that data quality can partly make up for size. Its Phi-3 technical report describes phi-3-mini, a 3.8-billion-parameter model trained on 3.3 trillion tokens of heavily filtered web data and synthetic data. Microsoft says it rivals much larger models such as Mixtral 8x7B and GPT-3.5 on benchmarks, “despite being small enough to be deployed on a phone.” Those are Microsoft’s own results.
Knowledge distillation
With knowledge distillation, a big “teacher” model’s outputs are used to train a smaller “student.” IBM notes that DistilBERT used this method to become 40% smaller and 60% faster than BERT while keeping 97% of its language understanding.
Quantization
Quantization stores each parameter in fewer bits, for example 4 bits instead of 16. That shrinks memory needs dramatically, which is how many SLMs fit on a phone or a laptop GPU.
Fine-tuning and adapters
Because SLMs are cheap to train further, companies often fine-tune them on one job. Apple describes its roughly 3-billion-parameter on-device model using small swappable “adapters” that specialize it for tasks such as summarizing or proofreading, each typically needing only tens of megabytes.
Examples of small language models
- Microsoft Phi: Phi-2 (2.7B) and Phi-3-mini (3.8B), designed to show how far small models can go with curated data.
- Google Gemma: a family of open models built from the same research as Gemini, sized for laptops, phones and single GPUs.
- Apple’s on-device foundation model: about 3 billion parameters, running Apple Intelligence features directly on iPhones, iPads and Macs.
- Mistral Ministral 3B and 8B: Mistral’s small models aimed at on-device and edge use.
- IBM Granite: business-focused open models, including 2B and 8B versions.
New SLMs appear almost every month, so treat this as a sample rather than a ranking.
What SLMs are good at (and where they struggle)
Good fits
- Summarizing emails, notes or documents
- Classifying support tickets or sorting feedback
- Autocomplete, rewriting and grammar help
- Extracting fields from forms and invoices
- Offline assistants on phones, cars and factory equipment
- High-volume tasks where cost per request matters
Weak spots
- Less world knowledge: fewer parameters store fewer facts, so SLMs are more prone to hallucination on obscure topics.
- Harder reasoning: multi-step math, coding and planning usually favor bigger models.
- Shorter memory: many SLMs have smaller context windows, although this varies by model.
A common workaround is pairing an SLM with retrieval-augmented generation (RAG), which feeds the model relevant documents at question time so it doesn’t need to memorize everything.
Should you use a small or large model?
Ask three questions:
- Is the task narrow and repeatable? If yes, an SLM, possibly fine-tuned, is often enough.
- Must data stay private or work offline? An SLM running locally is the simplest answer.
- Do you need deep reasoning or broad knowledge? Reach for a large model, or route only the hard requests to one.
Many products now mix both: a small model handles routine requests quickly, and a large model is called only when needed. That keeps costs down without giving up quality on the tough questions.
Small language model FAQ
Is a small language model the same as an on-device model?
Not always. Many SLMs are designed to run on phones and laptops, but you can also host one on a server or in the cloud. “Small” describes the model’s size; “on-device” describes where it runs.
Can I run a small language model on my own computer?
Often, yes. Open models such as Gemma and Phi can be downloaded and run locally with free tools, especially in quantized form. How well they run depends on your RAM and GPU, so check the model card for hardware requirements first.
Are small language models more private?
They can be. When a model runs entirely on your device, your prompts don’t have to leave it. If you use an SLM through a cloud service, the provider’s privacy policy still applies.
Bottom line
A small language model is a compact AI text model that trades some knowledge and reasoning power for speed, low cost and the ability to run on your own device. Thanks to better data, distillation and quantization, today’s SLMs handle many everyday tasks well. For focused jobs and privacy-sensitive apps, they are often the smarter choice; for hard, open-ended problems, large models still lead.




