LoRA (Low-Rank Adaptation) is a way to fine-tune a large AI model without retraining all of it: you freeze the original weights and train two small “adapter” matrices that nudge the model toward a new task. Because only the adapters learn, you need far less GPU memory and storage than full fine-tuning. In the original 2021 LoRA paper, Microsoft researchers reported that, compared with fine-tuning GPT-3 175B with Adam, LoRA cut the number of trainable parameters by 10,000 times and the GPU memory requirement by 3 times, while matching or beating full fine-tuning on the models they tested.
This guide explains how LoRA works in plain English, what “rank” and “alpha” mean, how it compares with full fine-tuning, RAG and plain prompting, and where it falls short. No linear algebra degree required.
Quick note on the name: LoRA in AI is not the same as LoRa (long-range radio) or LoRaWAN, the wireless networking standards used for IoT sensors. Same letters, unrelated technologies.
LoRA in one sentence (and why it exists)
LoRA is a parameter-efficient fine-tuning (PEFT) method: it keeps a pretrained model frozen and learns a small, low-rank correction to some of its weight matrices.
The problem it solves is cost. Fine-tuning normally means updating every weight in the model. For a model with billions of parameters, that requires a lot of GPU memory during training (for the weights, their gradients and the optimizer state) and produces a full-size copy of the model for every task. The LoRA authors put it bluntly: with GPT-3 175B, deploying independent fine-tuned copies, each with 175 billion parameters, is “prohibitively expensive.”
LoRA’s bet is that the change a model needs for a new task is much simpler than the model itself. So instead of rewriting the whole model, you learn a compact “patch.”
How LoRA works (without the scary math)
Think of a pretrained model as a huge book of numbers (weights) organized into grids called matrices. Fine-tuning edits those numbers. LoRA leaves the book untouched and adds a thin set of sticky notes that adjust what each page says.
Step 1: Freeze the base model
The original weights, often written as W, are locked. They are not updated during training, so the training process does not need to store gradients or optimizer state for them. According to the PyTorch torchtune LoRA tutorial, that is where most of LoRA’s memory savings come from: fewer parameters with gradients and less optimizer state.
Step 2: Train two small matrices, A and B
Next to a chosen weight matrix, LoRA adds two skinny matrices. In torchtune’s description, matrix A projects the input down to a much smaller size (the rank, r), and matrix B projects it back up to the layer’s normal output size. Multiplied together, B × A has the same shape as the original weight matrix, so it can act as a correction to it.
When the model runs, each adapted layer produces two outputs that get added together:
- the normal output of the frozen weights (W × input), and
- a small learned adjustment (B × A × input), scaled by a setting called alpha.
Only A and B learn. Everything else stays exactly as the original model creators trained it.
Step 3: Choose where to plug it in
Modern language models are built on the transformer architecture, which is full of matrices. LoRA is usually attached to the attention layers. The original paper adapted only the attention weights and, in most experiments, only the query and value projections (Wq and Wv) for simplicity. Today’s tools let you go further: Hugging Face’s course notes that target_modules can be specific modules like q_proj and v_proj, or “all-linear”, with more modules giving more adaptability at the cost of more memory.
Why LoRA is so much smaller: a worked example
The savings come from simple arithmetic. In Llama 2 7B, the attention projections are 4,096 by 4,096 matrices. Full fine-tuning of one of those projections means training:
- Full fine-tuning: 4,096 × 4,096 = 16,777,216 weights
- LoRA at rank 8: A is 4,096 × 8 and B is 8 × 4,096, so 8 × (4,096 + 4,096) = 65,536 weights
That is about 0.4% of the original count for that one layer. Across the whole model, torchtune’s example LoRA setup trains 4,194,304 of 6,742,609,920 parameters, or 0.06%.
At GPT-3 scale the numbers are dramatic. The LoRA paper reports that on GPT-3 175B, training VRAM dropped from 1.2TB to 350GB, and with rank 4 on only the query and value matrices, the saved checkpoint shrank from 350GB to about 35MB. The authors also measured a 25% training speedup over full fine-tuning, because gradients are not computed for most parameters.
One honest caveat from torchtune: LoRA’s memory savings come mainly from gradients and optimizer state, so if a model’s peak memory happens during the forward pass, LoRA may not reduce peak memory much. You still need to fit the full base model in memory. If you are sizing hardware, our guide to RAM vs VRAM explains why GPU memory is usually the bottleneck.
LoRA vs full fine-tuning vs RAG vs prompting
LoRA is one of several ways to make a general model fit your needs. They solve different problems, and many real systems combine them.
| Approach | What changes | Best for | Main trade-off |
|---|---|---|---|
| Prompting | Nothing in the model; you change the instructions | Quick experiments, formatting, tone | Limited by what the model already does; long prompts cost tokens every call |
| RAG (retrieval-augmented generation) | Nothing in the model; relevant documents are fetched and added at answer time | Fresh, private or frequently changing facts | Needs a search/retrieval pipeline; does not change the model’s style or skills |
| LoRA | Small adapter matrices; base model frozen | Teaching a consistent behavior, format, style or narrow skill from example data on modest hardware | Can learn less than full fine-tuning on hard new domains |
| Full fine-tuning | Every weight | Maximum adaptation when you have lots of data and compute | Expensive to train and store; higher risk of forgetting general skills |
A practical rule of thumb: if the model lacks information, try RAG. If it lacks a behavior you can show with examples (always answer in a certain JSON shape, write in your brand voice, classify support tickets your way), LoRA is usually the cheapest kind of fine-tuning to try. Start with prompting either way, since it costs nothing to test.
Rank and alpha in plain English
Two settings come up in every LoRA guide.
Rank (r): how big the adapter is
The rank is the width of the narrow middle between A and B. A higher rank means a more expressive adapter with more trainable parameters. Hugging Face’s course says rank is typically between 4 and 32, and suggests starting small (4 to 8) and checking results against a validation set.
Surprisingly small ranks can work. In the LoRA paper’s GPT-3 experiments, a rank as small as one was enough for adapting Wq and Wv on the WikiSQL and MultiNLI datasets. The authors add that they do not expect a small rank to work for every task or dataset.
Bigger can still help. In torchtune’s Llama 2 7B experiments, moving from rank 8 on query and value layers only to rank 64 on all layers raised accuracy on a TruthfulQA task from 0.475 to 0.514, while peak memory rose from 15.57GB to 17.04GB.
Alpha: how strongly the adapter pushes
Alpha scales the adapter’s output before it is added to the frozen layer. In torchtune’s reference code the update is multiplied by alpha divided by rank. Hugging Face describes alpha as usually set to about 2x the rank, with higher values giving stronger adaptation, and torchtune recommends scaling alpha and rank together.
You do not need to memorize these. The useful mental model is: rank is how much the adapter can learn; alpha is how loudly it speaks.
What LoRA is good at, and where it falls short
Strengths
- Cheap to train: far fewer trainable parameters and less optimizer memory, as the numbers above show.
- Tiny files: you store a small adapter per task instead of a full model copy. The LoRA paper estimates that storing 100 adapted GPT-3 models would take about 354GB instead of about 35TB.
- Less forgetting: a 2024 study titled “LoRA Learns Less and Forgets Less” found LoRA better preserved the base model’s performance on tasks outside the training domain than full fine-tuning did.
- Swappable: one base model can host many adapters (more on this below).
Limitations
- It can learn less: the same 2024 study found that, in standard low-rank settings, LoRA substantially underperformed full fine-tuning on programming and math. So “LoRA matches full fine-tuning” is true for some tasks, not all.
- Adapters are model-specific: IBM Research notes that a LoRA trained for one base model won’t work with a different model, even a newer version of the same family.
- It is still fine-tuning: you need good example data and a way to evaluate results. Too little data or too many training steps can lead to overfitting, just like any training.
- It does not add live knowledge: facts baked in by training go stale. For changing information, retrieval is the better tool.
Deploying LoRA: merge it or swap it
After training you have two choices.
Merge the adapter into the base weights. Because B × A has the same shape as the original matrix, you can add it in once and get a normal model back. The LoRA paper highlights that, unlike earlier adapter methods, this means no additional inference latency. Hugging Face’s PEFT library does this with a merge_and_unload() call. This is the simple route when you serve one specialized model. (New to the term? See what AI inference is.)
Keep adapters separate and swap them. If you need many specialists (one per customer, language or task), you can load one base model and switch adapters. IBM describes LoRAs as like custom bits for a multi-head screwdriver that can be swapped in and out. Serving systems have pushed this hard: the S-LoRA research system was designed to serve thousands of LoRA adapters on a single GPU or across multiple GPUs with small overhead.
The paper itself notes the trade-off: if you merge adapters to remove latency, it is not straightforward to batch requests for different tasks in a single forward pass.
LoRA vs QLoRA
QLoRA is LoRA on a compressed base model. Instead of keeping the frozen model in 16-bit precision, QLoRA stores it in 4-bit form and trains LoRA adapters on top. Its authors reported that this reduced memory enough to fine-tune a 65B-parameter model on a single 48GB GPU while preserving 16-bit fine-tuning task performance. If “4-bit” is new to you, our explainer on quantization in AI covers how shrinking number precision saves memory.
In short: LoRA reduces what you train; QLoRA also reduces what you store in memory while training.
Where LoRA fits with SFT, RLHF and other training methods
People often list LoRA next to SFT or RLHF as if they were alternatives. They answer different questions:
- SFT, RLHF, DPO describe what the model is trained to do: copy good example answers, or prefer answers humans rated higher. See our explainer on RLHF.
- LoRA describes which weights get updated: a small adapter instead of the whole model.
So you can do supervised fine-tuning with LoRA. Hugging Face’s course shows exactly that, passing a LoRA config into TRL’s SFTTrainer. Other ways to make models cheaper, such as knowledge distillation or simply choosing a small language model, can be combined with LoRA too.
LoRA beyond chatbots: image models
If you have browsed AI art communities, you have probably seen downloadable “LoRAs” for a particular style or character. That is the same idea applied to image generators. IBM notes that since 2021 LoRA has spread to image generation and voice recognition, and Hugging Face’s Diffusers library documents LoRA training for image models. As with language models, an image LoRA only works with the base model family it was trained for.
A simple example of when you would use LoRA
Here is a hypothetical scenario to make it concrete. A small online store wants a support assistant that always replies in its friendly house style and sorts every message into one of six ticket categories. Prompting gets it most of the way, but the model keeps drifting in tone and inventing category names.
- The team collects a few hundred real (anonymized) messages with the ideal replies and correct categories.
- They take an open-weight model and train a small LoRA adapter on those examples, holding some back to test.
- They compare the adapter against the prompt-only version on the held-back examples.
- If it wins, they merge the adapter for serving, and use RAG for product details and order policies that change often.
That split, LoRA for behavior and retrieval for facts, is the pattern most teams end up with.
Frequently asked questions
What is LoRA in AI?
LoRA (Low-Rank Adaptation) is a fine-tuning method that freezes a pretrained AI model and trains two small adapter matrices beside some of its layers. The adapters learn a compact correction for a new task, so you customize the model with far fewer trainable parameters and less GPU memory than updating every weight.
What does LoRA stand for?
LoRA stands for Low-Rank Adaptation. “Low-rank” refers to representing the weight update as the product of two thin matrices, which needs far fewer numbers than a full-size matrix. It was introduced by Microsoft researchers in a 2021 paper.
How is LoRA different from full fine-tuning?
Full fine-tuning updates every weight and produces a full model copy per task. LoRA keeps the original weights frozen and trains only small adapters that can be stored separately, swapped, or merged into the model. It is much cheaper, though research shows it can learn less than full fine-tuning on hard new domains such as code and math.
Is LoRA better than RAG?
They do different jobs. RAG gives a model fresh or private information at answer time without changing it. LoRA changes the model’s behavior, style or skills using training examples. Use RAG when the model lacks facts and LoRA when it lacks a behavior; many production systems use both.
What rank should I use for LoRA?
Hugging Face’s course says rank is typically between 4 and 32 and suggests starting small, around 4 to 8, then checking results on a validation set. Higher ranks can improve quality on harder tasks but use more memory, so increase rank only if evaluations show the adapter is underfitting.
Does LoRA slow down the model?
Not if you merge it. Because the adapter has the same shape as the original weights once multiplied out, it can be added into the base model, giving no additional inference latency. Keeping adapters separate adds a small amount of computation but lets you swap specialists on one base model.
What is the difference between LoRA and QLoRA?
QLoRA is LoRA trained on top of a base model stored in 4-bit precision. That cuts memory further: the QLoRA authors fine-tuned a 65-billion-parameter model on a single 48GB GPU. LoRA alone keeps the frozen base model at normal precision.
The bottom line
LoRA is the reason fine-tuning stopped being something only big labs could afford. By freezing a pretrained model and training a small low-rank adapter, it cuts trainable parameters, memory and storage dramatically, and the original paper showed this can match full fine-tuning on many tasks with no extra inference latency once merged. It is not magic: adapters are tied to one base model, they can learn less than full fine-tuning on very different domains, and they do not replace retrieval for up-to-date facts. If you want a model to reliably behave a certain way and you have good examples, LoRA is usually the first fine-tuning method worth trying. For the bigger picture, start with our guide to fine-tuning in AI.
The Teck Hustlers Editorial Team covers technology, business, finance and gaming news for readers who want the story behind the headline. Every article is researched, written and edited by our in-house team, checked against company announcements, filings and reputable reporting, and updated when new facts emerge. We focus on what each development means for founders, professionals, investors and everyday users. Have a tip or a correction? Reach us through our contact page.
