AirLLM is an open-source Python library that runs very large language models on very small GPUs. Llama 3.3 70B in fp16 wants about 140GB of VRAM, which normally means a cluster of datacenter cards. AirLLM runs it on a single 4GB GPU, or an Apple Silicon MacBook, with no quantization required. The repo (lyogavin/airllm) sits at 22.4k GitHub stars. This is the full setup: install, model access, the exact script, and the honest expectations.
How it actually works
A transformer is a deterministic stack of layers, roughly 80 of them in a 70B Llama. Inference executes them strictly in order: layer 1's output feeds layer 2, and once a layer has run it is not needed again until the next token. Nothing about the architecture requires all 80 layers to sit in memory at once. That is the whole exploit.
AirLLM keeps exactly one layer on the GPU at a time. Load a layer from disk, run it, free it, load the next. A single layer of a 70B model is around 1.6GB in fp16, which fits a 4GB card with room to spare. The math is unchanged, so the output is identical to running the same weights on an A100 node. You are trading VRAM for time, and we will get to exactly how much time.
What you need
- An NVIDIA GPU with 4GB+ of VRAM, or an Apple Silicon Mac (AirLLM runs on MLX there).
- Python 3.10 or newer.
- Disk space, and this is the requirement people miss: the Llama 3.3 70B weights are ~132GB, and AirLLM writes a layer-sharded copy of them on first run. Budget ~300GB free if you keep the original download, ~150GB if you let AirLLM delete it after splitting. Use an NVMe SSD. Layer loading is the entire bottleneck, so a hard drive technically works and practically hurts.
- A Hugging Face account for the Llama license.
Step 1: Install
python3 -m venv airllm-env
source airllm-env/bin/activate
pip install airllmThat pulls in PyTorch, Transformers, and friends. One package, no compilation step.
Step 2: Get access to Llama 3.3
The official meta-llama/Llama-3.3-70B-Instructrepo is gated. Open the model page on Hugging Face, accept Meta's license, then create a read token under Settings, Access Tokens. Approval is usually minutes.
If you do not want to wait, the unsloth/Llama-3.3-70B-Instruct mirror hosts the same weights ungated. Swap the model ID in the script and skip the token entirely.
Step 3: The script
from airllm import AutoModel
model = AutoModel.from_pretrained(
"meta-llama/Llama-3.3-70B-Instruct",
hf_token="hf_your_token_here",
)
input_text = ["Explain what a git worktree is in one paragraph."]
input_tokens = model.tokenizer(
input_text,
return_tensors="pt",
truncation=True,
max_length=128,
)
output = model.generate(
input_tokens["input_ids"].cuda(),
max_new_tokens=64,
use_cache=True,
return_dict_in_generate=True,
)
print(model.tokenizer.decode(output.sequences[0]))That is the entire program. AutoModel detects the architecture, and generation works like any Transformers model. The only AirLLM-specific behaviour is what happens on the first run.
Step 4: The first run (be patient)
On first execution AirLLM downloads the ~132GB of weights, then splits them into one shard per layer and saves those to disk. Depending on your connection and SSD this takes a few hours. It is a one-time cost; every run after that skips straight to inference.
Two constructor arguments worth setting before you kick it off:
model = AutoModel.from_pretrained(
"meta-llama/Llama-3.3-70B-Instruct",
layer_shards_saving_path="/path/to/your/biggest/ssd",
delete_original=True,
)layer_shards_saving_path points the sharded copy at whichever drive actually has room. delete_original=True reclaims the ~132GB original download once the split copy exists, which is what gets the total footprint down to ~150GB.
Running it on a Mac
AirLLM supports Apple Silicon through MLX, and the script is the same with one change: drop the .cuda()call on the input tokens. Everything else, including the sharding behaviour and disk requirements, is identical. Unified memory means your Mac's RAM plays the role of the 4GB card, and an M-series SSD is fast enough that this is one of the more pleasant ways to run it.
Optional: 3x faster with compression
AirLLM ships block-wise quantization that compresses the layer shards to 4-bit or 8-bit, which matters because inference time is dominated by reading shards off disk. Smaller shards, faster tokens. The repo claims roughly 3x faster inference with minimal accuracy loss.
pip install -U bitsandbytes
model = AutoModel.from_pretrained(
"meta-llama/Llama-3.3-70B-Instruct",
compression="4bit",
)Note this does change the numbers slightly, unlike the default mode. If you came here specifically for lossless 70B output, skip compression and accept the slower tokens.
The catch (read this before you start the download)
Every generated token streams the model through the GPU one layer at a time. In fp16 that is ~132GB of reads per token. A good consumer NVMe drive sustains ~3.5GB/s, so you are looking at tens of seconds per token uncompressed, and on the order of 10 seconds per token with 4-bit compression. A one-paragraph answer is a make-coffee-and-come-back job.
So no, this does not replace a chat app, and anyone implying real-time 70B chat on a 4GB card is selling something. What it is actually for: batch and offline jobs where a big model's answer quality beats latency, testing 70B-class output before renting serious hardware, and the simple fact that hardware you already own can now do something that used to require a thousand-dollar compute node.
Quick reference
- Install:
pip install airllm. Mac (Apple Silicon) supported via MLX, drop the.cuda(). - Model access: accept the license on
meta-llama/Llama-3.3-70B-Instructplus an HF read token, or use the ungatedunslothmirror. - Disk: ~300GB free for the first run, ~150GB with
delete_original=True. NVMe strongly recommended. - Shard location:
layer_shards_saving_pathinfrom_pretrained. - Speed: seconds to tens of seconds per token.
compression="4bit"(needsbitsandbytes) is roughly 3x faster. - Repo:
lyogavin/airllmon GitHub.