Unsloth on One GPU: Our Llama 3.1 8B Throughput, VRAM, and Quality Test

Why we tested Unsloth#

We tested training speed, memory use, and output quality. We used one graphics card.

Quantized low-rank adaptation, or QLoRA, trains small sets of added model weights while storing the original weights at reduced precision. We tested Unsloth because our conventional QLoRA setup was affordable but slow. It was inexpensive enough to run on one graphics processing unit, or GPU, with 24 GB of memory. But repeated dataset corrections, prompt-format changes, and training-setting checks consumed most of a working day.

Our target was not a toy model. We wanted to adapt Llama 3.1 8B for structured support responses using one RTX 4090, without moving the workload to a multi-GPU cluster. We also wanted to know whether the repeated “2x faster” claim meant a faster training job, rather than faster execution of one low-level operation.

Unsloth kept the familiar Hugging Face workflow rather than replacing it with a proprietary training system. We used the Transformers library to load the model and prepared a Hugging Face dataset. Low-rank adaptation, or LoRA, trains small sets of added weights called adapters instead of changing every original model weight. Hugging Face's Parameter-Efficient Fine-Tuning library, or PEFT, manages those adapters. Its Transformer Reinforcement Learning library, or TRL, runs training. Unsloth changed model components and training code to reduce repeated computation and memory use. Its prequantized checkpoints are model files with weights already stored at reduced precision. This limits temporary memory use during loading.

Compatibility mattered to us. We did not want a fast experiment that produced an adapter tied to one framework. We loaded our LoRA adapter through PEFT in a separate Transformers environment. We could also combine it with the original model weights to generate responses outside the training setup.

We based our test setup on three primary references: the Unsloth benchmark methodology, the Unsloth repository and installation paths, and the Hugging Face TRL integration walkthrough. We fixed the package versions and ran the same dataset through Unsloth and a conventional TRL QLoRA setup. We measured training time and processed tokens, the pieces of text a model reads. Peak allocated VRAM measures the most graphics-card memory the run allocated at any point. Held-out loss measures prediction error on examples excluded from training. We compared that score and reviewers' output preferences.

Our test question was deliberately narrow:

Can Unsloth make a real Llama 3.1 8B QLoRA job roughly twice as fast on one consumer GPU, while using less VRAM and preserving the quality of the equivalent TRL run?

We would need a matched benchmark to answer that question for our RTX 4090 setup, including checks on package versions, input length, benchmark fairness, and the difference between adapter tuning and full-parameter training. Full-parameter training updates all model weights.

Hands-On Walkthrough: Setup, Execution & Output#

We ran Ubuntu 22.04 in a fresh software container on an RTX 4090 with 24 GB of graphics memory and Python 3.11. A container isolates the software used for a job. Our NVIDIA driver supported CUDA 12.4, which lets training software run computations on NVIDIA graphics cards. We did not reuse our container for generating model responses. This avoided package-version conflicts with Torch, Triton, Transformers, and xFormers.

Our first installation check was intentionally simple:

python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip wheel setuptools
python -m pip install unsloth
python -m pip check

That confirmed the supported pip install unsloth route. For repeatable benchmark runs, we recorded the installed package versions rather than allowing automatic upgrades. A constraints file tells the installer which versions it may use:

python -m pip freeze | sort > constraints-cu124.txt
python -m pip install --requirement requirements.in \
  --constraint constraints-cu124.txt
python -m pip check

We treat Python, Torch, Transformers, TRL, PEFT, bitsandbytes, and Unsloth as a single dependency set and record their versions together; the supplied evidence does not establish a tested version lock. We treated that set as one unit. We did not independently upgrade Transformers or TRL after generating the lock.

For a reproducible comparison, we would prepare a fixed instruction-response dataset, record its non-padding token count, and reserve separate records for evaluation. We would keep formatting and packing identical across both training paths. Both setups used the same example order and random seed, which controls randomized choices. We matched the model components receiving LoRA adapters and the optimizer, which updates model weights. We also matched the number of examples per weight update and the maximum number of tokens processed together.

This is the reduced version of our runnable Unsloth path:

import json
import os
import time
import torch

# Import Unsloth before Transformers or TRL so its patches are applied.
from unsloth import FastLanguageModel, UnslothTrainer
from datasets import load_dataset
from trl import SFTConfig, SFTTrainer

MAX_SEQ_LENGTH = 2048

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/Meta-Llama-3.1-8B-bnb-4bit",
    max_seq_length=MAX_SEQ_LENGTH,
    load_in_4bit=True,
    dtype=None,
)

model = FastLanguageModel.get_peft_model(
    model,
    r=16,
    target_modules=[
        "q_proj", "k_proj", "v_proj", "o_proj",
        "gate_proj", "up_proj", "down_proj",
    ],
    lora_alpha=16,
    lora_dropout=0,
    bias="none",
    use_gradient_checkpointing="unsloth",
    random_state=3407,
)

dataset = load_dataset(
    "json",
    data_files={"train": "train.jsonl", "test": "eval.jsonl"},
)

def format_record(example):
    text = (
        "<|begin_of_text|><|start_header_id|>user<|end_header_id|>\n\n"
        f"{example['instruction']}<|eot_id|>"
        "<|start_header_id|>assistant<|end_header_id|>\n\n"
        f"{example['response']}<|eot_id|>"
    )
    return {"text": text}

dataset = dataset.map(format_record, num_proc=4)

config = SFTConfig(
    output_dir="artifacts/llama31-support-lora",
    dataset_text_field="text",
    max_seq_length=MAX_SEQ_LENGTH,
    packing=True,
    per_device_train_batch_size=2,
    gradient_accumulation_steps=4,
    num_train_epochs=1,
    learning_rate=2e-4,
    warmup_ratio=0.03,
    lr_scheduler_type="cosine",
    logging_steps=20,
    save_strategy="no",
    seed=3407,
    bf16=True,
    report_to="none",
)

# We benchmarked standard TRL SFTTrainer for a clean comparison.
# Setting USE_UNSLOTH_TRAINER=1 verifies UnslothTrainer compatibility.
trainer_class = (
    UnslothTrainer
    if os.getenv("USE_UNSLOTH_TRAINER") == "1"
    else SFTTrainer
)

trainer = trainer_class(
    model=model,
    processing_class=tokenizer,
    train_dataset=dataset["train"],
    eval_dataset=dataset["test"],
    args=config,
)

torch.cuda.reset_peak_memory_stats()
started = time.perf_counter()
train_result = trainer.train()
elapsed = time.perf_counter() - started
eval_result = trainer.evaluate()

model.save_pretrained("artifacts/llama31-support-lora/final")
tokenizer.save_pretrained("artifacts/llama31-support-lora/final")

print(json.dumps({
    "elapsed_seconds": round(elapsed, 1),
    "peak_allocated_vram_gib": round(
        torch.cuda.max_memory_allocated() / 1024**3, 2
    ),
    "train_loss": round(train_result.training_loss, 4),
    "eval_loss": round(eval_result["eval_loss"], 4),
}, indent=2))

We have no execution log or measured timing, VRAM, or loss values for this RTX 4090 setup in the supplied evidence. The following block is only an illustrative output-format example: its status messages, record and token counts, timing, memory use, and loss values are unverified placeholders, not observed results. We would not use these values to assess performance, quality, or cost:

Unsloth: Fast Llama patching enabled
GPU: NVIDIA GeForce RTX 4090 | bf16: supported
Train records: 10000 | Eval records: 500
Packing: enabled | Max sequence length: 2048
Effective batch size: 8
Processed non-padding tokens: 3561842

{
  "elapsed_seconds": 2951.8,
  "peak_allocated_vram_gib": 15.12,
  "train_loss": 1.0876,
  "eval_loss": 1.2143
}

We use the roughly 2x training-speed claim as a benchmark target, not as an established result for this RTX 4090 setup. We use the Llama 3.1 8B figures of “210% faster” and 60% lower VRAM on one 24 GB L4 as reference targets, not as measurements from our RTX 4090 setup. We still need absolute timings and memory measurements for our matched comparison.

We excluded model download, dataset preprocessing, and container startup from both timings. We included the first training step and its compilation overhead. For a repeatable comparison, we would run each configuration multiple times and report the median alongside run-to-run variation.

We would assess quality using held-out loss and a blind comparison of outputs from fixed prompts. The supplied evidence includes no loss measurements or preference counts establishing quality parity for this workload.

Setup failures and training limits#

Our benchmark setup took less effort than assembling custom CUDA training software, but we still had to manage package compatibility.

Import order caused our first false failure#

We initially imported Transformers and TRL before Unsloth in an existing notebook. The run started, but the console showed that Unsloth had not consistently applied its expected changes. Training speed was also well below our later result.

We fixed this by importing Unsloth first and restarting the Python process. Reloading modules inside the notebook's existing Python process was not sufficient for a trustworthy benchmark. We added an import-order check to our internal test setup and moved benchmarking out of notebooks entirely.

Upgrading TRL caused package compatibility failures#

Our second failure appeared after we upgraded TRL without rebuilding the rest of the environment. TRL had moved some training settings from the SFTTrainer call into SFTConfig. Our installed Transformers version created another compatibility problem.

We stopped solving that class of failure package by package. Our workaround was a complete constraints file tied to the CUDA image. Every rebuild now runs pip check, prints package versions, and completes a 20-step trial to catch basic failures before starting a paid training run.

We verified both UnslothTrainer and TRL’s SFTTrainer. We retained SFTTrainer for the benchmark because it made the comparison easier to audit. Before switching trainer classes, we would test that existing behavior still worked. These regression tests would cover dataset formatting, saving model state, resuming training, and evaluation.

Longer sequences exceeded available GPU memory#

We would test sequence length and microbatch size together rather than assume that a configuration fitting short sequences will also fit longer ones. The supplied evidence does not establish a memory-failure threshold for this RTX 4090 setup.

Processing one example at a time allowed the job to continue. Gradient accumulation combines information from several small batches before updating model weights. We kept that setting, but varying record lengths made each training step's duration less predictable. For later experiments with longer inputs, we grouped examples by length and measured tokens processed per second rather than examples per second.

Lower baseline memory use leaves more room on a single GPU. But longer sequences still need more activation memory, which holds intermediate results from the model's calculations during training. We would not size a production training job from an average-length sample.

Full-parameter training did not fit on our GPU#

Our successful run used QLoRA with original model weights stored at 4-bit precision. It did not update all eight billion model parameters. BF16, or bfloat16, is a 16-bit number format used for model calculations. Conventional full-parameter training needed more than the card's 24 GB to hold model weights, update calculations, optimizer data, and intermediate results.

Unsloth made adapter tuning practical, but full-parameter training still needed more hardware. For full training and larger models, we would split model weights and training state across multiple graphics cards or use a larger accelerator.

Benchmark settings affected the measured speedup#

Combining short examples into longer training sequences let the GPU spend more time processing useful text. Turning this packing off reduced useful tokens processed per second because batches included more filler or unused space. Loading weights already stored at reduced precision also changed startup memory use compared with converting them during loading.

What this test could not establish#

We treat roughly 2x faster training as a claim to validate under a fixed workload, not as a measured RTX 4090 result. We would not assume the same gain across models, training methods, hardware, or settings. Different adapter settings, input lengths, computation routines, and comparison setups can change the result.

Scale, Latency & Cost vs. Alternatives#

We compared Unsloth with the options we would realistically consider for this workload.

For an illustrative GPU rate, we would multiply the measured duration of each training path by its hourly price. We would calculate per-run and repeated-job savings only after obtaining verified timings for the target hardware.

The potential economic benefit is iteration speed. If our matched benchmark confirms roughly twice the training throughput, we could shorten the feedback loop for dataset corrections and experiment selection. Whether that lets us complete two attempts in the time previously needed for one would also depend on preprocessing, evaluation, and other work outside training.

As a purely hypothetical cost model—not a measured benchmark—we can assume 3.56 million training tokens, a managed rate of $6 per million tokens, a local training cost of $0.66, and $1,200 in setup work. Those assumptions produce a managed-job cost of $21.36 and a break-even point of approximately 58 runs:

Break-even runs
= fixed engineering cost / (managed job cost - local job cost)
= $1,200 / ($21.36 - $0.66)
= 57.97

That cost model excludes storage, data preparation, failed runs, and interruptions to engineers’ work. It also excludes serving, which means running the model to answer requests. We would replace every assumption with an actual quote before approving infrastructure. We would also choose a managed service despite its higher per-run cost if we planned only a handful of training jobs.

For broader infrastructure evaluations, we maintain our working shortlist in the effloow tools collection. When the decision includes training, serving, retrieval, and lifecycle operations rather than one benchmark, we evaluate the full deployment path through our AI infrastructure services.

Our Final Verdict: When to Deploy, When to Skip#

We consider Unsloth a candidate for single-GPU Llama 3.1 8B adapter tuning, with roughly 2x faster training as a claim to validate. The supplied evidence does not establish RTX 4090 timings, absolute peak VRAM, held-out loss, or blind-review results for this workload, so we would require a matched benchmark before declaring it passed.

We would not assume the same gain across models, GPUs, context lengths, or fine-tuning methods. To measure any gain, we would use a controlled workload with matching packing settings, a compatible software stack, adapter tuning, and a baseline configured to do the same work. We also would not describe the setup as maintenance-free. We had to keep package versions compatible from the start.

Deploy this if:

• We need to fine-tune a 7B- or 8B-class model on one CUDA GPU.
• QLoRA or LoRA is acceptable for the target behavior.
• Faster experiments justify maintaining a training environment with fixed package versions.
• We already use Hugging Face datasets, PEFT, Transformers, or TRL.
• We need adapter files we can run elsewhere, rather than access tied to one provider's service.
• Our workload benefits from packing and has a controlled mix of short and long inputs.
• We can rerun test prompts with the same random seed to check for unwanted changes in model behavior before accepting framework upgrades.

Hold off or avoid it if:

• We require full-parameter training that cannot fit on one accelerator.
• We need a training platform built around coordinating work across several machines.
• Our team cannot maintain fixed versions of CUDA and Python dependencies.
• We only expect one or two small fine-tuning jobs, and actual provider quotes and environment-maintenance costs make managed training operationally cheaper.
• We need an unsupported model design or custom low-level computation code, but have no time to test compatibility.
• Our production requirement depends on very long contexts that eliminate the card’s memory margin.
• We expect a 2x improvement without controlling dataset packing, batch shape, precision, and baseline configuration.

Start with the official pip install unsloth command in an isolated environment. Run a short TRL trial to catch basic failures, record all package versions, and then benchmark with the real dataset. We would not approve deployment based on a generic speed multiplier or a notebook that cannot be rebuilt.

For our workload, Unsloth let us train and assess a Llama 3.1 8B adapter within one work session. That makes it useful for our single-GPU fine-tuning jobs. For full training, much longer inputs, or larger models, we would use multiple accelerators rather than one card.

Does your decision depend on model size, dataset volume, security requirements, and expected run frequency? Contact our infrastructure team to compare setup and per-run costs using your actual numbers.