Fine-Tuning LLMs with Unsloth

Overview

This playbook shows how to fine-tune a language model locally with Unsloth on AMD hardware.

It uses a short Supervised Fine-Tuning (SFT) example with LoRA adapters on unsloth/gemma-4-E4B-it, using a subset of the mlabonne/FineTome-100k dataset. The goal is to give you a simple end-to-end workflow that covers setup, training, inference, and saving the fine-tuned result.

The example is designed to be practical and easy to modify, so you can use it as a starting point for your own datasets and models.

What You’ll Learn

• How to set up the Unsloth environment
• How to fine-tune a LLM using SFT with Unsloth
• How to save the fine-tuned result in local storage

Why Unsloth?

Unsloth makes LLM fine-tuning easier to run on local hardware by reducing memory usage and speeding up training compared to a standard setup.

In this playbook, we use Unsloth together with LoRA-based SFT. That means the base model stays mostly frozen, while a much smaller set of adapter weights is trained. This is a good fit for local development because it is lighter than full fine-tuning and faster to iterate on.

Unsloth also supports other training approaches, including QLoRA and reinforcement learning workflows. This playbook focuses on the simplest path first: a small LoRA fine-tuning example that users can run, understand, and extend.

Setting the Memory Configuration

sudo apt install pipx
pipx ensurepath

pipx install amd-debug-tools

amd-ttm

amd-ttm --set <NUM>

Installing Software Prerequisites

Create a Virtual Environment

sudo apt update
python3 -m venv unsloth-env --system-site-packages
source unsloth-env/bin/activate

sudo usermod -aG render,video $LOGNAME

sudo apt update
sudo apt install -y python3-venv
python3 -m venv unsloth-env
source unsloth-env/bin/activate

python -m venv unsloth-env --system-site-packages
.\unsloth-env\Scripts\activate

python -m venv unsloth-env
.\unsloth-env\Scripts\activate

Installing Basic Dependencies

python -m pip install --index-url https://repo.amd.com/rocm/whl/gfx1151/ "torch==2.11.0+rocm7.13.0" "torchvision==0.26.0+rocm7.13.0" "torchaudio==2.11.0+rocm7.13.0"

python -m pip install --index-url https://repo.amd.com/rocm/whl/gfx1150/ "torch==2.11.0+rocm7.13.0" "torchvision==0.26.0+rocm7.13.0" "torchaudio==2.11.0+rocm7.13.0"

python -m pip install --index-url https://repo.amd.com/rocm/whl/gfx1152/ "torch==2.11.0+rocm7.13.0" "torchvision==0.26.0+rocm7.13.0" "torchaudio==2.11.0+rocm7.13.0"

python -m pip install  --index-url https://repo.amd.com/rocm/whl/gfx1200-all/ torch torchvision torchaudio

Additional Dependencies

pip install "unsloth[amd] @ git+https://github.com/unslothai/unsloth.git"

pip install "unsloth[amd] @ git+https://github.com/unslothai/unsloth.git"
pip install triton-windows

Download the Unsloth Fine-Tuning Script

Instead of manually executing each step, this playbook provides a clean, end-to-end script here: test_unsloth.py .

Run the following code to execute the script:

python test_unsloth.py

The rest of the playbook will conceptually go through each major step of the script.

How It Works

The test_unsloth.py script performs the following steps:

• Load Model: Loads unsloth/gemma-4-E4B-it using FastModel.
• Prepare Data: Standardizes the dataset (e.g., FineTome-100k) and applies the Gemma-4 chat template.
• Apply LoRA: Adds adapters to language, attention, and MLP modules for efficient training.
• Train: Uses SFTTrainer with response-only loss masking.
• Inference: Runs a quick generation test to verify performance.
• Save: Exports LoRA adapters locally.

Key Configuration

You can modify the following constants to customize your run:

MODEL_NAME = "unsloth/gemma-4-E4B-it"
MAX_SEQ_LEN = 1024
DATASET_NAME = "mlabonne/FineTome-100k"
OUTPUT_DIR = "gemma_4_lora"

Example of the Unsloth welcome message and output when loading the model weights:

Prepare Dataset

We use a subset of:

mlabonne/FineTome-100k

The dataset is:

• Converted into chat format
• Processed using the Gemma-4 chat template
• Cleaned to remove duplicate BOS tokens

Train the Model

The script runs a short training demo, with the following parameters:

• ~50 steps
• Small batch size
• Gradient accumulation

During training, you will see logs such as:

Saving and Deployment

Local Saving (LoRA)

The script automatically saves LoRA adapters to the OUTPUT_DIR.

model.save_pretrained("gemma_4_lora")
tokenizer.save_pretrained("gemma_4_lora")

Save merged model (for vLLM)

model.save_pretrained_merged("gemma-4-finetune", tokenizer)

Export GGUF (for llama.cpp)

Convert directly to GGUF for local inference:

model.save_pretrained_gguf("gemma_4_finetune", tokenizer, quantization_method="Q8_0")

Next Steps

• Try Unsloth Studio, an intuitive GUI for Unsloth
• Train on your own specific datasets
• Try finetuning with different hyperparameters
• Deploy with vLLM or llama.cpp
• Try QLoRA for a lower-memory setup

Resources

Below are some additional resources to learn more about Unsloth and finetuning:

• Unsloth Docs

• Unsloth GitHub

• Unsloth Fine-tuning Guide