Running LLMs with PyTorch and AMD ROCm™ software

Overview

Want to run powerful AI language models on your own hardware? This guide shows you how. This tutorial uses PyTorch powered by AMD ROCm™ software to run models that can summarize documents, answer questions, generate text, and more, all running locally.

What You’ll Learn

• Run LLMs like gpt-oss-20b and Mistral-7B-Instruct locally using PyTorch and ROCm
• Create a document summarization tool using LLMs

Initial Setup

Create a Virtual Environment

python -m venv llm-env --system-site-packages
llm-env\Scripts\activate

sudo apt update
sudo apt install -y python3-venv
python3 -m venv llm-env --system-site-packages
source llm-env/bin/activate

python -m venv llm-env
llm-env\Scripts\activate

sudo apt update
sudo apt install -y python3-venv
python3 -m venv llm-env
source llm-env/bin/activate

Installing Basic Dependencies

sudo apt update
wget https://repo.radeon.com/amdgpu-install/7.2.1/ubuntu/noble/amdgpu-install_7.2.1.70201-1_all.deb
sudo apt install ./amdgpu-install_7.2.1.70201-1_all.deb
sudo amdgpu-install -y --usecase=rocm --no-dkms

sudo usermod -aG render,video $USER

sudo reboot

ls -l /opt/rocm
ls -l /opt/rocm/lib/libroctx64.so*

# Check ROCm device files (Device files owned by the render group should be visible)
ls -l /dev/kfd
ls -l /dev/dri/renderD*

# Check user groups ($USER should be listed in both render and video)
id
groups

# Check ROCm with rocminfo ('Permission denied' error should NOT be seen)
rocminfo | sed -n '1,120p'

# Check installed ROCm version
cat /opt/rocm/.info/version

python -m pip install --upgrade pip
python -m pip install --force-reinstall --no-cache-dir --index-url https://repo.amd.com/rocm/whl/gfx1151/ torch torchvision torchaudio

python -m pip install --upgrade pip
python -m pip install --force-reinstall --no-cache-dir --index-url https://repo.amd.com/rocm/whl/gfx1152/ torch torchvision torchaudio

python -m pip install --upgrade pip
python -m pip install --force-reinstall --no-cache-dir --index-url https://repo.amd.com/rocm/whl/gfx1150/ torch torchvision torchaudio

python -m pip install --upgrade pip
python -m pip install --force-reinstall --no-cache-dir --index-url https://repo.amd.com/rocm/whl/gfx1151/ torch torchvision torchaudio

python -m pip install --upgrade pip
python -m pip install --force-reinstall --no-cache-dir --index-url https://repo.amd.com/rocm/whl/gfx1152/ torch torchvision torchaudio

python -m pip install --upgrade pip
python -m pip install --force-reinstall --no-cache-dir --index-url https://repo.amd.com/rocm/whl/gfx1150/ torch torchvision torchaudio

Installing Additional Dependencies

pip install transformers==4.57.1 safetensors==0.6.2 accelerate sentencepiece protobuf

pip install transformers safetensors accelerate sentencepiece protobuf

Quick Start with Example Scripts

This playbook includes ready-to-use scripts. Click them to preview and download them to the same directory as the environment you created.

Both scripts support:

• Model selection: openai/gpt-oss-20b (default) or other models such as Mistral
• Chat template formatting for proper model prompting, especially useful for document summarization

Loading and Running Your First LLM

The included run_llm.py script shows how to generate text with LLMs using PyTorch and AMD ROCm.

The snippet below shows how to use the model and customize the questions asked.

model_name = "openai/gpt-oss-20b"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

# Create system and user prompts
prompt = "Explain what a large language model is in 2 brief sentences."
print(f"Prompt: {prompt}\n")

messages = [
    {"role": "system", "content": "You are a helpful technology assistant"},
    {"role": "user", "content": f"{prompt}"},
]

Try out the downloaded script:

python run_llm.py

Building a Document Summarizer

Now that you’ve generated local LLM output, you can build on that by making a practical document summarizer. In this section, you will use the summarizer.py script to feed in a .txt file and automatically generate a concise summary, all running locally on your GPU.

The script is designed to work out of the box. Open the script in an editor to explore the code, customize prompts, and tweak parameters like length and temperature.

Usage Examples

# Summarize a large piece of text (at Line 152 in summarizer.py)
python summarizer.py

# Summarize a text file
python summarizer.py --file example_document.txt

# Adjust creativity with temperature
python summarizer.py --file document.txt --temperature 0.5

# Longer summaries with more tokens
python summarizer.py --file document.txt --max-length 400

Learn about Generation Parameters

Real-World Applications

• Research Paper Analysis: Extract key findings from complex publications for quick review
• News Aggregation: Summarize news articles into brief daily digests or highlights
• Meeting Notes: Condense transcripts into actionable items and concise summaries
• Legal Document Review: Extract relevant clauses or obligations from long legal texts quickly
• Code Documentation: Generate concise repository overviews and function explanations

Next Steps

• Fine-tuning: Adapt models to your specific field or jargon for better accuracy (see Fine-tuning Playbooks)
• RAG Systems: Combine LLMs with document retrieval for context-aware answers and search
• Model Exploration: Experiment with new models like Llama 3, Phi-3, or Qwen for better results
• Production Deployment: Use tools like vLLM for scalable LLM serving in organizations

Your system gives you the power to run sophisticated language models locally. Experiment with different models, prompts, and parameters to discover what works best for your applications.