Running LLMs on PyTorch with AMD ROCm™ Software

Overview

Want to run powerful AI language models on your own hardware? This guide shows you how. This tutorial uses PyTorch powered by AMD ROCm™ software to run models that can summarize documents, answer questions, generate text, and more, all running locally.

What You’ll Learn

• Run LLMs like gpt-oss-20b and qwen3.5-4B locally using PyTorch and ROCm
• Create a document summarization tool using LLMs

Setting the Memory Configuration

sudo apt install pipx
pipx ensurepath

pipx install amd-debug-tools

amd-ttm

amd-ttm --set <NUM>

Installing Software Prerequisites

Create a Virtual Environment

sudo apt update
sudo apt install -y python3-venv
python3 -m venv pytorch-env --system-site-packages
source pytorch-env/bin/activate

sudo usermod -aG render,video $LOGNAME

sudo apt update
sudo apt install -y python3-venv
python3 -m venv pytorch-env
source pytorch-env/bin/activate

python -m venv pytorch-env --system-site-packages
pytorch-env\Scripts\activate

python -m venv pytorch-env
pytorch-env\Scripts\activate

Installing Basic Dependencies

python -m pip install --index-url https://repo.amd.com/rocm/whl/gfx1151/ "torch==2.11.0+rocm7.13.0" "torchvision==0.26.0+rocm7.13.0" "torchaudio==2.11.0+rocm7.13.0"

python -m pip install --index-url https://repo.amd.com/rocm/whl/gfx1150/ "torch==2.11.0+rocm7.13.0" "torchvision==0.26.0+rocm7.13.0" "torchaudio==2.11.0+rocm7.13.0"

python -m pip install --index-url https://repo.amd.com/rocm/whl/gfx1152/ "torch==2.11.0+rocm7.13.0" "torchvision==0.26.0+rocm7.13.0" "torchaudio==2.11.0+rocm7.13.0"

python -m pip install --index-url https://repo.amd.com/rocm/whl/gfx110X-all/ "torch==2.11.0+rocm7.13.0" "torchvision==0.26.0+rocm7.13.0" "torchaudio==2.11.0+rocm7.13.0"

python -m pip install --index-url https://repo.amd.com/rocm/whl/gfx120X-all/ "torch==2.11.0+rocm7.13.0" "torchvision==0.26.0+rocm7.13.0" "torchaudio==2.11.0+rocm7.13.0"

Installing Additional Dependencies

pip install transformers==4.57.1 safetensors==0.6.2 accelerate sentencepiece protobuf

pip install transformers safetensors accelerate sentencepiece protobuf

pip install transformers==5.10.1 safetensors accelerate sentencepiece protobuf

pip install "transformers>=5.9.0" safetensors accelerate sentencepiece protobuf

Quick Start with Example Scripts

This playbook includes ready-to-use scripts. Click them to preview and download them to the same directory as the environment you created.

Both scripts support:

• Model selection via --model flag
• Chat template formatting for proper model prompting, especially useful for document summarization

Loading and Running Your First LLM

The included run_llm.py script shows how to generate text with LLMs using PyTorch and AMD ROCm.

The snippet below shows how to use the model and customize the questions asked.

model_name = "${hf_model}"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

prompt = "Explain what a large language model is in 2 brief sentences."
print(f"Prompt: {prompt}\n")

messages = [
    {"role": "system", "content": "You are a helpful technology assistant"},
    {"role": "user", "content": f"{prompt}"},
]

Try out the downloaded script:

python run_llm.py --model ${hf_model}

Building a Document Summarizer

Now that you’ve generated local LLM output, you can build on that by making a practical document summarizer. In this section, you will use the summarizer.py script to feed in a .txt file and automatically generate a concise summary, all running locally on your GPU.

The script is designed to work out of the box. Open the script in an editor to explore the code, customize prompts, and tweak parameters like length and temperature.

Usage Examples

# Summarize the built-in example text (defaults to openai/gpt-oss-20b)
python summarizer.py --model ${hf_model}

# Summarize a text file
python summarizer.py --file example_document.txt

# Adjust creativity with temperature
python summarizer.py --file document.txt --temperature 0.5

# Longer summaries with more tokens
python summarizer.py --file document.txt --max-length 400

Learn about Generation Parameters

Real-World Applications

• Research Paper Analysis: Extract key findings from complex publications for quick review
• News Aggregation: Summarize news articles into brief daily digests or highlights
• Meeting Notes: Condense transcripts into actionable items and concise summaries
• Legal Document Review: Extract relevant clauses or obligations from long legal texts quickly
• Code Documentation: Generate concise repository overviews and function explanations

Next Steps

• Fine-tuning: Adapt models to your specific field or jargon for better accuracy (see Fine-tuning Playbooks)
• RAG Systems: Combine LLMs with document retrieval for context-aware answers and search
• Model Exploration: Experiment with new models like Llama 3, Phi-3, or Qwen for better results
• Production Deployment: Use tools like vLLM for scalable LLM serving in organizations

Your system gives you the power to run sophisticated language models locally. Experiment with different models, prompts, and parameters to discover what works best for your applications.