Skip to content
Filters & more
Device Family
Device
OS

Getting Started with vLLM

Learn how to run inference and serving using containerized vLLM on the integrated GPU

Overview

vLLM is a high-performance inference engine designed for large language models (LLMs). It provides optimized serving with continuous batching for high throughput and an OpenAI-compatible API for seamless application integration. This makes vLLM great for production deployments where speed and resource efficiency are critical.

This playbook teaches you how to serve LLMs using containerized vLLM on the integrated GPU and interact with models through the OpenAI Python API.

What You’ll Learn

  • How to set up and start a vLLM server with AMD ROCm™ support
  • How to interact with models via OpenAI-compatible API endpoints
  • How to send prompts to the local server with vllm-prompt

Setting the Memory Configuration

Install AMD Software: Adrenalin Edition

Download and install the latest AMD Software: Adrenalin Edition from amd.com/en/products/software/adrenalin.html.

Terminal window
if command -v rocminfo >/dev/null 2>&1 && rocminfo | grep -q gfx; then
echo "OK: GPU visible to ROCm"
else
echo "GPU not visible via rocminfo" >&2
exit 1
fi

Advanced: increasing GPU memory (optional)

You may need to increase the memory allocated to the GPU to run certain larger models or longer contexts. These devices use unified memory — the GPU shares system RAM — so this means letting the GPU claim more of that shared pool. Leave roughly 20% of system RAM for the operating system. This step is optional; the default allocation is enough for most workloads.

On Windows, adjust the memory in AMD Software: Adrenalin Edition: open it and navigate to Performance → Tuning → AMD Variable Graphics Memory. Set the value you need and reboot for the change to take effect.

AMD Software Adrenalin Edition — AMD Variable Graphics Memory panel

Backup method (BIOS). If your system does not show the Variable Graphics Memory option in Adrenalin (some OEM systems hide it), set the UMA Frame Buffer Size in the system BIOS/UEFI instead: reboot, enter setup (usually Del, F2, or Esc), find UMA Frame Buffer Size — also labeled Integrated Graphics or dedicated GPU memory, often under Advanced or AMD CBS → NBIO Common Options — set it to the size you need, save, and reboot. Menu names vary by vendor, and some BIOS do not expose it either.

On the AMD Ryzen™ AI Halo, adjust the shared memory in the pre-installed AMD Ryzen™ AI Developer Center: open it, go to the Settings tab, and under Graphics Performance Settings increase the Shared Video Memory slider. Click Apply Changes and reboot for the change to take effect.

AMD Ryzen AI Developer Center — Graphics Performance Settings with Shared Video Memory slider

On Linux, raise the shared GTT/TTM pool so the GPU can map more system memory. AMD recommends setting the minimum dedicated VRAM in the BIOS (0.5 GB) so the maximum is available as shared memory.

  1. Install the amd-debug-tools helper:
Terminal window
sudo apt install pipx
pipx ensurepath
pipx install amd-debug-tools
  1. Query the current shared-memory limit:
Terminal window
amd-ttm
  1. Raise it (value in GB — pick a size that leaves headroom for the OS):
Terminal window
amd-ttm --set <NUM>
  1. Reboot for the change to take effect.

Check for Software Updates

Before starting, ensure your Ryzen AI Halo has the latest software installed. Open the AMD Ryzen™ AI Developer Center and check for available updates, both to the app itself and additional software.

Go to the Updates tab. If updates are available, install them and reboot before continuing.

AMD Ryzen AI Developer Center — Updates tab on Windows

Go to the Manage tab. If updates are available, install them and reboot before continuing.

AMD Ryzen AI Developer Center — Manage tab on Linux

Installing Software Prerequisites

vLLM runs in a prebuilt container with ROCm and its dependencies pre-matched. No additional installation is required.

There is no host-side vLLM installation step. Start vLLM with:

Terminal window
vllm-launch

The launcher starts the container, targets the integrated GPU, and exposes a local OpenAI-compatible vLLM server. Alternatively, click the vLLM icon in the taskbar.

Quick Start

1. Confirm the vLLM Server Is Running

The vllm-launch may take a couple minutes to initialize everything. Once it starts, the server is available at http://localhost:8001. Keep the launch terminal open because the server runs in the foreground, then open a separate terminal for the remaining steps. The examples below use Qwen/Qwen3-1.7B; if your launcher is configured for a different model, substitute that model ID in the requests.

2. Send a Prompt

Use the provided vllm-prompt script to send a request to the local vLLM OpenAI-compatible server:

Terminal window
vllm-prompt "Tell me a story"

3. Chat with the model using the OpenAI Python API

Since vLLM exposes an OpenAI-compatible API, you can use the openai Python package to interact with it.

First, create a Python virtual environment:

Terminal window
sudo apt install -y python3-venv
python3 -m venv vllm-env
source vllm-env/bin/activate

Install the OpenAI package

Terminal window
pip install openai

Create an OpenAI client pointed at the local vLLM server instead of OpenAI’s servers. The api_key is required by the client but vLLM doesn’t validate it, so any string works:

from openai import OpenAI
client = OpenAI(
base_url="http://localhost:8001/v1",
api_key="EMPTY",
)

Then, send a chat completion request. This uses the same message format as the OpenAI API — a list of messages with roles like "user" and "assistant". Setting stream=True means the response will arrive incrementally rather than all at once:

response = client.chat.completions.create(
model="Qwen/Qwen3-1.7B",
messages=[
{"role": "user", "content": "Tell me a short story"},
],
max_tokens=2048, # Maximum number of tokens the model will generate in its response
stream=True,
)

Finally, iterate over the streamed chunks and print each piece of text as it arrives:

for chunk in response:
content = chunk.choices[0].delta.content
if content:
print(content, end="", flush=True)

The included script contains the entire example and can be downloaded.

Choosing and Configuring a Model

By default, vllm-launch serves Qwen/Qwen3-1.7B as a test model on port 8001. You can change the model, the port, and the vLLM serving parameters without rebuilding or editing the container.

Models tested by AMD

The following models are pre-configured and validated by AMD:

ModelNotes
Qwen/Qwen3-1.7BDefault model. Lightweight and fast to load.
openai/gpt-oss-20bLarger model for higher-quality responses.

Launching a different model

Pass the model ID with --model (or -m):

Terminal window
vllm-launch --model openai/gpt-oss-20b

Changing the port

Pass a port above 1024 with --port (or -p); the default is 8001:

Terminal window
vllm-launch --port 8080 --model openai/gpt-oss-20b

If you change the port, point your client’s base_url at the same port (for example http://localhost:8080/v1).

Passing extra vLLM parameters

Any additional arguments are forwarded directly to vLLM, so you can tune serving behavior such as context length or data type. There are two ways to supply them.

Inline, after the launcher options:

Terminal window
vllm-launch --model openai/gpt-oss-20b --max-model-len 8192

Persistently, in a config file at ~/.local/share/vLLM/vllm-launch.conf. This file does not exist by default — create it and add your arguments as a Bash array:

Terminal window
VLLM_EXTRA_ARGS=(--max-model-len 8192 --dtype float16)

Use += to append to the default arguments instead of replacing them:

Terminal window
VLLM_EXTRA_ARGS+=(--max-model-len 8192)

To see all launcher options at any time, run:

Terminal window
vllm-launch --help

Where models are stored

vllm-launch looks for models in two locations:

LocationPath
System models/var/cache/models
User models~/.local/share/vLLM/models

You can place a downloaded model in either directory and launch it by passing its path or ID to --model:

Terminal window
vllm-launch --model /var/cache/models/my-model

Troubleshooting

Connection refused

Make sure the server is running:

Terminal window
curl http://localhost:8001/health

Summary

In this playbook, you learned how to:

  • Start containerized vLLM with ROCm support on the integrated GPU
  • Start a vLLM server with OpenAI-compatible API endpoints on port 8001
  • Send prompts with vllm-prompt
  • Make API calls to the vLLM server using both streaming and non-streaming requests
  • Troubleshoot common issues with server startup, memory, and client connections

You now have a containerized vLLM deployment for serving large language models with optimized performance on the integrated GPU.

Next Steps

  • Try different models — Use vllm-launch --model <model> to experiment with different LLMs and compare performance (see Choosing and Configuring a Model).
  • Build an application — Use the OpenAI-compatible API to integrate vLLM into a Python app, chatbot, or automation workflow.
  • Fine-tune and serve — Fine-tune a model using LoRA or QLoRA, then deploy it with vLLM for optimized inference.

Additional Resources

Need help with this playbook?

Run into an issue or have a question? Open a GitHub issue and our team will take a look.