Getting Started with Ollama

Overview

Ollama is a popular lightweight tool for running large language models locally. It handles model downloading, quantization, and serving behind a simple command-line interface and desktop app, so you can go from zero to chatting with an LLM in minutes.

This playbook walks you through installing Ollama, pulling the GPT-OSS 20B model, and having a conversation with it, through both the terminal and the desktop app.

What You’ll Learn

• How to install and launch Ollama on your system
• Pull and run the GPT-OSS 20B model locally
• Chat with models using the CLI
• Query models programmatically through the REST API

if command -v rocminfo >/dev/null 2>&1 && rocminfo | grep -q gfx; then
  echo "OK: GPU visible to ROCm"
else
  echo "GPU not visible via rocminfo" >&2
  exit 1
fi

sudo apt install pipx
pipx ensurepath
pipx install amd-debug-tools

amd-ttm

amd-ttm --set <NUM>

Installing Software Prerequisites

Installing Ollama

ollama --version

curl -fsSL https://ollama.com/install.sh | sh

ollama --version

Pulling Your First Model

Ollama manages models through a registry similar to container images. To download GPT-OSS 20B:

ollama pull gpt-oss:20b

This downloads the model weights to your local machine (approximately 12 GB). The download only happens once, and subsequent runs load the model from disk.

You can confirm the model is available with:

ollama list

You should see gpt-oss:20b in the output along with its size and last-modified date.

Model Naming

Ollama model names follow the format name:tag. The tag usually indicates the parameter count or quantization variant. Some useful commands for managing models:

Chatting from the Terminal

Launch an interactive chat session directly from the command line:

ollama run gpt-oss:20b

Ollama loads the model into memory and drops you into a prompt. Try asking it something:

>>> What is the capital of France and why is it historically significant?

The model streams its response token-by-token directly in the terminal. Type /bye or press Ctrl+D to exit the session.

Using the REST API

After installation, Ollama runs as a background service and exposes a REST API on http://localhost:11434 that you can use to integrate models into your own applications and scripts.

Generate a Response in Terminal

curl http://localhost:11434/api/generate -d '{"model": "gpt-oss:20b", "prompt": "Explain GPU acceleration in two sentences.", "stream": false}'

curl.exe http://localhost:11434/api/generate -d '{"model": "gpt-oss:20b", "prompt": "Explain GPU acceleration in two sentences.", "stream": false}'

The response is a JSON object containing the model’s output in the response field.

Python Example

Now that we can hit the Ollama API programmatically, let’s call it from Python.

Create a Virtual Environment in Terminal

sudo apt install -y python3-venv
python3 -m venv ollama-env
source ollama-env/bin/activate
pip install requests

python -m venv ollama-env
ollama-env\Scripts\activate
pip install requests

Create a Python file

In the same directory, use VS Code or another editor to create a .py file and copy the following code into it. Then, run the file in your activated environment with python your_file_name.py

import requests

response = requests.post(
    "http://localhost:11434/api/generate",
    json={
        "model": "gpt-oss:20b",
        "prompt": "Write a haiku about local AI inference.",
        "stream": False,
    },
)

print(response.json()["response"])

Key API Endpoints

For the full API reference, see the Ollama API documentation.

Next Steps

• Try different models: Browse the Ollama model library to explore hundreds of available models, from small coding assistants to large reasoning models.
• Create custom models: Use a Modelfile to set custom system prompts, temperature, and other parameters for a tailored experience.
• Build with the API: Use the Python or JavaScript client libraries to integrate Ollama into your applications.
• Connect to frontends: Pair Ollama with tools like Open WebUI for a feature-rich chat interface with search, personas, and document upload.

For more information, check out the Ollama documentation.