Transformers pipeline: run open models in Python, on CPU or GPU

Before you start

You need the setup from lesson 1: a Hugging Face account and a token saved with hf auth login. For the Gemma 2 example, you also need your access request approved. Transformers is tested on Python 3.11 or later with PyTorch 2.6 or later. PyTorch is the math library that does a model's number crunching, and Transformers is the Hugging Face library that loads Hub models on top of it.

A pipeline wraps the three jobs every model run needs. It turns your input into tokens, runs the model, then turns the output back into labels or text. That run is called inference: using a trained model on new input, as opposed to training it. Each task, such as text-classification or text-generation, gets its own pipeline with the right steps built in. For finer control, Transformers also has model classes such as AutoModelForCausalLM, which the chat templates page uses. The pipeline is the shorter road, and this lesson stays on it.

The same call covers audio and images. The pipeline tutorial transcribes a speech clip with openai/whisper-large-v3 under the automatic-speech-recognition task, and labels a photo with google/vit-base-patch16-224 under image-classification. This lesson sticks to text.

A GPU is optional. The classifier and GPT-2 in this lesson are small enough for a laptop CPU, and the device table further down shows what changes once a GPU is present.

Transformers reached version 5 in January 2026. Run pip show transformers before you copy an example from an older tutorial: later releases raised the oldest supported Python and changed what a retired pipeline task does.

Install Transformers and run your first pipeline

The installation page uses uv, a fast Python package manager, and says to drop uv from each command if you prefer pip. Its install steps end with a check command, the final line below. It downloads the default sentiment model and labels one sentence.

The first call to any model downloads its files into the cache from lesson 1, under ~/.cache/huggingface/hub unless you moved it. Later calls load from that cache, so only the first run of each model waits on the network.

Every pipeline returns plain Python data. A classifier gives a list of dictionaries, each with a label and a score between 0 and 1. A score near 1 means the model is sure of its label. A text generator gives dictionaries with a generated_text key. Pass a list of inputs and you get a list back, one result per input.

uv venv .env
source .env/bin/activate
uv pip install "transformers[torch]"
python -c "from transformers import pipeline; print(pipeline('sentiment-analysis')('hugging face is the best'))"

1. Create a virtual environment with uv venv .env and activate it with source .env/bin/activate. On Windows, run the activate script inside .env\Scripts instead. Every later command assumes the environment is active.
2. Install with uv pip install "transformers[torch]", which brings PyTorch along. On a machine without an NVIDIA GPU, the docs give a CPU-only route: uv pip install torch --index-url https://download.pytorch.org/whl/cpu, then uv pip install transformers.
3. Run the check command. In the docs' example it prints one result: a POSITIVE label with a score just under 1.
4. Name the model instead of taking the default: pipeline(task="text-classification", model="distilbert/distilbert-base-uncased-finetuned-sst-2-english"). Call it with one string, or with a list of strings when you have several.
5. Switch to generation with pipeline(task="text-generation", model="google/gemma-2-2b") and call it with the start of a sentence. Gemma 2 is gated, so this step needs your approved access and saved token.
6. Read the result. In the docs' example, the reply repeats the whole prompt about baking a cake, then adds "1. the right ingredients 2. the". The prompt is included by default. Add return_full_text=False to get only the new words.

Chat with an instruction model

An instruction model, often marked Instruct in its name, is one trained further to follow requests and hold a conversation. Chat models take a list of messages, each a dictionary with a role and content. Start with one user message. A system message is optional and tells the model how to behave.

The code is the chat page's own example. It uses SmolLM2-1.7B-Instruct, an Apache 2.0 model with no access request. dtype="auto" loads the weights in the precision the model's config names, and device_map="auto" places them on the fastest device found. That second setting needs the Accelerate library: pip install -U accelerate.

The reply is not a plain string. response[0]["generated_text"] holds the whole chat, your messages plus the new one, so [-1]["content"] reads the answer. max_new_tokens=512 caps how many tokens the model adds to it. To keep talking, take that list, append the next user message and call the pipeline again. A chat can grow until it fills the model's context window, the most tokens it reads at once, or until memory runs out.

Behind each call, the pipeline applies a chat template: the model's own way of marking who said what, using control tokens the model learned in training. Two models from the same Mistral-7B base mark turns differently, as the table shows. With the wrong tokens, these models perform drastically worse.

The pipeline also appends the start of an assistant turn for you, a setting called add_generation_prompt. Leave it out and a model may simply continue your message. When your last message has the assistant role, the pipeline treats it as the opening of the reply and carries on from there. This is a prefill, and it helps when you know how the answer should begin.

With no code at all, transformers chat Qwen/Qwen2.5-0.5B-Instruct opens a chat in the terminal. It uses transformers serve under the hood, so the chat page asks you to start transformers serve first.

chat = [
    {"role": "system", "content": "You are a helpful science assistant."},
    {"role": "user", "content": "Hey, can you explain gravity to me?"}
]

import torch
from transformers import pipeline

pipeline = pipeline(task="text-generation", model="HuggingFaceTB/SmolLM2-1.7B-Instruct", dtype="auto", device_map="auto")
response = pipeline(chat, max_new_tokens=512)
print(response[0]["generated_text"][-1]["content"])

Choose the device, precision and batch size

Leave device unset and the pipeline takes the first accelerator it finds: an NVIDIA GPU through CUDA, an Apple Silicon GPU through MPS, or an Intel GPU through XPU. Each is the vendor's route for running math on its GPU. With none of them present, the pipeline falls back to the CPU. Set the device yourself when a script must behave the same on every machine.

Memory decides whether a model runs at all. Every parameter is a number, and the dtype, the number format, sets its size: 4 bytes in float32, 2 in float16 or bfloat16. An 8B model, one with 8 billion parameters, needs about 32GB in float32, by the chat page's count. Half precision halves that. On hardware that supports it, bfloat16 keeps more range than float16. For most models, the pipeline tutorial says, the quality lost to half precision is negligible, and larger models lose least.

By default, Transformers loads the dtype written in the model's config.json. Pass dtype="auto" or a type such as torch.bfloat16 to choose. To go lower, the pipeline accepts a model quantized to 8 or 4 bits with the bitsandbytes library, passed through model_kwargs. The tutorial's own example loads google/gemma-7b that way, in bfloat16 with 8-bit weights and device_map="auto", in a single call. Before you fetch anything big, the local LLM hardware lesson sets out memory needs by model size.

Speed follows memory too. A model reads every active parameter from memory for each token it writes. That is why the chat page names quantizing and faster memory as the levers, and says extra compute power does not meaningfully help.

A mixture-of-experts model bends that rule. It splits its layers into many expert blocks and sends each token through only a few, so Mixtral, Qwen2MoE and GPT-OSS read a small share of their parameters per token. They need less memory bandwidth and can beat a regular model of the same size on speed, though every parameter still has to fit in memory.

The chat page names one more lever, speculative decoding: a smaller model drafts tokens and the large one checks them, so it can accept several per step. It helps little with mixture-of-experts models, because each drafted token wakes more of their parameters.

Batching sends several inputs through the model at once with batch_size. It is off by default because it is not always faster. Skip it on a CPU and when each answer has to come back quickly, and plan for out-of-memory errors when you use it. Measure on your own model, data and hardware before you keep it.

For a large dataset, give the pipeline an iterator instead of one big list: a generator function, or a dataset column wrapped in KeyDataset. The pipeline then works through the data without first holding all of it in memory.

Lysandre's chat CLI tip and Andrej Karpathy's warning on special tokens

Lysandre, Chief Open-Source Officer at Hugging Face, posted on X in May 2025 that version 4.52 made transformers chat plus a model id the whole command, with generation settings typed straight after it. Today's chat page writes them as key=value pairs, as in transformers chat Qwen/Qwen2.5-0.5B-Instruct do_sample=False max_new_tokens=10, and transformers chat -h lists the rest. Get a prompt right there before you write pipeline code around it.

Andrej Karpathy, an AI researcher and educator who was a founding member of OpenAI, warned on X in August 2024 that tokenizers parse special tokens out of plain text. A user who types <|endoftext|> into your app can slip a control token into the model's input, much like SQL injection. His fix was add_special_tokens=False plus split_special_tokens=True, with the special tokens added by your own code. For chat models, he thought a chat template would also do.

The warning still holds. In Transformers 5.22.1, the tokenizer reference still defaults split_special_tokens to False. If your app tokenizes text that users type, pass split_special_tokens=True and print the tokens it produces, as he advised.

Screenshots of X's own embeds of the posts, taken on 5 November 2026, with the time in UTC. X cuts Andrej Karpathy's post short with Show more. Select one to open the post on X.

Common mistakes

• Leaving out the model id. Each task falls back to a default model, which may not suit your text, your memory or your license check.
• Calling Gemma 2 before the authors approve your request, or with no token saved. The download stops until both are in place.
• Asking a base model to chat. google/gemma-2-2b continues text, while its chat-trained sibling is google/gemma-2-2b-it, and the message list only gets sensible replies from the second.
• Typing control tokens into a prompt by hand. A single wrong token sends the model off its training format, so pass the message list instead.
• Turning on batch_size on a CPU, where the pipeline docs advise against batching at all.
• Setting trust_remote_code=True with no revision. That flag runs Python code from the model repo, and pinning a commit hash with revision= keeps that code from changing under you.
• Using device_map="auto" before Accelerate is installed. One pip install -U accelerate in the same environment fixes it.

Try it yourself

Start python inside your virtual environment and type these lines from the pipeline tutorial. GPT-2 is public and small, so you need no token or GPU. The Python prompt prints the four continuations the call returns.

from transformers import pipeline

pipeline = pipeline(task="text-generation", model="openai-community/gpt2")
pipeline("the secret to baking a good cake is", num_return_sequences=4, return_full_text=False)

• Run the call twice. Do the four continuations change between runs?
• Drop return_full_text=False and compare what comes back.
• Load the DistilBERT classifier from the steps and pass it a list of three short reviews. Which one scores lowest?
• Run the SmolLM2 chat once with device="cpu" and once with the device unset, and time one reply each way.
• Add an assistant message that starts with "Answer:" to the end of the chat. How does the reply pick up from it?
• Continue the SmolLM2 chat the way the chat page does: append a user message to the returned list and call the pipeline again. Does the second reply build on the first?
• Run hf cache ls from lesson 1. Which models did this lesson add, and how much disk does each one take?

Next

You can now run any model your machine holds. Many it can't: a 70B model needs about 140GB at 2 bytes per number, far beyond a laptop. Lesson 3 sends those to hosted providers by the same Hub ids, billed to your Hugging Face account. The message lists from this lesson carry over unchanged, and the token you saved in lesson 1 is the only key the calls need.

Frequently asked questions

Sources

1. Transformers: Pipeline (read November 5, 2026)https://huggingface.co/docs/transformers/pipeline_tutorial
2. Transformers: Installation (read November 5, 2026)https://huggingface.co/docs/transformers/installation
3. Transformers: Chat basics (read November 5, 2026)https://huggingface.co/docs/transformers/conversations
4. Transformers: Chat templates (read November 5, 2026)https://huggingface.co/docs/transformers/chat_templating
5. Transformers: Loading models (read November 5, 2026)https://huggingface.co/docs/transformers/models
6. Hugging Face: google/gemma-2-2b model page (read November 5, 2026)https://huggingface.co/google/gemma-2-2b
7. Hugging Face: HuggingFaceTB/SmolLM2-1.7B-Instruct model page (read November 5, 2026)https://huggingface.co/HuggingFaceTB/SmolLM2-1.7B-Instruct
8. Hugging Face: distilbert-base-uncased-finetuned-sst-2-english model page (read November 5, 2026)https://huggingface.co/distilbert/distilbert-base-uncased-finetuned-sst-2-english
9. Transformers: Tokenizer reference (read November 5, 2026)https://huggingface.co/docs/transformers/main_classes/tokenizer
10. Transformers release v5.0.0rc0 (read November 5, 2026)https://github.com/huggingface/transformers/releases/tag/v5.0.0rc0
11. Transformers release v5.0.0 (read November 5, 2026)https://github.com/huggingface/transformers/releases/tag/v5.0.0
12. Transformers release v5.19.0 (read November 5, 2026)https://github.com/huggingface/transformers/releases/tag/v5.19.0
13. Transformers release v5.21 (read November 5, 2026)https://github.com/huggingface/transformers/releases/tag/v5.21
14. Transformers release v5.22.0 (read November 5, 2026)https://github.com/huggingface/transformers/releases/tag/v5.22.0
15. Lysandre on X: transformers chat from v4.52 (read November 5, 2026)https://x.com/LysandreJik/status/1925124636498645407
16. Andrej Karpathy on X: special tokens and untrusted user input (read November 5, 2026)https://x.com/karpathy/status/1823418177197646104
17. Hugging Face on YouTube: The pipeline function (read November 5, 2026)https://www.youtube.com/watch?v=tiZFewofSLM