Ollama API: call a local model from your code

Before you start

You need Ollama running, as set up in lesson 1, and one model on disk. The examples use gemma4, the model in most of Ollama's API examples, so pull it first or swap in any tag you already have. For the code you also need Python 3 with two packages, ollama and openai.

An API here is a set of web addresses that take JSON and return JSON. The local server answers on port 11434. Requests from your own machine need no key, and the server listens on 127.0.0.1 only, so another computer cannot reach it until you change OLLAMA_HOST.

Ollama does not version this API strictly. Its introduction page says the API is expected to stay stable and backwards compatible, and there are official libraries for Python and JavaScript.

There are two ways in, and the choice decides what you can reach. The native routes under /api expose everything Ollama does, including per-request options, keep_alive and thinking control. The /v1 routes copy OpenAI's format, so code and tools written for OpenAI run unchanged, minus a few fields.

The same API also runs in Ollama's cloud. For a cloud model the base becomes https://ollama.com/api, and each request carries an Authorization: Bearer header with an OLLAMA_API_KEY from your account settings. That key belongs on a server, never in browser code or in source control. Everything below uses the local server, where no key exists to leak.

The API grew in steps, each announced on Ollama's blog. A tutorial written before a step will not mention it, so check its date against this list.

ollama pull gemma4
pip install ollama openai

Send your first request with curl

Start with the simplest endpoint. The generate endpoint, /api/generate, takes one prompt and returns one answer. Adding "stream": false makes the server wait and send a single JSON object, which is easier to read the first time. The chat endpoint, /api/chat, takes a list of messages instead, each with a role and its content.

Neither endpoint remembers earlier calls. With /api/chat you send the whole conversation every time: the earlier questions, the assistant's earlier answers, then the new question. Leave a turn out and the model never sees it. Every turn also makes the prompt longer, and the prompt_eval_count in each reply shows it growing.

Both endpoints accept more than text. An options object holds the settings for one request: temperature, top_p, a seed for repeatable output, stop sequences, num_predict to cap the length of the reply and num_ctx for the context length. For a model that reads pictures, images takes a list of base64-encoded files.

Thinking models add one more field. With think you ask for the model's reasoning or switch it off, and some models, such as gpt-oss, take a level like "low" instead of true or false. The reasoning comes back in its own thinking field, apart from the answer. To see which values a model accepts, ask /api/show.

A request with a model name and nothing else is useful too. It loads the model without generating anything, so the first real question does not wait for the load.

# is the server up, and which models does it have?
curl http://localhost:11434/api/tags

# one prompt, one answer
curl http://localhost:11434/api/generate -d '{
  "model": "gemma4",
  "prompt": "Why is the sky blue?",
  "stream": false
}'

# a conversation: send the messages so far
curl http://localhost:11434/api/chat -d '{
  "model": "gemma4",
  "messages": [
    {
      "role": "user",
      "content": "why is the sky blue?"
    }
  ],
  "stream": false
}'

1. Run the /api/tags call. It returns a models list with each model's name, its size on disk in bytes and details such as parameter_size and quantization_level. An error here means the server is not running.
2. Run the /api/generate request. The answer is in the response field, and done is true.
3. Run the /api/chat request. The answer now sits in message.content, and its role is assistant.
4. Append that assistant message and a follow-up question to messages, then send the request again. The model can now refer to its own first answer.
5. Remove "stream": false from either request and run it once more. The reply arrives as many lines, one JSON object per line, each holding the next piece of text.

Call it from Python: the Ollama library or the OpenAI SDK

The official Python library wraps the same endpoints. Its chat() function takes a model name and a list of messages. With stream=True the call returns chunks instead of one answer, and each chunk holds the next piece of the reply in message.content. The loop below prints them as they arrive, so the first words show up before the model has finished.

Streaming is the server's default, and it uses newline-delimited JSON: one complete JSON object per line, sent as soon as there is more text. A chunk carries its piece of the answer and done: false. The last chunk has done: true, a done_reason such as stop, and the statistics for the whole run, as the streaming page shows.

Those statistics answer most questions about speed. Every duration is in nanoseconds. Tokens per second, the usual speed figure, is eval_count divided by eval_duration, times one billion. A token is a piece of text, often part of a word, and what tokens are covers why models count in them.

The generate reference prints one reply in full as its example. It came from gemma4 and ends with done_reason: stop, prompt_eval_count: 11 and eval_count: 18, so that prompt cost eleven input tokens and the answer eighteen.

Chat is one call of many. Next to it sit generate, list, show, create, copy, delete, pull, push, embed and ps, which cover what the command line does, so a script can pull a model before it uses it.

To reach a server on another address, build a client yourself: Client(host='http://localhost:11434') takes the host and any extra headers.

Plenty of tools already speak OpenAI's API, and Ollama answers the same requests under /v1. You change the base URL and keep the rest of the code. The client refuses to start without an API key, and Ollama ignores whatever you pass, so any string works.

The second snippet is the compatibility page's own example with the model swapped for gemma4. The routes it lists are /v1/chat/completions, /v1/completions, /v1/models, /v1/embeddings and /v1/responses, plus audio routes for speech, transcription and translation.

Chat completions ignore logprobs, tool_choice, logit_bias, user and n, and images must be base64 data, not URLs. The Responses route is stateless, so previous_response_id and conversation don't work and you resend the history each time.

Nothing in the OpenAI API sets the context length. The workaround is a new model built from a Modelfile with a larger num_ctx, which any OpenAI client can then name in model.

JavaScript works the same way. The OpenAI client there takes baseURL: "http://localhost:11434/v1/" and apiKey: "ollama", and the rest of the call does not change.

Code written for Claude has a route as well. Ollama supports a subset of Anthropic's Messages API, so the Anthropic client works with its base URL set to http://localhost:11434. That client adds /v1/messages to the address by itself.

from ollama import chat

stream = chat(
  model='gemma4',
  messages=[{'role': 'user', 'content': 'Why is the sky blue?'}],
  stream=True,
)

for chunk in stream:
  print(chunk['message']['content'], end='', flush=True)

# the same model through the OpenAI SDK
from openai import OpenAI

client = OpenAI(
    base_url='http://localhost:11434/v1/',
    api_key='ollama',  # required but ignored
)

chat_completion = client.chat.completions.create(
    messages=[
        {
            'role': 'user',
            'content': 'Say this is a test',
        }
    ],
    model='gemma4',
)
print(chat_completion.choices[0].message.content)

Save a system prompt in a Modelfile

A Modelfile is a short text file that builds a new model from one you already have. The FROM line names the base model. Each PARAMETER line sets an option, such as num_ctx, the context length in tokens, or temperature, which controls how random the wording gets. The SYSTEM line holds the system prompt that every request to the new model carries.

ollama create reads the file and registers the name. From then on the new model works everywhere a model name does: in ollama run, in /api/chat and in the OpenAI SDK. The SYSTEM line is a prompt like any other, and the system prompt lesson shows how to write a good one.

A model pulled from Hugging Face takes the same settings another way. A file named system in its repository sets the system prompt, and a JSON file named params sets the parameters.

The Modelfile reference lists more instructions: TEMPLATE for the full prompt template, MESSAGE for message history, LICENSE and REQUIRES for the minimum Ollama version.

FROM gemma4
PARAMETER num_ctx 8192
PARAMETER temperature 0.3
SYSTEM You answer questions about the user's notes in plain English. If the notes do not say, answer that you do not know.

1. Save the four lines above as Modelfile in an empty folder.
2. Run ollama create notes-helper -f ./Modelfile.
3. Run ollama run notes-helper and ask it something your notes would not cover.
4. Print what you built with ollama show --modelfile notes-helper. Running the same command on gemma4 shows the template and parameters your file inherits.

Georgi Gerganov on why local models seem to fail

Georgi Gerganov builds ggml-org, the GitHub organization behind llama.cpp, and works as an engineer at Hugging Face. In a reply on X in March, he answered a question about local models. He recommended Qwen 3.5 for its range of sizes. Then he warned that most problems people hit sit outside the model: in the program that drives it, in the chat template and in the way the prompt is built.

His advice was to start with full-quality models that fit your hardware, and to know what the program around the model does. Hooking Claude Code or Codex to a local model and expecting magic, he wrote, does not work. Quantize for speed only once things click. For his own tests he picked Q8_0 variants of most models to keep most of the original quality.

The API puts the template in your hands. On /api/chat, Ollama applies the model's own chat template to your messages. The raw option on /api/generate skips it, so leave raw off unless you rebuild the template yourself. ollama show --modelfile prints the template a model uses. When a local model fails a task, read that template and the request your client sends before you blame the model.

A screenshot of X's own embed of the post, taken on 5 November 2026, with the time in UTC. X cuts the post short with Show more. Select it to open the post on X.

Common mistakes

• Parsing a streamed reply as one JSON document. Without "stream": false the body is many objects, one per line, so read it line by line.
• Sending only the newest message to /api/chat. The server keeps no conversation, so the model forgets every turn you leave out.
• Calling the API from a web page or an extension and getting blocked. By default Ollama accepts browser requests only from 127.0.0.1 and 0.0.0.0 origins. Add others to OLLAMA_ORIGINS, for example chrome-extension://*.
• Ignoring done_reason. A reply capped by num_predict can stop mid-sentence, and done_reason is the field that says why the reply ended.
• Letting the model unload in the middle of a batch job. Each request can carry keep_alive: a negative value keeps the model loaded, and 0 unloads it right after the answer.
• Putting a cloud key in a desktop app or a web page. Calls to https://ollama.com/api need OLLAMA_API_KEY, so send them from your own server instead.
• Asking for JSON in the prompt and hoping. Set format to "json", or to a JSON schema object, and the reply comes back in that structure.
• Naming a model in /v1 code that ollama ls does not list. The local server runs only models on disk, so the fix is ollama pull, not a new key.

Try it yourself

• Send the same prompt to /api/generate twice, once with "stream": false and once without. How many lines does the streamed reply have?
• Work out tokens per second from eval_count and eval_duration in one response.
• List your models from Python with the library's list call and print each name.
• Change the OpenAI example's base_url back and forth between Ollama and a cloud service. Which lines changed?
• Add PARAMETER num_ctx to your Modelfile, create the model again and check the CONTEXT column in ollama ps while it runs.
• Send "keep_alive": 0 with a request and watch the model leave ollama ps straight after the answer.
• Ask a thinking model the same question with think on and off, and compare the thinking field with the answer.

Next: LM Studio runs the same kind of local server inside a desktop app, with a model browser and an OpenAI-compatible endpoint of its own. Lesson 3 sets it up and compares its lms command line with the Ollama commands you know now.

Frequently asked questions

Sources

1. Ollama Docs: API introduction (read November 5, 2026)https://docs.ollama.com/api/introduction
2. Ollama Docs: Generate a response (read November 5, 2026)https://docs.ollama.com/api/generate
3. Ollama Docs: Generate a chat message (read November 5, 2026)https://docs.ollama.com/api/chat
4. Ollama Docs: List models (read November 5, 2026)https://docs.ollama.com/api/tags
5. Ollama Docs: Streaming (read November 5, 2026)https://docs.ollama.com/api/streaming
6. Ollama Docs: OpenAI compatibility (read November 5, 2026)https://docs.ollama.com/api/openai-compatibility
7. Ollama Docs: Anthropic compatibility (read November 5, 2026)https://docs.ollama.com/api/anthropic-compatibility
8. Ollama Docs: Modelfile reference (read November 5, 2026)https://docs.ollama.com/modelfile
9. Ollama Docs: FAQ (read November 5, 2026)https://docs.ollama.com/faq
10. Ollama Docs: Windows (read November 5, 2026)https://docs.ollama.com/windows
11. GitHub: ollama/ollama-python (read November 5, 2026)https://github.com/ollama/ollama-python
12. Ollama blog: OpenAI compatibility (read November 5, 2026)https://ollama.com/blog/openai-compatibility
13. Ollama blog: Tool support (read November 5, 2026)https://ollama.com/blog/tool-support
14. Ollama blog: Structured outputs (read November 5, 2026)https://ollama.com/blog/structured-outputs
15. Ollama blog: Streaming responses with tool calling (read November 5, 2026)https://ollama.com/blog/streaming-tool
16. Ollama blog: Claude Code with Anthropic API compatibility (read November 5, 2026)https://ollama.com/blog/claude
17. Ollama blog: Claude Desktop support with Ollama (read November 5, 2026)https://ollama.com/blog/claude-desktop
18. Georgi Gerganov on X: the main issues with local models (read November 5, 2026)https://x.com/ggerganov/status/2038674698809102599