What Can You Do with Ollama?

Ollama is a local runtime and model manager for running supported language and multimodal models on your own computer.
It can download models, manage them locally, and expose an API that many OpenAI‑compatible clients can use.

By keeping your models on your own machine, you reduce cloud costs and ensure that your prompts are processed locally. Because the data never leaves your network, you avoid usage fees and gain tighter control over privacy.
However, this still requires suitable hardware, storage, and the responsible handling of outputs. You need enough RAM or GPU memory for inference, sufficient disk space for the model files, and you must monitor what the model generates to prevent misuse.

Getting Started

The installation command varies by operating system.
Follow the current instructions available at https://ollama.com/download.
A typical workflow includes:

ollama pull llama3.1
ollama run llama3.1

Remember that the exact model tag and resource requirements may differ. For example, a “llama3.1‑q8_0” tag denotes a quantized version that consumes less memory than the full‑precision model.
A local model does not automatically match a hosted model of the same family name. Even if both are called “llama3.1”, the locally stored version may be a different size, quantization, or fine‑tuning, which leads to distinct behavior.

Privacy and Limitations

With a local model, prompts are processed by the local Ollama service instead of being sent to a hosted provider. This eliminates the need to trust an external API with your data.
However, applications can still send data elsewhere through their own integrations, and local machines still need normal access controls, updates, and disk protection. A web front‑end, for instance, might forward logs to an analytics service, so you must configure firewalls and keep the operating system patched.

Local models can also make mistakes, misunderstand context, or perform poorly on specialized tasks. They generate output based on patterns in their training data, not on verified facts.
Before using them for decisions related to medical, legal, financial, or safety‑critical matters, evaluate their performance. Run targeted benchmarks, compare against ground‑truth answers, and consider involving domain experts to validate the results.

What You Can Build

Text and Code Assistance

An OpenAI‑compatible client can point at the local endpoint:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:11434/v1",
    api_key="ollama",
)

response = client.chat.completions.create(
    model="llama3.1",
    messages=[{"role": "user", "content": "Explain a database index simply."}],
)
print(response.choices[0].message.content)

This setup can support drafting, summarization, code explanation, and review, but it does not eliminate the need for tests or human review. The model may produce syntactically correct code that contains logical errors, so you should still run unit tests and inspect the output.

Document Analysis

A document pipeline can extract text and provide selected passages to a model:

import pymupdf
from openai import OpenAI

def analyse_document(path: str, question: str):
    doc = pymupdf.open(path)
    text = "\n".join(page.get_text() for page in doc)
    client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
    result = client.chat.completions.create(
        model="llama3.1",
        messages=[
            {"role": "system", "content": "You analyse the supplied document."},
            {"role": "user", "content": f"Document:\n{text}\n\nQuestion: {question}"},
        ],
    )
    return result.choices[0].message.content

For large documents, chunking and retrieval are usually preferable to sending the entire file in one request. By breaking the text into manageable pieces you stay within token limits and you can retrieve only the most relevant sections, which improves both speed and answer quality.

Images and Multimodal Models

Some Ollama models support images. Check the current model documentation and API format before relying on a particular model:

ollama pull llava

Multimodal output should be treated as an interpretation, not as a guaranteed description of every detail in an image. The model infers captions from visual patterns, so ambiguous or low‑resolution inputs can lead to hallucinations.

Retrieval‑Augmented Generation

Ollama can generate answers from context retrieved from a database such as PostgreSQL with pgvector. The retrieval layer must enforce tenant and access permissions before context reaches the model. This prevents unauthorized users from pulling private documents into the inference pipeline.

query_vector = client.embeddings.create(
    model="nomic-embed-text",
    input="What was decided at the last meeting?",
).data[0].embedding

cursor.execute("""
    SELECT content FROM documents
    ORDER BY embedding <=> %s::vector
    LIMIT 5
""", (query_vector,))

Local Developer Tools

A CLI can pass a Git diff or log to a local model for explanation. This lets you ask, for example, “What changed in this commit?” and receive a concise summary.
Avoid sending secrets, credentials, or unreviewed commands to an automated tool, and keep the model's output advisory. Treat the response as a suggestion that you must verify before acting on it.

Choosing a Model

Model size, quantization, context length, and hardware determine the result. Smaller models use fewer resources but may be less capable; larger models may require substantial RAM or GPU memory.
Do not assume a single RAM table is universal because tags and quantizations can change. Each model’s page lists its memory footprint, so you should consult that information before downloading.

Benchmark the models and prompts that matter to your application. Measure latency, token throughput, and answer quality on a representative test set. In many cases, a 7B–8B class model is a practical starting point, but the right choice depends on the task. If you need nuanced reasoning, a larger model may justify the extra cost.

Performance and Resource Use

Ollama can use supported GPUs, including Apple Metal and selected NVIDIA or AMD setups. Leveraging a GPU typically yields faster token generation than CPU‑only inference.
Actual token rates depend on model, quantization, context, drivers, and thermal limits. A hot GPU may throttle, reducing throughput, while an optimized driver can squeeze out extra performance.

Quantization reduces memory use by storing weights at lower precision. It often trades some quality for a smaller model, so compare outputs on a test set instead of assuming that one quantization is always best. Run side‑by‑side evaluations to see whether the speed gain outweighs any loss in accuracy for your use case.

Custom Modelfiles

A Modelfile can set a base model and parameters:

FROM llama3.1
PARAMETER temperature 0.7
PARAMETER top_p 0.9
SYSTEM You are a concise coding assistant.

ollama create helpful-coder -f Modelfile
ollama run helpful-coder

By customizing temperature and top‑p you control the randomness of the output, while the SYSTEM prompt defines the model’s persona.

Local and Cloud Models

Ollama's local API is separate from services such as OpenRouter or Groq. An OpenAI‑compatible Python client can call those hosted services directly, but that sends prompts to the provider and requires its authentication and terms.
Do not describe this as local inference or assume that Ollama automatically proxies every provider. When you point the client at a remote endpoint, you are effectively using a cloud model, not a locally hosted one.

The Bottom Line

Ollama makes local experimentation approachable. It lowers the barrier to trying out large language models without committing to expensive cloud credits.
It is useful for prototypes, private workflows, offline tools, and learning about model serving. Choose it when local control is valuable and the hardware and model quality meet your needs; choose a hosted service when you need capabilities or scale that local inference cannot provide.