Local models

Connect Orek to a model running on your computer with Ollama, llama.cpp, vLLM, or LM Studio. You can select that model for the current conversation, keep it selected across sessions, or include it in automatic routing alongside cloud models.

Start with a running local server and a downloaded model. Every local model needs its own [[models]] entry in config.toml; setup discovers and writes these rows for Ollama and llama.cpp. orek models lists configured rows. Connection profiles are ollama, llama_cpp, vllm, and lm_studio; set provider_type when using a different connection name. See Custom model definitions for every model field.

A local model can take minutes to read a long prompt before it replies, so Orek waits longer for local servers than for cloud providers. This applies to all four local profiles, including servers on your network, and to a generic compatible server whose base_url is localhost, a 127.x.x.x address, or [::1]. Use an address clients can connect to rather than 0.0.0.0. timeout_secs overrides the profile. Each local connection admits one request at a time by default; adjust max_concurrent_requests for a server that can handle more. Waiting and active requests can be canceled.

Quick start: Ollama

Install and start Ollama using the official setup guide. Download a model in a terminal:

ollama pull qwen2.5-coder:14b

This example uses Qwen2.5 Coder 14B, which the Ollama catalog lists with tool support. Choose a model your hardware can run, and verify its tool calls on a small task before using it for larger changes.

Ollama’s default address is http://localhost:11434. Set base_url to the server address without a path; Orek adds /v1 itself.

In Orek, /setup provider ollama discovers installed tags asynchronously and saves the selected model’s row without changing routing pins. Select a tag and press L to consent to loading it when its loaded context is unavailable; R refreshes the list. The edited server address is saved.

From a shell, orek setup --provider ollama --yes discovers installed models and pins the selected model (--model <tag> chooses one explicitly). Headless setup requires a loaded context and stops with an actionable error when it cannot read one. Load the model in Ollama first, or provide verified metadata:

orek providers setup ollama --yes --credential-store none --model-id qwen2.5-coder:14b --context-window 32768

Discovery uses /api/tags, /api/ps, and /api/show; loaded context takes precedence over the trained maximum. It saves advertised tools, vision, and thinking levels, without guessing an output cap. Existing selectors and wire IDs keep their physical rows unchanged. A loaded context below about 8k tokens produces a warning naming OLLAMA_CONTEXT_LENGTH. The equivalent manual project configuration in .orek/config.toml is:

[providers.ollama]
base_url = "http://localhost:11434"

[[models]]
id = "ollama/qwen2.5-coder:14b"
provider = "ollama"
api_model_id = "qwen2.5-coder:14b"
display_name = "Qwen2.5 Coder 14B (Ollama)"
context_window = 32768
supports_tools = true
supports_streaming = true
capability_source = "user_override"
capability_confidence = "high"

id is the name you select in Orek, api_model_id must match the Ollama tag, and context_window must match the context size set on your Ollama server; setting it in Orek does not resize the server. capability_confidence = "high" tells automatic routing that you have checked the model; the default, low, ranks it below checked models.

Verify the connection

/model list

orek providers check

The first command should list ollama/qwen2.5-coder:14b. The second checks that your configured providers respond.

With Orek open in your project, select the configured model, then ask a question:

/model ollama/qwen2.5-coder:14b

Read the README and summarize how to run this project.

Confirm that the model reads the file and returns a useful answer. If the provider responds but the agent cannot call tools, check the model’s tool support and chat template.

Quick start: llama.cpp

For automatic discovery, run orek providers setup llama_cpp --yes --credential-store none. Use --base-url for a different server. Orek reads the served model and loaded context instead of guessing. For another connection name, add --provider-type llama_cpp. The setup wizard also offers llama.cpp directly. At startup, a smaller server context lowers Orek’s effective window and is reported as a warning. The saved model configuration is retained.

llama.cpp provides an OpenAI-compatible server. Install it and obtain a compatible GGUF model using the llama.cpp server documentation.

Start the server with your model file:

llama-server -m models/qwen2.5-coder-14b.Q4_K_M.gguf --host 127.0.0.1 --port 8080 --alias qwen2.5-coder-14b --ctx-size 32768 --jinja

Replace the file path with your downloaded model. --alias gives the API a stable model name. Tool calls require a compatible Jinja chat template; some models need an explicit template file. Check the server documentation for your model’s setup.

For the connection name llamacpp, run orek providers setup llamacpp --provider-type llama_cpp or choose llama.cpp in the setup wizard. Provider-only setup keeps the routing pins. Discovery reads /props and /v1/models, writes the loaded context and advertised capabilities, and keeps existing rows unchanged. The equivalent manual .orek/config.toml is:

[providers.llamacpp]
provider_type = "llama_cpp"
base_url = "http://127.0.0.1:8080/v1"

[[models]]
id = "llamacpp/qwen2.5-coder-14b"
provider = "llamacpp"
api_model_id = "qwen2.5-coder-14b"
context_window = 32768
supports_tools = true
supports_streaming = true
capability_source = "user_override"
capability_confidence = "high"

The api_model_id matches the server’s --alias. Verify the catalog and connection, then try the model:

/model list

orek providers check

/model llamacpp/qwen2.5-coder-14b

Read the README and summarize how to run this project.

Quick start: vLLM

Install vLLM and download a compatible model using the vLLM documentation. Start its OpenAI-compatible server:

vllm serve /path/to/qwen2.5-coder-14b --host 127.0.0.1 --port 8000 --served-model-name qwen2.5-coder-14b

Replace the model path and served name with your model. Enable the tool-call flags required by that model before declaring tool support. The equivalent project configuration is:

[providers.vllm]
provider_type = "vllm"
base_url = "http://127.0.0.1:8000/v1"

[[models]]
id = "vllm/qwen2.5-coder-14b"
provider = "vllm"
api_model_id = "qwen2.5-coder-14b"
context_window = 32768
supports_tools = true
supports_streaming = true
capability_source = "user_override"
capability_confidence = "high"

Check GET /v1/models if the served model ID is unclear, then run orek providers check and select vllm/qwen2.5-coder-14b with /model.

Quick start: LM Studio

Install LM Studio, download a model, load it, and start its local server. Use the model identifier shown by LM Studio or returned by GET /v1/models. Declare the server and model in .orek/config.toml:

[providers.lmstudio]
provider_type = "lm_studio"
base_url = "http://localhost:1234/v1"

[[models]]
id = "lmstudio/local-model"
provider = "lmstudio"
api_model_id = "local-model"
context_window = 32768
supports_tools = true
supports_streaming = true
capability_source = "user_override"
capability_confidence = "high"

Match context_window to the loaded model’s server context and set supports_tools only after verifying tool calls. Run orek providers check, then select lmstudio/local-model with /model.

Keep a local model selected

To use the Ollama model across sessions:

/config set routing.pinned_model ollama/qwen2.5-coder:14b

Inside the TUI, /model <selector> selects a model and /model auto returns to automatic routing. You can persist the provider preference with:

/config set routing.pinned_provider ollama

routing.pinned_model always uses that model, while routing.pinned_provider limits automatic routing to that provider. If both are set and conflict, Orek warns and uses the pinned provider’s default model. Model and provider pins do not block network access or subagent fallback; see Prepare a disconnected workstation for offline use.

Route between local and cloud models

Routes let you prefer a local model for particular tasks. For example, create .orek/routes/local-default.toml:

[[routes]]
name = "simple/lookup"
description = "Short factual lookups and straightforward explanations"
sample_queries = [
    "What does this type do?",
    "Explain this error message",
    "Explain this regex"
]
tags = ["what", "explain", "lookup"]
provider_preference = ["ollama"]
preferred_models = ["ollama/qwen2.5-coder:14b"]
complexity_bias = "low"
enabled = true

This replaces the built-in simple/lookup route in full. Orek favors the preferred model and provider, but can still choose another. Configure your cloud models separately if you want them available for other tasks.

Check that the route loaded and that its keywords match:

orek routes list
orek routes test "explain this regex"

routes test shows the matching route’s settings. It does not predict which model a real request will use; check /routing in Orek for the latest choice. See Routing for route fields, selection rules, and diagnostics.

Prepare a disconnected workstation

Selecting a local model runs inference on your machine, but does not automatically make every Orek feature offline. Use this checklist when preparing a local-only or disconnected workstation:

Use operating-system network controls if you need an enforced no-egress boundary.

Model selection tips

Evaluate a model on a small example from your project before assigning a long task:

1. Ask it to locate and read a file.
2. Ask for a small change, then inspect the diff and test result.
3. Check whether it can handle the context size and concurrent requests your workflow needs.

Set supports_tools = true only for models that can call tools; the setting describes the model and cannot add tool support. It defaults to false, and automatic routing skips a model without tool support when tools are needed. If you pick such a model yourself (--model or /model), it works without tools: earlier tool calls stay in the conversation, and Orek shows a notice once per session.

Performance knobs

Ollama

Use ollama ps to inspect loaded models. Keeping a model loaded can reduce startup delay, while larger context windows and parallel requests require more memory. Ollama documents OLLAMA_KEEP_ALIVE, OLLAMA_NUM_PARALLEL, and context settings in its FAQ; apply them through the startup method for your operating system.

GPU use

For the chat model, check your server’s logs or model status (for example, ollama ps) to confirm that it uses the GPU. GPU support for that model comes from how you install and start the server, not from an Orek setting.

Orek’s own local helper models for code search and command auto-review use your GPU automatically and fall back to the CPU. See GPU acceleration.

Context length

If the server runs out of memory on long turns, lower both the server’s context size and Orek’s matching context_window (or reduce concurrent requests).

Combining with other features

• Retrieval can supply relevant code to either local or cloud chat models. Its embedding model is configured separately.
• Subagents can use different models from the main agent. Allow for their extra parallel requests when planning memory use, and note that they can fall back to other configured providers.
• Plan mode works with local models. You can switch models with /model before implementation; automatic model selection never switches you in or out of Plan mode.