Documentation v0.2.0

Highlights

What the engine does above the kernels; the math is ggml through llama.cpp.

Getting Started runtime install

1. Install taraengine

Install has the verified download (cosign and sha256), the .deb and .rpm packages, and the build from source.

2. Install the runtime

3. Start the engine and pull a model

gemma4:12b suits 16 GB of RAM. On a smaller machine, pull the model that taraengine doctor names, and use that name in step 4.

4. Ask it something

The chat request loads the model on first use. taraengine doctor shows the machine, the runtime and the daemon, and recommends the catalog's model for the RAM it finds: for the machine of the daemon when one answers, and for this machine when none does. taracode's next release finds the engine at this address without configuration, and other OpenAI-compatible clients can use it now; a web page from another origin needs allowed_origins.

The verified download, the packages and the source

Download the binary for your machine from the releases page: taraengine-darwin-arm64 for Apple Silicon, taraengine-linux-amd64 or taraengine-linux-arm64 for Linux. Every release also carries checksums.txt, a keyless cosign signature of it as a Sigstore bundle (checksums.txt.sigstore.json) and SLSA provenance.

taraengine version prints the version you installed. The signature covers checksums.txt, and the checksums cover every binary and package of the release, so verifying the one verifies the rest. The provenance can be checked with slsa-verifier. If macOS blocks a binary that you downloaded with a browser, remove the quarantine mark with xattr -d com.apple.quarantine taraengine-darwin-arm64.

On Linux the release also has a .deb and an .rpm for amd64 and arm64:

A package installs the binary as /usr/bin/taraengine and puts LICENSE and NOTICE under /usr/share/doc/taraengine/. The Homebrew cask, the installer script and the service commands come in the next release.

To build from source, clone the repository and run make build; Development lists the other targets.

Runtime runtime status

taraengine does not carry llama.cpp inside its binary. taraengine runtime install fetches it once from the llama.cpp releases on GitHub, and each taraengine release pins exactly one upstream build.

• Pinned. The binary embeds internal/runtime/runtime.lock.yaml: the build number and the sha256 of every release archive it may install, per platform.
• Verified. The download is hashed and compared with the lock before anything is unpacked. The unpacking refuses any entry that would land outside the target directory, and any link that does not point at a sibling file. The sha256 of the installed llama-server goes into a manifest.json next to it, and taraengine serve checks the binary against that file when it starts. It refuses to start when the runtime is missing or does not match, and it never downloads anything itself.
• Kept whole. The directory holds llama-server, its libraries and upstream's LICENSE. A failed install leaves the previous one as it was.
• Moved on request. taraengine runtime update moves to the build that the running taraengine pins, never to another one, and says already at <build> when there is nothing to do. taraengine runtime status says whether the runtime that the daemon would run on is verified, which is the runtime_dir of the configuration when it is set and the pinned build under the data directory otherwise, and --dir-only prints just that directory.

The build is chosen for the machine:

The driver version comes from nvidia-smi. The Linux arm64 builds are pinned too, but have not been tested on a machine. Other systems, such as Intel Macs, are not supported in this release. A CUDA build needs the CUDA runtime libraries of its major version. When ldconfig does not list them, runtime install also fetches upstream's CUDA runtime bundle, a few hundred megabytes; --cudart fetches it always and --no-cudart never. taraengine doctor shows the GPU, the RAM tier and the runtime it found, and names a model that needs a newer runtime than the pinned one.

The runtime lives in <data dir>/runtime/<build>/. The data directory is ~/Library/Application Support/taraengine on macOS, and $XDG_DATA_HOME/taraengine (default ~/.local/share/taraengine) on Linux.

Supervision

taraengine serve runs in the foreground and starts one llama-server in router mode on a loopback port of its own. It writes the process id of that server to child.pid in the data directory each time the server starts, and removes the file when it stops the server cleanly. If the server exits, taraengine ends the model processes that it left behind and starts it again after a pause of 1, 2, 4 and then 8 seconds. The fifth exit within ten minutes ends the restarts and marks the backend unhealthy until serve is started again.

When a daemon dies without stopping its server (it was killed, or the machine ran out of memory), the server and its model processes keep running and keep their GPU memory. The next serve looks at child.pid before it starts its own server. If that process is still there and runs the llama-server of the runtime in use, taraengine logs it and ends its process group, with SIGTERM and, ten seconds later, SIGKILL, so that a stale server cannot take the memory that a new load needs. A server of another runtime directory, or any other process, is left alone.

One daemon runs per data directory. serve takes a lock on the file lock in the data directory and writes its own process id into it, and a second serve on the same data directory, whatever its listen address, stops at once with an error that names the directory and that process id. The lock ends with the daemon, however it ends, and the file stays.

A model whose load through /api/load fails twice in a row is marked faulty and refused, on /api/load and on /v1, until taraengine load <model> --force; taraengine doctor shows its exit code and the end of the server log. A load that /v1 starts by itself, and a server that dies in the middle of a chat, are not counted yet.

SIGINT or SIGTERM stops the daemon: requests in flight get five seconds, and llama-server gets SIGTERM and, ten seconds later, SIGKILL.

Commands --help

The commands that talk to the daemon (pull, list, ps, load, unload, rm and doctor) take its address and key from the configuration file; --host, --api-key and TARAENGINE_API_KEY win over it, as Configuration explains. taraengine <command> --help lists the flags.

Models taraengine list

The catalog is built into the binary. Each entry pins a Hugging Face file by size and sha256, together with the projector that lets a model read images and the draft model that speeds it up (speculative decoding), when the model has them. A pull fetches all of them.

Size is the main file, as taraengine list shows it; the projector and the draft file come on top. KV cache is the type of the model's attention cache: q8_0 holds less memory than f16 at the same window, and an entry keeps f16 where q8_0 cost it more than a tenth of its speed on the RTX 5090 board. A cache-type-k or cache-type-v in preset_global sets another for every model, these included. llama.cpp quantizes the V cache only with flash attention, so with flash-attn: "off" in preset_global these models keep an f16 V cache. A tier is the RAM class a model is meant for. small suits any machine, and a machine with at least 14, 30 or 46 GiB of RAM counts as 16, 32 or 48. (default) marks the model that taraengine doctor recommends for a tier. Each model keeps the license of its authors; read it on the model's Hugging Face page before you use it.

STATUS is available for a model in the catalog that is not on disk, pulled once it is, and loading, loaded or sleeping while the engine runs it; failed and faulty mark a load that did not work. CONTEXT is the context window in tokens that the model runs at, or would run at. Sizes below 1 GB, such as a small imported model, are shown in MB with one decimal, and from 1 GB in GB.

Pull, load, unload, remove

A pull resolves each file with Hugging Face, refuses it when the sha256 that Hugging Face reports is not the one the catalog pins, checks the free disk space, and downloads into a .part file, so that running the same pull again resumes an interrupted one. The complete file is hashed before it is moved into place; a file that fails the check is deleted and never becomes a model. The command prints a line for every five percent of a download, in MB below 1 GB and in GB from there.

A model from hf:// or from a local path is named after its file without .gguf, has no tier and no promise of a tuned preset, and its capabilities are marked (inferred): taraengine reads them from the chat template while the model runs and keeps them, so taraengine list still shows them after the model stops. Its native context window comes from the file's header when it is pulled, and its default window is never larger; a load that asks for another window gets that one. A local file is linked, not copied, and taraengine rm removes the link and never the file. A name that the catalog owns is refused with a 409, and so is one that llama-server would serve under the same id as another model's name (gemma4:12b and gemma4-12b are one id), and a different file under a name that an import already holds.

A gated or private repository needs hf_token in the configuration, or TARAENGINE_HF_TOKEN in the environment of serve. To use a catalog of your own, put a registry.yaml in the data directory; it replaces the built-in one. It is checked like the built-in one: a key that taraengine does not know, a preset key that does not tune how a model runs (the error lists the keys that do), two names that llama-server would serve under one id, and a second default for a tier are refused with an error that names them. The engine itself sets where a model is, its context window, its projector and its draft.

The context window defaults to 32768 tokens, or to the model's native maximum when that is smaller. context in the configuration changes the default, and context_by_tier sets it per RAM tier. The memory of a loaded model is measured on NVIDIA, as the GPU memory that the driver reports for the model's process. On a Mac the GPU has no per-process figure, so the number is the process's resident memory and is labeled estimated. A dash means that there is no figure. A model that a request through /v1 brings up on its own gets its figure from a later read of the model list (taraengine list, taraengine ps or the API) that finds no load, unload or catalog change under way. A model left idle for sleep_idle_seconds sleeps: ps shows sleeping with the figure of its last run, the next request wakes it, and taraengine load of a sleeping model at its window answers at once.

API /api

The control API is under /api on the listen address, http://127.0.0.1:8272 by default. Bodies are JSON, and an error is {"error":{"message":"...","hint":"..."}}.

/v1 is llama-server's OpenAI and Anthropic surface, proxied without buffering the answers, so streams arrive as they are produced: /v1/chat/completions, /v1/completions, /v1/embeddings, /v1/models, /v1/messages and /v1/messages/count_tokens among them.

• Models. Name the model by its catalog name, such as gemma4:12b. A request for a pulled model that is not loaded loads it first, and with models_max at 1 it replaces the model that was loaded. A request has to name a pulled model: a name the engine does not know, or has not pulled, is a 404 that lists up to three of the nearest pulled names; a model marked faulty is a 409 until taraengine load <model> --force; and a body without a model, or one that is not a JSON object, is a 400. /v1/models lists the pulled models under their catalog names and nothing that the router found elsewhere. Only GET, HEAD and POST are served: OPTIONS is answered by the daemon itself, as described under Privacy and security, and every other method is a 405 with an Allow header. The model field of a chat answer carries the id that llama-server knows the model by, which writes a hyphen for the colon: gemma4-12b.
• Thinking. "chat_template_kwargs": {"enable_thinking": false} switches thinking off and true on, and "reasoning_effort": "low", "medium" or "high" sets the effort where the model's template honors it. The thoughts come back in reasoning_content, on the message and on streamed deltas.
• Timings. Every chat answer carries usage and timings (prompt_n, prompt_ms, prompt_per_second, predicted_n, predicted_ms, predicted_per_second, cache_n); in a stream they arrive on the last chunk. Answers that /v1 passes on carry an X-Taraengine-Version header.
• Limits. A request body over 32 MiB is refused with 413.

The JSON types of the API are in pkg/api, and a typed Go client is in pkg/client.

Configuration config.yaml

taraengine needs no configuration file: every key has a default. When there is one, it is $XDG_CONFIG_HOME/taraengine/config.yaml or ~/.config/taraengine/config.yaml, on macOS too, and --config FILE names another, which then has to exist. A key that this version does not know is an error that names the key and its line and lists the accepted keys, so that a misspelled api_key cannot leave an engine open without a word. serve also takes --listen, --data-dir, --runtime-dir and --api-key, which win over the file. The environment variables TARAENGINE_API_KEY and TARAENGINE_HF_TOKEN win over the file as well. The other commands read the same file for the data directory. They reach the daemon at --host, or, when --host is left out, at the listen address of the file, http://127.0.0.1:8272 when there is none. The key is --api-key, else TARAENGINE_API_KEY, and either goes with any address. The api_key of the file goes only with the address of the file: it belongs to the daemon that the file configures, so with --host it is not used, and a daemon that asks for a key then needs --api-key or TARAENGINE_API_KEY.

Keep the key in the configuration file (with a mode that only you can read) or in the environment. Do not put it in --api-key on a machine that other people use: the arguments of a command show in the process list of every user.

The data directory holds:

Privacy and security no telemetry

No telemetry. taraengine sends no usage data, makes no update checks and files no crash reports. It talks to two hosts, and only when you run a command that needs them: huggingface.co on taraengine pull, and github.com on taraengine runtime install and runtime update. Both redirect downloads to their own content delivery hosts (under hf.co and githubusercontent.com). Nothing else leaves the machine.

A pull is carried out by the daemon, so it is the serve process that connects to Hugging Face, during the pull and for a short while after it. At any other time neither serve nor the llama-server it supervises holds a connection beyond the loopback. A Hugging Face token is sent to Hugging Face only, and it is never logged.

What can reach the API:

• Localhost by default. listen is 127.0.0.1:8272. To serve other machines, set listen, for example to 0.0.0.0:8272, and set an api_key. taraengine logs a warning when it listens beyond localhost without a key.
• A Bearer key. With api_key set, every route but GET and HEAD /api/health needs Authorization: Bearer <key>, and a missing or wrong key is a 401. No other header counts: a client that sends its key as x-api-key is refused.
• Web pages. A page that you open in a browser can send requests to 127.0.0.1:8272, and the browser attaches an Origin header to them. The daemon answers a request with an Origin header with 403 unless the page is served from localhost, 127.0.0.1 or [::1] (http or https, any port) or its origin is listed under allowed_origins, with or without an api_key. The preflight that a browser sends before a request across origins is answered by the daemon, for the allowed origins only, and is never passed on to llama-server. The CORS headers that llama-server puts on its own answers are removed, so that the headers of the daemon are the only ones a page ever sees. Programs that are not browsers, such as curl, the OpenAI and Anthropic SDKs and taracode, send no Origin and are not asked about it.
• JSON only. Every POST under /api and /v1 must be application/json, a charset parameter allowed, and anything else is a 400 with a hint. A web page can make the browser post a form or plain text to any address without asking first, and the body would otherwise be read as JSON all the same. The OpenAI and Anthropic SDKs send application/json; with curl, send -H 'Content-Type: application/json'.
• The Host header. With no api_key, the API refuses a request whose Host is not one of its own addresses (localhost, 127.0.0.1, [::1] or the host in listen) with a 403. That keeps out a web page that points its own DNS name at 127.0.0.1. With a key set, the key does that job. A daemon that listens on every interface skips the check.
• The supervised server. The llama-server that taraengine starts listens on a loopback port with a generated key. The key is in child.key in the data directory, readable by you only, and llama-server reads it from that file, so it is on no command line. The /v1 proxy swaps your key for it. The processes that llama-server starts for each loaded model listen on random loopback ports and do not check that key, so any program on the same machine can reach them, as it can with any inference server on localhost. Keep that in mind on a machine that you share with people you do not trust. The server runs with an environment of its own: PATH, HOME, TMPDIR, LANG, the LC_* and CUDA_* variables as you have them, and the runtime's library paths, and nothing else. It inherits neither your API key nor your Hugging Face token, nor any LLAMA_ARG_* setting, and its LLAMA_CACHE is the empty llama-cache directory in the data directory, so it never serves a model that is not in the catalog.
• Verified downloads. Runtime archives are checked against the sha256 values pinned in the binary, and model files against the values pinned in the catalog, or for hf:// files against the checksum that Hugging Face reports. A model name is checked before it touches the file system, so it cannot carry a path.
• Logs. logs/taraengine.log and logs/llama-server.log in the data directory are readable by you only. taraengine does not log prompts or answers, and the supervised llama-server runs at warning verbosity.

To report a vulnerability, see SECURITY.md.