# Run a model

> [!NOTE]
> WebGPU has been tested on a software adapter (Mesa llvmpipe through Dawn, the `webgpu` package) only. Real GPUs and real browsers have not been run, so `AUTO_WEBGPU_MIN_MACS` (51,314,688) is a placeholder for real devices. See [Status](https://gpu-engine.dev/docs/start/status).

## Loading and running

`loadModel(weights, options)` loads a binary weights file for `tag`, `pooled` or `retrieve` models and never rejects. Because the file embeds its canonical definition, no external config is required to run it, though passing `definition` gives strict TypeScript output types and checks that the file matches your definition hash.

```ts
import { readFile } from 'node:fs/promises'
import { loadModel } from 'gpu-engine'
import definition from './my-model.model.ts'

const model = await loadModel(await readFile('weights/my-model.bin'), { backend: 'auto', definition })
const [error, tags] = await model.run('call me tomorrow at 5pm')
if (error) console.log(error.code, error.fix)
else console.log(tags) // ('start' | 'inside' | 'none')[]
console.log(model.inspect()) // { backend, reason, bytes, macs }
```

`weights` must be a `Uint8Array` or `ArrayBuffer` (anything else fails with `WEIGHTS_FORMAT_INVALID`). If loading fails, `loadModel` still resolves to a `Model` handle: every call to `model.run(...)` returns that load error, and `model.inspect().reason` names it.

| Option | Required | Meaning |
| --- | --- | --- |
| `backend` | Yes | `'auto'`, `'webgpu'` or `'cpu'`. There is no default; missing or unknown values return `RUN_BACKEND_UNAVAILABLE`. |
| `definition` | No | Types the return value of `run` from the head labels and verifies that its definition hash matches the file (`WEIGHTS_DEFINITION_MISMATCH`). Budgets are excluded from the hash. |
| `gpu` | No | Explicit WebGPU entry point for environments without `navigator.gpu` (in Node, `create([])` from the `webgpu` package, kept referenced while the model is alive). |
| `table` | `retrieve` only | Non-empty `{ id, text }[]` rows for a `retrieve` head (unique non-empty string `id`, string `text`). Encoded once at load on CPU; WebGPU uploads the row vectors after the weights. Missing or invalid rows fail with `RUN_TABLE_MISSING`. Ignored by other heads. |

`model.run(input)` accepts a string or `{ text, flags }`, where `flags` supplies one boolean per token of `tokenize(text)` for each declared flag row (using own properties only). Empty text runs on every head: `tag` returns `[]`, `pooled` decodes its bias (with no tokens the pool is zero), and `retrieve` ranks every table row against the head's bias vector. Inputs longer than `input.max` tokens run on the first `max` tokens only.

### Running a `retrieve` model

Pass `table` to `loadModel`. When `table` is written inline or `as const`, TypeScript narrows each match's `id` to the literal union of table ids; otherwise `id` is `string`. `run(input)` returns `RetrieveMatch<Id>[]`: one `{ id, score }` per table row, highest score first, preserving table order on a tie.

```ts
import { readFile } from 'node:fs/promises'
import { loadModel } from 'gpu-engine'
import definition from './command-lookup.model.ts'

const table = [
  { id: 'open-file', text: 'open a file' },
  { id: 'save-file', text: 'save the current document' },
] as const

const model = await loadModel(await readFile('weights/command-lookup.bin'), { backend: 'auto', definition, table })
const [error, matches] = await model.run('save my work')
if (!error) console.log(matches[0]) // { id: 'open-file' | 'save-file', score: number }
```

## Non-throwing `[error, data]` results

`model.run(input)` resolves to a `RunResult<Data>` tuple: `[error, undefined]` or `[undefined, data]`. It never throws.

- **Load errors** returned on every `run`: `WEIGHTS_FORMAT_INVALID`, `WEIGHTS_VERSION_UNSUPPORTED`, `WEIGHTS_DEFINITION_MISMATCH`, `RUN_BACKEND_UNAVAILABLE`, `RUN_GPU_FAILED`, `RUN_TABLE_MISSING`.
- **Input errors:** non-string / non-`{ text, flags }` values, undeclared flag names, or flag arrays whose length does not match `tokenize(text).length` return `RUN_INPUT_INVALID` without quoting the input text.
- **GPU runtime errors:** pipeline validation errors or lost devices return `RUN_GPU_FAILED`. After a device is lost, WebGPU runs return `RUN_GPU_FAILED` while `'auto'` inputs below the threshold still run on CPU.
- **Fallback error:** any other unexpected backend exception returns `RUN_FAILED`.

Every `RunError` is a `Diagnostic` (`{ code, message, why, fix, file?, line? }`). See [Diagnostics](https://gpu-engine.dev/docs/reference/diagnostics) for every runtime code.

## Backend selection and `availability`

The runtime resolves a WebGPU entry point from `options.gpu`, falling back to `navigator.gpu`. Importing `gpu-engine` touches no WebGPU globals.

| Requested `backend` | Chosen backend | `inspect().reason` reports |
| --- | --- | --- |
| `'cpu'` | CPU | The explicit `'cpu'` request |
| `'webgpu'` | WebGPU (or `RUN_BACKEND_UNAVAILABLE` / `RUN_GPU_FAILED`) | The request and adapter name, or why startup failed |
| `'auto'` | CPU or WebGPU by the first matching rule below | The rule name and measured numbers |

`'auto'` rules, checked in order:

1. No WebGPU entry point: CPU.
2. `macsPerToken * input.max < AUTO_WEBGPU_MIN_MACS`: CPU without opening a WebGPU device, because no input of this model can reach the threshold.
3. No WebGPU adapter found (`no WebGPU adapter`), or device/pipeline startup fails (`WebGPU failed to start`): CPU.
4. WebGPU opens and each `run` picks dynamically: CPU when `macsPerToken * tokens < AUTO_WEBGPU_MIN_MACS`, WebGPU at or above the threshold. `inspect()` updates after each run.

`availability({ gpu? })` checks which backend `'auto'` would choose before loading any weights. Without a model it applies rule 1 and the adapter check of rule 3, returning `{ backend: 'webgpu', reason }` (naming `AUTO_WEBGPU_MIN_MACS`) when an adapter is available.

### Running on WebGPU in Node

`gpu-engine` has zero runtime dependencies and does not ship `webgpu`. Install the `webgpu` package (`^0.6.2`) yourself to use WebGPU in Node (on Linux without a GPU, install `mesa-vulkan-drivers` for Mesa lavapipe):

```ts
import { readFile } from 'node:fs/promises'
import { loadModel } from 'gpu-engine'
import { create } from 'webgpu'
import definition from './my-model.model.ts'

const gpu = create([])
const model = await loadModel(await readFile('weights/my-model.bin'), { backend: 'webgpu', definition, gpu })
const [error, tags] = await model.run('call me tomorrow at 5pm')
console.log(error ?? tags, model.inspect().reason)
```

Keep the `gpu` instance referenced for the lifetime of the model.

### The `auto` threshold

`'auto'` avoids WebGPU when an input is too small to pay back GPU dispatch and readback overhead. `AUTO_WEBGPU_MIN_MACS` in `src/runtime/auto.ts` is `51314688` (51,314,688 multiply-adds per run), measured by `scripts/bench-threshold.ts` and recorded in `docs/measurements/auto-threshold.json` (a test keeps the constant equal to the recorded file).

The benchmark runs a fixed sweep of tag models (widths 8 to 256, inputs of 1 to 256 tokens) on both backends through `loadModel` using Dawn (`webgpu`) on Mesa llvmpipe on a shared 4-core VM (load average 2.5 to 4.2 during the run). It runs the full sweep five times in a row and takes each cell's median across its five sweep medians. The rule selects the smallest multiply-add count from which WebGPU beats CPU in every measured cell. WebGPU was faster in 6 cells below the threshold (from 258,048 multiply-adds up) and won 4 of the 6 cells from 3,207,168 multiply-adds up (threshold cell included), while CPU won width 128 at 256 tokens (13,074,432 multiply-adds, 23.9 ms against 26.9 ms), so the rule picked the largest cell measured (51,314,688). Earlier single-sweep runs picked 51,314,688, 12,828,672 and 13,074,432.

On that machine, `'auto'` runs almost every input on CPU. Real GPUs and real browsers have not been measured yet, so treat this value as a placeholder for real devices; request `'webgpu'` or `'cpu'` directly when you know which backend you want.

## Inspecting a loaded model

`model.inspect()` returns an `Inspection`:

| Field | Type | Meaning |
| --- | --- | --- |
| `backend` | `'webgpu' \| 'cpu'` | Backend that ran (or was chosen at load) |
| `reason` | `string` | Rule or request that selected it, or the load error |
| `bytes` | `number` | Byte length of the weights file |
| `macs` | `number` | Multiply-adds per token (`plan.macsPerToken`) |

## How the WebGPU pipeline runs

- The shared CPU featurizer computes token feature rows and uploads them to the GPU.
- Dequantized float32 weights upload once and compute pipelines compile once at load time.
- Every op in the plan dispatches its hand-written WGSL passes inside one command encoder, alternating across two scratch buffers sized `input.max * maxWidth`.
- For a `retrieve` model, every table row is encoded once on the CPU at load using the same forward pass, the resulting row vectors upload after the model parameters in binding 0, and the score step runs as one trailing WGSL pass (one invocation per table row) reading back one score per row.
- One readback returns raw logits (or table scores) to the CPU, where the head decodes them using the exact same decode code as the CPU backend.
- The pipeline reads templates from the op registries and names no op itself, so any layer or head with a `wgsl` template runs automatically.

The CPU backend calls `forward(plan, weights, rows, n)` (also exported from `gpu-engine/harness`), which is the exact forward pass used during training.

## The weights file

Weights use version 1 of a documented, little-endian binary format (`GPEW`) so external trainers can export directly into gpu-engine. Every padding byte, reserved field and unused bit must be zero.

`toWeightsFile(plan, weights)`, `encode(file)`, `decode(bytes)` and `dequantize(file, plan)` live in `gpu-engine/harness`. `encode(decode(bytes))` is byte-identical because `decode` rejects any non-canonical byte.

| Offset | Bytes | Field |
| --- | --- | --- |
| 0 | 4 | ASCII magic `GPEW` |
| 4 | 2 | u16 format version (`1`) |
| 6 | 2 | u16 flags (`0`) |
| 8 | 4 | u32 definition hash, lane 0 |
| 12 | 4 | u32 definition hash, lane 1 |
| 16 | 4 | u32 `D`, byte length of canonical definition JSON |
| 20 | 4 | u32 `T`, tensor count |
| 24 | `D` | Canonical definition JSON in UTF-8, zero-padded to a 4-byte boundary |
| next | | `T` tensor records in plan order |
| next | | `T` tensor payloads in the same order, each zero-padded to a 4-byte boundary |

Each tensor record stores the UTF-8 name (`<op index>.<kind>.<tensor>`, for example `0.embed.table`), bit width (`6` for int6 when `quant` is set, `32` for float32 otherwise), rank and dimensions, f32 scale (`1` for float32), and payload byte length `P`. Int6 payloads pack `level + 32` (`1` to `63`; code `0` is invalid) LSB-first in 6 bits per weight. Canonical definition JSON serialises `{ head, input, name, net, quant? }` with sorted keys and no whitespace, hashed with two 32-bit FNV-1a lanes. SPEC section 5 specifies every offset and bit.

Reader error codes:

| Code | Trigger |
| --- | --- |
| `WEIGHTS_FORMAT_INVALID` | Wrong magic, nonzero flags/padding/reserved bytes, truncated or trailing bytes, non-canonical JSON, invalid definition, bad bit width/payload length, non-finite or negative scale, or int6 code `0` |
| `WEIGHTS_VERSION_UNSUPPORTED` | Magic matches `GPEW` and version is not `1` |
| `WEIGHTS_DEFINITION_MISMATCH` | Hash mismatch with stored JSON or `options.definition`, or tensor count/name/shape/bit width disagrees with `plan(definition)` |

Source: SPEC sections 5, 9 and 11.

<!-- vitehub-docs:section-links -->
## In this section

- [Define a model](https://gpu-engine.dev/raw/docs/build/define-a-model.md)
- [Data and splits](https://gpu-engine.dev/raw/docs/build/data-and-splits.md)
- [Train and tune](https://gpu-engine.dev/raw/docs/build/train-and-tune.md)
- [Evaluate and promote](https://gpu-engine.dev/raw/docs/build/evaluate-and-promote.md)
- [Run a model](https://gpu-engine.dev/raw/docs/build/run-a-model.md) (this page)
- [Parity and ship](https://gpu-engine.dev/raw/docs/build/parity-and-ship.md)

Next: [Parity and ship](https://gpu-engine.dev/raw/docs/build/parity-and-ship.md)

All pages: [llms.txt](https://gpu-engine.dev/llms.txt)
