# Train and tune

> [!NOTE]
> Warm start, distillation and budget-bounded sweeps are planned for a later version (SPEC section 6, no milestone assigned). Everything else on this page works today. See [Status](https://gpu-engine.dev/docs/start/status).

## Explicit training options

`defineEval` provides no defaults for training. Every eval file writes out all six `TrainOptions` fields (plus `negatives` for `retrieve` heads):

```ts
import type { TrainOptions } from 'gpu-engine'

const train: TrainOptions = {
  seeds: [1, 2, 3],
  epochs: 20,
  batch: 32,
  learningRate: 0.01,
  warmup: 0.1,
  budgetSeconds: 40,
}
```

| Field | Meaning |
| --- | --- |
| `seeds` | Non-empty list of integer seeds; each seed runs sequentially |
| `epochs` | Positive integer number of passes over the training set |
| `batch` | Positive integer batch size per optimizer step |
| `learningRate` | Peak learning rate (`> 0`) |
| `warmup` | Share of planned steps spent on linear warm-up (`0 <= warmup < 1`) |
| `budgetSeconds` | Total wall-clock seconds shared across all seeds (`> 0`) |
| `negatives` | Required for `retrieve` heads (ignored by other heads): positive integer count of other table rows drawn per example per step |

## The TypeScript trainer

`train(definition, { train, dev, table? }, options, score?)` in `gpu-engine/harness` runs purely in memory using the hand-written backward passes. It reads and writes no files and returns checkpoints as `Float32Array` weights.

```ts
import { readFile } from 'node:fs/promises'
import type { Example } from 'gpu-engine'
import { encode, toWeightsFile, train } from 'gpu-engine/harness'
import model from './my-model.model.ts'

const read = async (path: string): Promise<Example[]> =>
  (await readFile(path, 'utf8'))
    .split('\n')
    .filter((line) => line.trim() !== '')
    .map((line) => JSON.parse(line) as Example)

const result = await train(
  model,
  { train: await read('data/train/main.jsonl'), dev: await read('data/dev/main.jsonl') },
  { seeds: [1, 2, 3], epochs: 20, batch: 32, learningRate: 0.01, warmup: 0.1, budgetSeconds: 40 },
)
console.log(result.seed, result.checkpoint, result.devBySeed, result.stoppedBy)
const bytes = encode(toWeightsFile(result.plan, result.weights))
```

How `train` runs:

- **One forward pass.** Every example is featurized once. Each step runs `forward` (the exact function the CPU runtime uses) with caches, computes the head's loss, and walks each op's `backward` from the head back to `embed`. Batch gradients are averaged over the batch.
- **Retrieve shared-tower training.** A `retrieve` head requires `data.table` (`RUN_TABLE_MISSING` without) and `options.negatives`. Table rows are featurized once. For each example, the candidates are its gold rows plus `options.negatives` other table rows drawn without replacement from the seed's stream (or all remaining rows if the table has fewer). The input and each candidate run forward with caches, joined by the dot-product score step and softmax cross-entropy loss (`log sum over candidates of exp(score) - log sum over gold of exp(score)`), and the backward walk runs once for the input and once per candidate so gradients flow through both sides of the shared tower. The default dev score is minus the mean of that loss over the whole table.
- **Optimizer and schedule.** Bias-corrected Adam (`beta1 = 0.9`, `beta2 = 0.999`, `epsilon = 1e-8`). With `total = epochs * ceil(train.length / batch)` planned steps and `W = round(warmup * total)`, step `k` uses `learningRate * k / W` for `k <= W` and cosine decay `learningRate * (1 + cos(pi * (k - W) / (total - W))) / 2` thereafter, reaching 0 on the final step.
- **Reproducible seeds.** Each seed initialises every op in plan order from mulberry32 (`createRandom(seed)`) and shuffles training order at each epoch start from the same stream (a `retrieve` head also draws negatives from it). One seed produces bit-identical weights whenever the time budget does not interrupt it.
- **Wall-clock budget.** Each seed receives an equal share of the remaining `budgetSeconds` when it starts, so time saved by earlier seeds carries forward. A seed that hits its deadline stops after the current step with `stoppedBy: 'budget'`.
- **Checkpoints.** At each epoch end (and once more at the final step if stopped mid-epoch by the budget), weights are quantized (when `quant` is set) and passed to `score` (higher is better; defaults to minus mean dev loss). Each seed tracks its `best` checkpoint (earliest on a tie) and `last` checkpoint. `train` returns the `best` checkpoint of the best-scoring seed (first seed on a tie), along with `devBySeed` and all per-seed `runs`.
- **Gold validation.** Malformed gold throws `DATA_ROW_INVALID`; an unknown label or table row id throws `DATA_LABEL_UNKNOWN`.

Initial weight distributions for every layer and head are listed in the [layer catalog](https://gpu-engine.dev/docs/reference/layers).

### Trainer scaling limit

The TypeScript trainer is proven only on models with a few thousand parameters. Train larger models outside gpu-engine (for example in PyTorch) and export them into the version 1 binary weights layout documented on [Run a model](https://gpu-engine.dev/docs/build/run-a-model#the-weights-file).

## Quantization-aware training

Adding `quant: int6({ qatFrom })` to a model enables symmetric per-tensor int6 quantization-aware training starting at step `round(qatFrom * total)` (0-indexed):

- **Per-tensor scale.** `scale = float32(max |w| / 31)` over the tensor, `level = clamp(Math.round(w / scale), -31, 31)`, and the quantized weight is `float32(level * scale)`. All-zero tensors get scale 0 and level 0.
- **Straight-through estimator.** From `qatFrom` onward, forward and backward run on the quantized copy while Adam updates the float32 master weights.
- **Bitwise match with runtime.** QAT and the weights encoder share the same `quantizeInto` routine, so the weights scored during training match what `loadModel` decodes bit for bit. `dequantize(file, plan)` in `gpu-engine/harness` recovers those same float32 values from a decoded weights file.
- **No float scores reported.** When `quant` is present, `score` only ever receives quantized weights, and `result.weights` holds those quantized weights.

## The dev tune loop

`gpu-engine loop` executes one tuning try against `data/dev/`:

```sh
gpu-engine loop --log "hidden 12"
```

Each invocation of `gpu-engine loop [--log "<idea>"] [--status kept|dropped]` runs seven steps:

1. Runs the level 1 static scan for stray `data/test` references. Any finding exits 1 before training.
2. Runs the data step (generate empty sets, run section 7 checks, update `data.lock.json`). Any diagnostic exits 1 before training.
3. Trains on all train sets combined using `train`, scoring dev epochs with the mean of the model's `metric` scores over all dev sets.
4. Writes `.gpu-engine/candidate.bin` (the chosen checkpoint), `.gpu-engine/candidate.json`, and `.gpu-engine/checkpoints/seed-<seed>-best.bin` and `seed-<seed>-last.bin` for every seed.
5. Prints the training summary, per-seed best dev scores, the dev score table (model, each baseline and oracle), every gate line (`pass` or `FAIL`), and the candidate path with its dev score and Brotli byte size.
6. Appends one row to `tries.jsonl`.
7. Exits 0 if every gate line passed, or 1 if any gate line failed. The candidate and tries row are written either way.

`candidate.json` records `at`, `commit`, `dirty`, `definition`, `data` (the SHA-256 of every train, dev and test file from `data.lock.json`), `seeds`, `seed`, `checkpoint`, `steps`, `trainSeconds`, `stoppedBy`, `devBySeed`, `dev`, `devScore`, `devSpread`, `brotliBytes` and `macsPerToken`. If you edit the model or data after `loop`, `promote` rejects the candidate with `PROMOTE_CANDIDATE_STALE`.

### Inspecting dev misses with `--fail`

```sh
gpu-engine loop --fail
gpu-engine loop --fail --set main
```

`loop --fail` trains nothing and writes no try. It runs the level 1 scan, loads `.gpu-engine/candidate.bin` (exiting 1 with `PROMOTE_CANDIDATE_MISSING` if no candidate exists), runs the data step, checks that the candidate is still fresh (`PROMOTE_CANDIDATE_STALE` otherwise), and then prints every dev example where the model scored below the oracle, showing the input, expected reference and model prediction. It never reads `data/test/`.

### Recording the decision with `--mark`

```sh
gpu-engine loop --mark kept
gpu-engine loop --mark dropped
```

`loop --mark` trains nothing and appends no try. Once you have read the last try's dev score and kept or reverted the change, `--mark` sets the `status` field of the last row of `tries.jsonl` from `null` to `'kept'` or `'dropped'`, leaving every other row and field unchanged. If `tries.jsonl` has no rows, if the last row already has a non-null status, or if any other `loop` flag besides `--dir` is passed, it exits 1 with `CLI_FLAG_INVALID` and changes nothing.

## The tries log

`tries.jsonl` is append-only with a closed 14-key schema. The one allowed edit is `loop --mark`, which sets a `null` status on the last row once:

```json
{
  "at": "2026-11-21T12:00:00.000Z",
  "commit": "<git HEAD, or none>",
  "dirty": false,
  "idea": "<--log, or an empty string>",
  "status": "kept | dropped | null",
  "definition": "<definition hash>",
  "seeds": [1, 2, 3],
  "dev": { "<dev set>": 0.94 },
  "devScore": 0.94,
  "devSpread": 0.01,
  "brotliBytes": 1234,
  "macsPerToken": 900,
  "trainSeconds": 4.2,
  "gate": "pass | fail"
}
```

- `dirty` is `true` when `git status --porcelain` is non-empty.
- `devSpread` is the highest minus lowest seed's best dev score (`0` for a single seed).
- `brotliBytes` is the candidate weights file compressed with Brotli at quality 11.
- Writing any key outside this list throws a `TypeError`. Any key matching `/test/i` at any nesting depth is rejected with `DATA_TEST_PRINTED`.

## Tuning discipline

Follow the [rules for agents](https://gpu-engine.dev/docs/start/agents):

- Change one variable per try so the dev delta has a single cause.
- Log every try with `--log`, then record whether you kept or dropped it with `gpu-engine loop --mark kept` or `--mark dropped`.
- Stay on `loop` and `dev` until the candidate is ready to promote.

Source: SPEC sections 5, 6 and 11.

<!-- vitehub-docs:section-links -->
## In this section

- [Define a model](https://gpu-engine.dev/raw/docs/build/define-a-model.md)
- [Data and splits](https://gpu-engine.dev/raw/docs/build/data-and-splits.md)
- [Train and tune](https://gpu-engine.dev/raw/docs/build/train-and-tune.md) (this page)
- [Evaluate and promote](https://gpu-engine.dev/raw/docs/build/evaluate-and-promote.md)
- [Run a model](https://gpu-engine.dev/raw/docs/build/run-a-model.md)
- [Parity and ship](https://gpu-engine.dev/raw/docs/build/parity-and-ship.md)

Next: [Evaluate and promote](https://gpu-engine.dev/raw/docs/build/evaluate-and-promote.md)

All pages: [llms.txt](https://gpu-engine.dev/llms.txt)
