# Evaluate and promote

## The eval file

Every model package holds one `<name>.model.ts` and one `<name>.eval.ts`. Nothing in `defineEval` has hidden defaults: you declare the data sets, dataset rules, training schedule, metric, baselines, gate and parity tolerance explicitly.

```ts [my-model.eval.ts]
import { defineDataset, defineEval, sealed, tokenize } from 'gpu-engine'
import type { ModelOutput } from 'gpu-engine'
import type model from './my-model.model.ts'

type Tags = ModelOutput<typeof model>

export default defineEval<Tags>({
  sets: { train: 'data/train/*.jsonl', dev: 'data/dev/*.jsonl', test: sealed('data/test/*.jsonl') },
  data: defineDataset<Tags>({ seed: 1, minSize: { train: 2_000, dev: 300, test: 300 } }),
  train: { seeds: [1, 2, 3], epochs: 20, batch: 32, learningRate: 0.01, warmup: 0.1, budgetSeconds: 40 },
  metric: (pred, gold) => (pred.length === gold.length && pred.every((tag, i) => tag === gold[i]) ? 1 : 0),
  baselines: { rules: (input) => tokenize(input).map((): Tags[number] => 'none') },
  gate: { floors: 'promoted', noise: 0.05 },
  parity: { tolerance: 1e-3 },
})
```

| Field | Purpose |
| --- | --- |
| `sets` | Globs for `train`, `dev` and `sealed(...)` for `test`. See [Data and splits](https://gpu-engine.dev/docs/build/data-and-splits). |
| `data` | `defineDataset({ seed, minSize, generate? })`. |
| `train` | Explicit `TrainOptions` (including `negatives` for `retrieve` heads). See [Train and tune](https://gpu-engine.dev/docs/build/train-and-tune). |
| `metric` | `(pred, gold) => number` returning a finite score per example (higher is better). |
| `decode` | Optional `(raw, input) => Output` turning raw head output (and gold) into your typed result. |
| `baselines` | Named `(input) => Output` functions scored in the same table as the model. |
| `gate` | `{ floors: 'promoted', noise }` setting the floor margin against the last promotion. |
| `parity` | `{ tolerance }` setting the maximum allowed absolute logit gap between CPU and WGSL. |
| `ship` | Optional `{ brotliBytes, files }` for built package ship checks. |
| `table` | `retrieve` heads only: path outside the three set folders to a JSONL file of `{ id, text }` rows. |

## Score tables

A score table puts every named file of a split in its own column (named after the file without `.jsonl`) and one row per method: `model`, each entry of `baselines`, and `oracle`.

For each row in a set, the reference is `row.expect` when present, otherwise `decode(row.gold, row.input)`:

- **Model:** `metric(decode(output, input), reference)`, where `output` comes from the CPU forward pass on the quantized weights.
- **Each baseline:** `metric(baseline(input), reference)`.
- **Oracle:** `metric(decode(gold, input), reference)`, measuring the ceiling allowed by your `decode` logic.

When `decode` is omitted, head outputs and gold values pass straight to `metric`. For a `retrieve` head, table rows are encoded once per scoring call with the weights being scored, and gold ids first become the head's output shape, the ranking a perfect model gives (the gold rows in gold order with score `1`, then every other row in table order with score `0`), so `decode` and `metric` receive the same `RetrieveMatch[]` shape for both the model and the gold. Each cell is the arithmetic mean across the set's examples. The overall `devScore` is the mean of the model's scores across dev sets. Float weights are never scored when `quant` is enabled.

## The gate

`gate.json` stores `{ level, promoted }`, where `promoted` is `null` before the first promotion and then `{ at, commit, weights, definition, dev, test }`. A package without `gate.json` defaults to level 2 with `promoted: null`; an unparseable file throws `GATE_FILE_INVALID`.

Every gate run checks five rules:

| Line | Passes when | Failure code |
| --- | --- | --- |
| Floor (per scored set) | `model >= promoted - gate.noise` (always passes if the set has no promoted score yet) | `GATE_FLOOR_FAILED` |
| Missing promoted set | Fails if a previously promoted set has no score in the current run | `GATE_FLOOR_FAILED` |
| Byte budget | Candidate weights Brotli size (quality 11) `<= budgets.brotliBytes` | `BUDGET_BYTES_EXCEEDED` |
| Compute budget | `macsPerToken <= budgets.macsPerToken` | `DEFINE_BUDGET_EXCEEDED` |
| Data hashes | Every train, dev and test file matches `data.lock.json` | `DATA_HASH_CHANGED` |

`loop` runs the gate on dev sets; `promote` runs it on both dev and test sets. Floor lines print `seed spread` next to `noise`: if seed spread is wider than `noise`, the floor sits inside run-to-run seed variance.

## Promoting a candidate

`gpu-engine promote` turns the current candidate in `.gpu-engine/candidate.bin` into the promoted release. It does not train.

```sh
gpu-engine promote
```

What `promote` executes:

1. If `gate.json` is at level 3 or a sealed `.enc` test file exists, verifies before anything else that `CI` is `'true'` (`PROMOTE_OUTSIDE_CI`), `GPU_ENGINE_TEST_KEY` is set and non-empty (`SEAL_KEY_MISSING`), and it is 64 hex characters (`SEAL_KEY_INVALID`). Then runs the level 1 scan for stray `data/test` references.
2. Loads the candidate (`PROMOTE_CANDIDATE_MISSING` if none), runs the data step, and checks that the candidate's definition hash and locked data hashes still match (`PROMOTE_CANDIDATE_STALE` otherwise). No test example is read before this passes.
3. Performs one counted read of `data/test/` through the guard (`src/harness/sealed.ts`), appending an `examples` row to `reads.jsonl`. At level 3, the guard decrypts each `.enc` file in memory and never writes plaintext (`SEAL_KEY_INVALID` on authentication failure, logged with `outcome: 'failed'`). Any test row whose gold does not match the head (or whose `retrieve` gold id names no table row) fails with `DATA_ROW_INVALID` or `DATA_LABEL_UNKNOWN` naming only the set and count.
4. Scores the candidate weights on dev and test. While a test set is scored, `promote` mutes every `console` printing method, `process.stdout.write` and `process.stderr.write`, restoring them when scoring ends. Any attempted write fails with `DATA_TEST_PRINTED` naming only the set and attempt count, printing nothing. If `metric`, `decode` or a baseline throws on a test set, `promote` drops the original error message (which could quote a test input) and raises `PROMOTE_SCORE_FAILED` with only the set name and row count. Code that opens a file, socket or file descriptor directly is not muted; level 3 is the guarantee against that.
5. Evaluates all dev and test gate lines. If any line fails, `promote` prints the tables and gate, writes nothing, and exits 1 (the test read still stays counted in `reads.jsonl`).
6. Before printing or writing anything, scans every outgoing string (stdout, files and errors) for test inputs of 8 or more normalised characters. Any hit aborts with `DATA_TEST_PRINTED` and the hit count only.
7. On a pass, writes six files and exits 0:
   - `weights/<name>.bin` (the candidate bytes)
   - `src/weights.ts` (base64 export of the weights)
   - `gate.json` (level preserved, `promoted` updated)
   - `report.json` (aggregates, hashes, training metadata, gate lines, `testReads` and `level`)
   - `MODEL_CARD.md` (rendered from `report.json` and `parity.json` when present)
   - `RESULTS.md` (rendered from `report.json`)

Always read reported scores from `report.json`, `MODEL_CARD.md` or `RESULTS.md`, never from memory.

## Verifying a promotion in CI

`gpu-engine promote --verify` is the CI gate. It reads no test examples, needs no level 3 key, and writes nothing except a `digest` row in `reads.jsonl`.

```sh
gpu-engine promote --verify
```

It runs every check below, prints all passes and diagnostics, and exits 1 on any failure:

| Check | Failure code |
| --- | --- |
| Level 1 code scan | `DATA_TEST_READ` |
| `report.json` exists and parses with the exact schema | `VERIFY_REPORT_INVALID` |
| Section 7 data checks pass and `data.lock.json` matches `report.json` hashes (or each sealed file's `plaintext` hash for a report written before level 3 sealing) | `DATA_*`, `DATA_HASH_CHANGED` |
| Current `<name>.model.ts` hashes to `report.definition` | `WEIGHTS_DEFINITION_MISMATCH` |
| `weights/<name>.bin` matches `report.weights` SHA-256, bytes and Brotli bytes | `VERIFY_WEIGHTS_CHANGED` |
| Re-scoring committed weights on dev reproduces `report.dev` exactly | `VERIFY_SCORE_CHANGED` |
| `gate.json` matches `report.level` (or level 3 over a pre-seal report) and promoted metadata and scores | `VERIFY_SCORE_CHANGED` |
| Budgets and dev floors pass | `BUDGET_BYTES_EXCEEDED`, `DEFINE_BUDGET_EXCEEDED`, `GATE_FLOOR_FAILED` |
| `reads.jsonl` has at least `report.testReads` rows of kind `examples` | `VERIFY_READS_MISSING` |
| `MODEL_CARD.md`, `RESULTS.md` and `src/weights.ts` regenerate byte-identical | `VERIFY_FILE_STALE` |

`--verify` cannot re-score `data/test/` without reading it, so it verifies that `report.json`, `gate.json` and the generated files agree. Editing all of those files together is visible only in repository history.

## The sealed test set (invariants)

SPEC section 8, verbatim:

- Only `promote` MAY read test data. A read is decoding examples (section 7, Reads and digests).
- No function the package supplies MAY be called with a test row outside `promote`. Generators that write the test set produce rows and read none.
- Any command MAY take a digest of the test set: hashes, counts and fingerprints only, never text.
- The loop and the tries log MUST record dev scores only.
- Test cases and test failures MUST NOT be printed or written to any file.
- Every read of the test set MUST be counted, and the count MUST be reported.
- Reports and comparison tools MUST NOT sample test cases.
- Data files MUST be hashed, and the gate MUST fail when a hash changes.
- At level 3 the test set MUST NOT exist in plaintext in the working tree, decrypted rows MUST stay in memory, and the key MUST NOT be printed or written inside the working tree.

Why: an agent that iterates against a check will fit it, so a test set it can see stops measuring anything. One leaked list of test failures is enough to tune against, so every path to the test set is closed, not most of them.

See [Data and splits](https://gpu-engine.dev/docs/build/data-and-splits#reads-and-digests) for how digests work without exposing text.

## Enforcement levels

| Level | Name | Mechanism | On by default |
| --- | --- | --- | --- |
| 1 | Convention and lint | Static scan rejects any code naming `data/test` except `sealed('data/test/...')` in `*.eval.ts` | Yes |
| 2 | Counted reads | Guard logs every access to `reads.jsonl` and `promote` writes the read count into `report.json` and `MODEL_CARD.md` | Yes |
| 3 | Out of reach | Test files are encrypted with AES-256-GCM to `.enc` files, plaintext is deleted, only CI holds the key, and `promote` runs only in CI | No (`promote --seal`) |

Levels 1 and 2 are active by default:

- **Single file opener (`src/harness/io.ts`).** Resolves `..` and symlinks before reading and throws `DATA_TEST_READ` on any path inside `data/test/`. Only the guard (`src/harness/sealed.ts`) claims the handle that reads test files.
- **Level 1 scan.** `loop`, `promote`, `promote --verify` and `doctor` scan code files (`.ts`, `.mts`, `.cts`, `.tsx`, `.js`, `.mjs`, `.cjs`, `.jsx`, `.vue`, `.svelte`, skipping `node_modules`, `dist`, `.git`, `.gpu-engine`) for `data/test` with either slash and for adjacent `'data', 'test'` segments. Only `sealed('data/test/...')` in `*.eval.ts` is permitted.
- **Level 2 log.** Every guard call appends `{ at, kind, command, commit, files, examples, outcome }` to `reads.jsonl`. Only `kind: "examples"` counts as a read, and `readSealed` rejects any command other than `promote` without `--verify`.

### Level 3 sealing

Levels 1 and 2 catch mistakes and record reads, but cannot stop an agent with direct filesystem access from opening `data/test/main.jsonl`. Level 3 encrypts the test set so no plaintext exists in the working tree. Run it once per package after the test set is final:

```sh
gpu-engine promote --seal --key-out ~/secrets/my-model.key
```

Without `--key-out`, the key is written to `~/.gpu-engine/keys/<model name>.key` (`--key-out` is resolved against the package folder).

What `promote --seal` does:

1. Refuses with `SEAL_REFUSED` (writing nothing) if `gate.json` is already at level 3, if a sealed `.enc` file or `.github/workflows/promote.yml` already exists, or if the key path is inside the package or its git working tree or already exists.
2. Runs the data step so every check passes and `data.lock.json` holds the test set's hashes and fingerprints.
3. Generates a 256-bit key and writes it as 64 lowercase hex characters and a newline with mode `0600`. The key is never printed.
4. Encrypts each test file inside the guard with AES-256-GCM to `<file>.enc` (`GPUESEAL` magic + 12-byte random nonce + ciphertext + 16-byte tag), using the plaintext relative path as additional authenticated data so a renamed file does not decrypt. Logs one `digest` row in `reads.jsonl`.
5. Updates the `test` entry in `data.lock.json` to name each `.enc` file with its sealed SHA-256, row count and `plaintext` SHA-256, keeping the existing fingerprints.
6. Sets `"level": 3` in `gate.json` (preserving `promoted`) and writes `.github/workflows/promote.yml`.
7. Deletes the plaintext test files and prints the sealed paths, key path and secret-upload instruction.

Store the key in your CI secret store, keep an offline copy where no agent can read it, and delete the local key file from any machine an agent uses:

```sh
gh secret set GPU_ENGINE_TEST_KEY < ~/secrets/my-model.key
```

A lost key loses the test set permanently. Commit `data/test/*.enc`, `data.lock.json`, `gate.json` and `.github/workflows/promote.yml`.

Working on a sealed package:

- `loop`, `loop --fail`, `doctor`, `inspect`, `parity` and `promote --verify` run locally without the key. Digests hash `.enc` files as stored and reuse locked row counts and fingerprints.
- Any plaintext test file placed back in `data/test/` causes `DATA_HASH_CHANGED` in every command and `SEAL_PLAINTEXT_FOUND` in `doctor`.
- When `gate.json` is at level 3, `doctor` checks that no plaintext test file exists (`SEAL_PLAINTEXT_FOUND`), that `data.lock.json` names exactly the sealed files with `plaintext` hashes (`SEAL_LOCK_INVALID`), and that `.github/workflows/promote.yml` exists and runs `gpu-engine promote` with `GPU_ENGINE_TEST_KEY` (`SEAL_WORKFLOW_MISSING`).
- Trigger `.github/workflows/promote.yml` manually (`workflow_dispatch`) to promote in CI. Because `.gpu-engine/` is not committed, the workflow installs dependencies, runs `gpu-engine loop --log "promote workflow"`, runs `gpu-engine promote` with `GPU_ENGINE_TEST_KEY`, and commits any changed promotion and log files back to the branch (even after a failed promote, so the test read stays counted). For a package inside a larger repository, move the workflow to the repository's `.github/workflows/` and set the working directory.

Limits of level 3: `CI=true` is a convention and the key is the real guarantee, lock fingerprints remain a membership oracle for guessed strings, and package code that opens files or sockets directly during test scoring is not muted by `promote`, so review changes to `metric`, `decode` and `baselines` before running the promote workflow.

Source: SPEC sections 6, 7, 8 and 11.

<!-- vitehub-docs:section-links -->
## In this section

- [Define a model](https://gpu-engine.dev/raw/docs/build/define-a-model.md)
- [Data and splits](https://gpu-engine.dev/raw/docs/build/data-and-splits.md)
- [Train and tune](https://gpu-engine.dev/raw/docs/build/train-and-tune.md)
- [Evaluate and promote](https://gpu-engine.dev/raw/docs/build/evaluate-and-promote.md) (this page)
- [Run a model](https://gpu-engine.dev/raw/docs/build/run-a-model.md)
- [Parity and ship](https://gpu-engine.dev/raw/docs/build/parity-and-ship.md)

Next: [Run a model](https://gpu-engine.dev/raw/docs/build/run-a-model.md)

All pages: [llms.txt](https://gpu-engine.dev/llms.txt)
