# Data and splits

> [!NOTE]
> Written or teacher data with provenance and blind second labels are planned for a later version (SPEC section 6, no milestone assigned). Everything else on this page works today. See [Status](https://gpu-engine.dev/docs/start/status).

## Three data splits

Every model package divides its examples across three folders:

| Split | Folder | Purpose |
| --- | --- | --- |
| `train` | `data/train/` | Examples the optimizer learns from |
| `dev` | `data/dev/` | Examples `loop` scores against while tuning |
| `test` | `data/test/` | Sealed split read only by `promote` |

Two splits are not enough. If a tuning loop checks its changes against the same split that produces the final reported score, kept changes gradually fit that split and the score climbs without real generalisation.

When examples come from timestamped logs, split by time: train on earlier windows and test on the newest window.

## Declaring set paths

In `<name>.eval.ts`, `sets` points each split at a glob inside its folder, wrapping the test pattern in `sealed(...)`. `defineEval` throws `DEFINE_SET_FOLDER` if any set points outside `data/train/`, `data/dev/` or `data/test/`, or if a `retrieve` model's `table` points inside any of those three folders:

```ts
import { sealed } from 'gpu-engine'
import type { EvalSets } from 'gpu-engine'

const sets: EvalSets = {
  train: 'data/train/*.jsonl',
  dev: 'data/dev/*.jsonl',
  test: sealed('data/test/*.jsonl'),
}
```

Passing a wrong folder fails immediately at define time:

```txt
DEFINE_SET_FOLDER: sets.dev is 'data/test/*.jsonl', expected a path inside 'data/dev/'
```

Wildcards (`*` and `?`) are allowed only in the file name, not in the folder path. Each matching `.jsonl` file becomes one named set in the score tables, named after the file without its extension.

## JSONL row format

Each non-empty line of a dataset file is a single JSON object:

```txt
{ "input": text | { text, flags }, "gold": tags[] | label | number[] | rowId[], "expect"?: result, "group"?: string, "vocab"?: string[] }
```

- `input`: a string, or `{ text, flags }` where each declared flag maps to one boolean per token of `tokenize(text)` (counted before `input.max` truncation).
- `gold`: labels, numbers or table row ids matching the head:
  - `tag`: one head label per token of `tokenize(text)`. A count mismatch is `DATA_ROW_INVALID`; a label outside `head.labels` is `DATA_LABEL_UNKNOWN`. Labels past `input.max` are dropped along with their tokens.
  - `pooled` (class form): one string from `head.labels`.
  - `pooled` (values form): an array of `head.outputs` finite numbers.
  - `retrieve`: an array of at least one table row id with no duplicates (`DATA_ROW_INVALID` otherwise). Every id must name a row in `eval.table` (`DATA_LABEL_UNKNOWN` otherwise). Train and dev ids are checked during the data step; test ids are checked when `promote` reads the test set.
- `expect` (optional): the reference result for scoring when it differs from `decode(gold, input)`.
- `group` (optional): a grouping key such as a schema, domain, site or time window that no two splits may share.
- `vocab` (optional): words in this row that no two splits may share (for instance unit names when test units must be unseen).

Because `group` and `vocab` are plain data fields on each row rather than user callbacks, the harness checks overlap and vocabulary separation on the test set without invoking any package-supplied code on a test row.

## Retrieve table

A `retrieve` head scores against `eval.table`, a JSONL file of `{ "id": string, "text": string }` rows outside the three set folders (such as `data/table.jsonl`). It is an input, not a split, so no overlap, group or vocabulary check compares it with `train`, `dev` or `test`.

Data checks on `eval.table`:

- Missing `eval.table` on a `retrieve` model, missing file, or a table file with 0 rows: `DATA_SET_MISSING`.
- A line that is not valid JSON, lacks a non-empty string `id` or a string `text`, or repeats an earlier `id`: `DATA_ROW_INVALID`.
- `data.lock.json` records its SHA-256 and row count under `table`, checked on every run (`DATA_HASH_CHANGED`).

`loadProject` reads the table automatically when loading a `retrieve` package, so scoring and `inspect` need no data step.

## `defineDataset`

`defineDataset` sits in the `data` field of `defineEval`:

```ts
import { defineDataset } from 'gpu-engine'

const data = defineDataset({
  seed: 1,
  minSize: { train: 2_000, dev: 300, test: 300 },
})
```

| Field | Meaning |
| --- | --- |
| `seed` | Base seed for generators |
| `minSize` | Minimum row count required in every file of `train`, `dev` and `test` |
| `generate` | Optional `{ train, dev, test }` functions, each `(random) => Example[]` |

## Generate, check and record

`loop`, `promote` and `promote --seal` run the data step before training, scoring or sealing:

1. **Generate.** When `generate` is declared, the harness checks each split's folder. If a folder holds no files at all, it runs `generate.<set>(random)` where `random` is mulberry32 seeded with `(seed + j * 2654435769) mod 2^32` (`j` = 0 for train, 1 for dev, 2 for test) and writes one `JSON.stringify(example)` per line to the pattern with `*` replaced by `main` and `?` replaced by `0` (for example `data/train/main.jsonl`). Folders that already contain any file are never touched, and writes use exclusive creation so existing files (especially `data/test/`) are never overwritten. Writing the test set produces rows and reads none.
2. **Check.** All section 7 checks run over hashes, row counts and fingerprints. `minSize.<set>` applies to each file in that set.
3. **Record.** When every check passes, any split (or `retrieve` table) missing from `data.lock.json` is added to it.

`data.lock.json` stores each file's SHA-256 and example count, plus the test set's fingerprints:

```json
{
  "train": { "files": { "data/train/main.jsonl": { "sha256": "<64 hex>", "examples": 48 } } },
  "dev": { "files": { "data/dev/main.jsonl": { "sha256": "<64 hex>", "examples": 16 } } },
  "test": {
    "files": { "data/test/main.jsonl": { "sha256": "<64 hex>", "examples": 16 } },
    "fingerprints": { "inputs": ["<16 hex>"], "groups": ["<16 hex>"], "vocabulary": ["<16 hex>"] }
  }
}
```

- A `retrieve` package adds `"table": { "files": { "data/table.jsonl": { "sha256": "<64 hex>", "examples": 20 } } }`, which follows the same lock rule.
- At level 3 (`promote --seal`), the `test` entry names the sealed `.enc` files instead, each with a `plaintext` field recording the SHA-256 of the file it replaced, while keeping the `fingerprints`. See [Evaluate and promote](https://gpu-engine.dev/docs/build/evaluate-and-promote#level-3-sealing).
- The harness never overwrites an existing entry in `data.lock.json`. Any file edit, addition or removal triggers `DATA_HASH_CHANGED`. Only when you changed a split on purpose, delete that split's entry from `data.lock.json` by hand and rerun to record the new hash; never do this just to make a failing check pass. For the `test` split, deleting and re-recording its lock entry means the test set itself changed, so promoted results before and after are not comparable.
- When every test file matches its locked SHA-256, checks reuse the locked test counts and fingerprints. Train and dev fingerprints are computed fresh from disk on every run.
- `gpu-engine doctor` runs the same checks without running generators and without writing `data.lock.json`.

## Section 7 data checks

| Check | When it runs | Code |
| --- | --- | --- |
| All three splits (and `eval.table` for `retrieve`) have at least one file | Before training (`loop`, `promote`, `doctor`) | `DATA_SET_MISSING` |
| Every file meets `minSize` for its split | Before training | `DATA_SET_TOO_SMALL` |
| Every line parses as `{ input, gold }` fitting the head (or `{ id, text }` for `eval.table`) | Before training (test gold checked at `promote`) | `DATA_ROW_INVALID` |
| Every gold label belongs to `head.labels` (or table row id in `eval.table`) | Before training for train/dev; at `promote` for test | `DATA_LABEL_UNKNOWN` |
| No normalised input appears in two splits | Before training | `DATA_SPLIT_OVERLAP` |
| No `group` value appears in two splits | Before training | `DATA_GROUP_SHARED` |
| No `vocab` word appears in two splits | Before training | `DATA_VOCAB_SHARED` |
| Every file matches its SHA-256 in `data.lock.json` | Every run | `DATA_HASH_CHANGED` |
| No command other than `promote` opens or references `data/test/` | Every run | `DATA_TEST_READ` |
| No output, report, try log or muted callback prints a test input or test key | `promote` and `tries.jsonl` | `DATA_TEST_PRINTED` |

Diagnostics report file paths and offending counts only. They MUST NOT print example text. See [Diagnostics](https://gpu-engine.dev/docs/reference/diagnostics) for every code's why and fix.

## Reads and digests

Checking split overlap, group isolation and vocabulary isolation requires facts about `data/test/`, yet only `promote` may read test examples. The harness splits test access into two operations:

- **Read (`kind: "examples"`):** decodes examples so a command can score them. Only `promote` (without `--verify`) can read, and every read increments the count in `reads.jsonl` and `MODEL_CARD.md`.
- **Digest (`kind: "digest"`):** opens the test files inside the guard (`src/harness/sealed.ts`) and returns only SHA-256 hashes, row counts, invalid-row counts and fingerprints. At level 3, the digest hashes `.enc` files as stored without decrypting them and needs no key. Any command (`loop`, `doctor`, `promote --verify`) can request a digest. No text, label or score leaves the guard, and no user-supplied function is called.

A fingerprint is the first 16 lowercase hex characters of `SHA-256(normalize(text))`, where `normalize` lowercases, collapses whitespace runs to one space and trims.

Fingerprints have one known limit: they act as a membership oracle. Someone who guesses a specific test input, group or `vocab` word can hash that guess and check `data.lock.json`. Fingerprints never reveal an example nobody guessed, and this tradeoff stays even under level 3 sealing.

See [Evaluate and promote](https://gpu-engine.dev/docs/build/evaluate-and-promote#enforcement-levels) for how the guard and enforcement levels work.

Source: SPEC sections 6, 7, 8 and 11.

<!-- vitehub-docs:section-links -->
## In this section

- [Define a model](https://gpu-engine.dev/raw/docs/build/define-a-model.md)
- [Data and splits](https://gpu-engine.dev/raw/docs/build/data-and-splits.md) (this page)
- [Train and tune](https://gpu-engine.dev/raw/docs/build/train-and-tune.md)
- [Evaluate and promote](https://gpu-engine.dev/raw/docs/build/evaluate-and-promote.md)
- [Run a model](https://gpu-engine.dev/raw/docs/build/run-a-model.md)
- [Parity and ship](https://gpu-engine.dev/raw/docs/build/parity-and-ship.md)

Next: [Train and tune](https://gpu-engine.dev/raw/docs/build/train-and-tune.md)

All pages: [llms.txt](https://gpu-engine.dev/llms.txt)
