# Layer catalog

The layer catalog is closed: `embed`, `conv`, `biScan`, `dense`, then one head, `tag`, `pooled` or `retrieve`. A model is `featurize -> net -> head -> decode`: `net` is an explicit list that starts with one `embed`, the head comes last, and your decode step turns the head's output into your result. Each section says what the layer does, what it costs in bytes and multiply-adds, when it helps and when it does not.

Widths are per token: `in` is the width the layer reads, `out` the width it writes, `n` the token count. Bytes are for int6 weights (6 bits per parameter, packed per tensor) before Brotli; the weights file adds the definition and a small record per tensor. The formulas are what `plan(definition)` from `gpu-engine/harness` computes; a test in this repository (`test/docs.test.ts`) evaluates every formula on this page and compares it with `plan()`.

| Op         | Output width | Parameters                 | Multiply-adds per token    | Sees other tokens          |
| ---------- | ------------ | -------------------------- | -------------------------- | -------------------------- |
| `embed`    | `size`       | `rows * size + size`       | `slots * size`             | no                         |
| `conv`     | `in`         | `taps * in + in`           | `taps * in`                | `(taps - 1) / 2` each side |
| `biScan`   | `2 * size`   | `4 * size * in + 4 * size` | `4 * size * in + 2 * size` | all, both sides            |
| `dense`    | `size`       | `size * in + size`         | `size * in`                | no                         |
| `tag`      | `labels`     | `labels * in + labels`     | `in * labels`              | no                         |
| `pooled`   | `out`        | `out * 2 * in + out`       | `2 * in + 2 * in * out`    | all, by pooling            |
| `retrieve` | `size`       | `size * 2 * in + size`     | `2 * in + 2 * in * size`   | all, by pooling            |

A model's totals are the sums over its ops: `loop` prints its multiply-adds per token and the Brotli bytes of the trained weights, and `defineModel` throws `DEFINE_BUDGET_EXCEEDED` when the layers already exceed `budgets`. In a small model the `embed` table is most of the bytes; hashed rows that no training token reaches stay zero and Brotli shrinks them to almost nothing, so the trained file is usually well under the packed size.

The contract every layer implements (forward, backward, WGSL bindings, numeric rules) is at the top of `src/kit/ops.ts` in this repository.

## embed

```ts
layers.embed(size) // must be the first layer, once
```

What it does: turns each token's feature rows into one vector. The featurizer gives every token `slots` embedding row indices (`-1` for an empty slot); `embed` adds the matching rows of a `[rows, size]` table to a bias of length `size`: `y[t] = b + sum of table[r]` over the token's filled slots. `in = slots`, `out = size`. A row that fills two slots, or two tokens, is added each time. `rows` and `slots` come from `features.tokens({ rows })` (SPEC section 5, Feature rows). It sees one token at a time.

| Cost                    | Formula                                          |
| ----------------------- | ------------------------------------------------ |
| Parameters              | `rows * size + size`                             |
| Bytes (int6, packed)    | `ceil(6 * rows * size / 8) + ceil(6 * size / 8)` |
| Multiply-adds per token | `slots * size` (additions: no multiplies)        |

For example `embed(20)` over `features.shape()` and `features.hash({ buckets: 2048, k: 2 })` (2061 rows, 4 slots): 41,240 parameters, 30,930 bytes, 80 multiply-adds per token. The table is nearly all of a small model's bytes, while its compute is the cheapest of any layer.

Initial values: the table starts at zero and the bias uniform in `±0.1`. A row that no training token reaches, such as a hash bucket only unseen words land in, stays zero, so it adds nothing at run time and Brotli shrinks it to almost nothing. The bias is not zero because an all-zero `embed` sends zeros forward: a `relu` layer after it then sits at its kink, passes no gradient, and the table never learns.

When it helps:

- Always: it is the only way in. Every model starts with one `embed`.
- A wider `size` gives every later layer more channels to work with, and costs `rows` parameters per channel. Raise it when dev scores improve with depth but plateau early.

When it does not:

- Bytes grow with `rows * size`. More hash buckets cut collisions between words but each bucket costs `size` parameters; on a small training set most buckets are never reached, so past a few times the training vocabulary they mostly add rows that stay zero.
- `k` hash rows per token make collisions rarer for the same bucket count, at `k * size` more additions per token. Two is usually enough.
- It adds no context and no non-linearity: two words that share no rows look unrelated, and a token's vector ignores its neighbours. Follow it with `conv` or `biScan`.

WGSL: one pass, one invocation per (token, output channel), `ceil(n * size / 64)` workgroups of 64; binding 2 is the featurizer's rows as `array<i32>`. Matches the CPU within 1e-5 on the software adapter (llvmpipe) for `n` = 1, 2 and the maximum token count (`test/gpu/embed.test.ts`). Real GPU: not run. Real browser: not run.

## conv

```ts
layers.conv({ taps }) // taps: odd, 1, 3, 5, ...
```

What it does: mixes each token with its neighbours, channel by channel. With `h = (taps - 1) / 2`, `y[t, i] = tanh(b[i] + sum over j of kernel[j, i] * x[t + j - h, i])`: tap `h` is the token itself, taps below it the tokens before, taps above it the tokens after. Tokens past either end read as zero. It is depthwise: channel `i` reads only channel `i` of its neighbours, so it never mixes channels. `out = in`.

| Cost                    | Formula                                      |
| ----------------------- | -------------------------------------------- |
| Parameters              | `taps * in + in`                             |
| Bytes (int6, packed)    | `ceil(6 * taps * in / 8) + ceil(6 * in / 8)` |
| Multiply-adds per token | `taps * in`                                  |

For example `conv({ taps: 3 })` after a 20-wide `embed`: 80 parameters, 60 bytes, 60 multiply-adds per token.

Initial values: the centre tap starts at 1 and the other taps uniform in `±0.1`, the bias at zero, so a new `conv` starts close to `tanh(x)`, passing each token through, and learns how much of its neighbours to mix in.

When it helps:

- When a token's label depends on the words right next to it ("to" before a unit, a number after "in"). Three taps see one word each side for 60 bytes, which a `dense` of any size cannot.
- Before `biScan`, to give the scan local features (word pairs) it would otherwise have to build up in its state.
- Cheap context: its cost grows with `taps * in`, not `in * in`, so it is the smallest layer that looks at other tokens.

When it does not:

- Context farther than `h` tokens away. Stacking `conv` layers widens the reach by `h` each time, but a `biScan` sees the whole input in both directions for a similar cost once the reach you need passes a few tokens.
- It never combines channels, so it cannot build a feature from two different features of the same token. Put a `dense` or a `biScan` after it for that.
- With `taps: 1` it is a per-channel scale and `tanh`, no context; it rarely earns its bytes.
- Positions near the ends see zeros for the missing neighbours. On inputs of one or two tokens most taps read nothing, so a wide kernel helps little there.

WGSL: one pass, one invocation per (token, channel), `ceil(n * in / 64)` workgroups of 64; only the first `n` tokens are read, whatever the buffer holds after them. Matches the CPU within 1e-5 on the software adapter (llvmpipe) for 1, 3, 5 and 7 taps with `n` = 1, 2 and the maximum token count (`test/gpu/conv.test.ts`). Real GPU: not run. Real browser: not run.

## dense

```ts
layers.dense(size, activation) // activation: 'relu' | 'tanh' | 'none'
```

What it does: a fully connected layer applied to each token on its own. For token `t`, `y[t] = act(W x[t] + b)` with `W` of shape `[size, in]` and `b` of length `size`. `out = size`. It mixes the channels of one token and never looks at other tokens.

| Cost                    | Formula                                        |
| ----------------------- | ---------------------------------------------- |
| Parameters              | `size * in + size`                             |
| Bytes (int6, packed)    | `ceil(6 * size * in / 8) + ceil(6 * size / 8)` |
| Multiply-adds per token | `size * in`                                    |

For example `dense(20, 'relu')` after a 40-wide `biScan`: 820 parameters, 615 bytes, 800 multiply-adds per token.

Initial values: weights uniform in `±sqrt(6 / in)` for `relu` (He) and `±sqrt(6 / (in + size))` for `tanh` and `none` (Glorot); biases zero.

When it helps:

- Between the sequence layers and the head, to combine what `conv` and `biScan` found into features the head can separate. One `dense` with `relu` or `tanh` gives the head a non-linear boundary instead of a linear one.
- To change width cheaply: shrink a wide `biScan` output (`2 * size`) before the head, which cuts the head's cost.

When it does not:

- It adds no context. If a token's label depends on its neighbours, add `conv` or `biScan`; a wider or deeper `dense` will not find what it cannot see.
- Its cost grows with `size * in`, the fastest-growing term in a small model. Two stacked `dense` layers rarely beat one wider one at the same byte budget on short inputs; try one first.
- `none` right before the head is a linear map the head already is (the head is linear too), so it only adds bytes. Use `relu` or `tanh` there, or drop the layer.
- `relu` units can die (stay at zero for every input) when the learning rate is high; if dev scores stall, try `tanh`.

WGSL: one pass, one invocation per (token, output channel), `ceil(n * size / 64)` workgroups of 64. Matches the CPU within 1e-5 on the software adapter (llvmpipe) for `n` = 1, 2 and the maximum token count (`test/gpu/dense.test.ts`). Real GPU: not run. Real browser: not run.

## biScan

```ts
layers.biScan(size)
```

What it does: a bidirectional gated scan over the tokens. Each direction keeps `size` channels of state. For channel `d` at token `t`, `gate = sigmoid(Wg x[t] + bg)`, `candidate = (1 - gate) * tanh(Wc x[t] + bc)` and `state[t] = gate * state[t-1] + candidate`, with the state at 0 before the first token. The forward direction runs left to right, the backward direction right to left, and the output for token `t` is the forward states followed by the backward states, so `out = 2 * size`. Because of the `(1 - gate)` factor each state is a weighted mix of the previous state and a value in (-1, 1), so it stays in (-1, 1) at any length. A gate near 1 keeps what the channel holds; a gate near 0 replaces it with the current token.

| Cost                    | Formula                                                |
| ----------------------- | ------------------------------------------------------ |
| Parameters              | `4 * size * in + 4 * size`                             |
| Bytes (int6, packed)    | `4 * ceil(6 * size * in / 8) + 4 * ceil(6 * size / 8)` |
| Multiply-adds per token | `4 * size * in + 2 * size`                             |

For example `biScan(20)` after a 32-wide `conv`: 2640 parameters, 1980 bytes, 2600 multiply-adds per token, output 40 wide.

Initial values: gate and candidate weights uniform in `±sqrt(6 / (in + size))` (Glorot); gate biases spread evenly from 0 to 3 across the channels, so at the start some channels forget within a step (gate 0.5) and others keep a token for about twenty steps (gate 0.95); candidate biases zero.

When it helps:

- When a token's label depends on tokens far away, on either side: "12 kg to pounds" needs to know that "pounds" follows "to", and the first word of a long phrase may decide the last one. `conv` sees only its taps; `biScan` sees the whole input in both directions.
- When inputs vary a lot in length. The cost is linear in the token count and the state stays bounded, so long inputs behave like short ones.
- Stacked twice, the second scan reads both directions of the first and can combine left and right context per token. Try one first; a second scan doubles the cost.

When it does not:

- When only the immediate neighbours matter. A `conv` with 3 or 5 taps costs `taps * in` per token, far less than `4 * size * in`, and trains faster.
- Its cost grows with `size * in`. A wide input (a large `embed`) makes it expensive; put a narrow `dense` or keep `embed` small before it.
- The scan is sequential over tokens: on the WGSL path one invocation walks all `n` tokens, so it parallelises over channels only. With few channels and long inputs it is the slowest op on a GPU.

WGSL: one pass, one invocation per (direction, channel), `ceil(2 * size / 64)` workgroups of 64, each looping over the `n` tokens in its direction. Matches the CPU within 1e-5 on the software adapter (llvmpipe) for `n` = 1, 2 and the maximum token count (`test/gpu/biscan.test.ts`). Real GPU: not run. Real browser: not run.

## tag head

```ts
head: { kind: 'tag', labels: ['amount', 'from', 'to', 'none'] }
```

What it does: a linear step from the last layer to one logit per label for every token, `logits[t] = W x[t] + b` with `W` of shape `[labels, in]`. `run` decodes each token to the label with the largest logit (the first label on a tie). Softmax is not part of the forward pass; training uses softmax cross-entropy per token, averaged over the tokens. Gold is one label per token of the tokenized text; labels for tokens cut at `input.max` are dropped with them.

| Cost                    | Formula                                            |
| ----------------------- | -------------------------------------------------- |
| Parameters              | `labels * in + labels`                             |
| Bytes (int6, packed)    | `ceil(6 * labels * in / 8) + ceil(6 * labels / 8)` |
| Multiply-adds per token | `in * labels`                                      |

For example four labels after `dense(20, 'relu')`: 84 parameters, 63 bytes, 80 multiply-adds per token.

Initial values: weights uniform in `±sqrt(6 / (in + labels))` (Glorot); biases zero.

When it helps:

- When the output is one label per token: spans (start, inside, none), roles of words in a phrase, or which tokens to keep.

When it does not:

- When the output is one answer for the whole input. Use the `pooled` head; tagging every token and voting wastes bytes and loses the input-level context.
- It is linear: put `dense` with `relu` or `tanh` before it when labels need a non-linear boundary.

WGSL: one pass, one invocation per (token, label), `ceil(n * labels / 64)` workgroups of 64. Argmax and decode stay on the CPU. Matches the CPU within 1e-5 on the software adapter (llvmpipe) for `n` = 1, 2 and the maximum token count, with the same decoded labels (`test/gpu/tag.test.ts`). Real GPU: not run. Real browser: not run.

## pooled head

```ts
head: { kind: 'pooled', labels: ['question', 'request', 'statement'] } // class form
head: { kind: 'pooled', outputs: 2 } // values form
```

What it does: one output for the whole input. It pools the last layer over the tokens, mean then max per channel, into a vector of `2 * in`, then takes one linear step to `out` logits: `logits = W pooled + b` with `W` of shape `[out, 2 * in]`. `out` is the label count in the class form and `outputs` in the values form. An empty input pools to zeros, so it returns the bias. The class form decodes to the label with the largest logit (the first on a tie) and trains with softmax cross-entropy; gold is one label. The values form returns the logits as numbers and trains with mean squared error over the outputs; gold is an array of `outputs` numbers.

In the backward pass the mean half spreads its gradient evenly over the tokens and the max half sends all of it to the token holding the max (the first such token on a tie), so a channel's max learns from one token per input.

| Cost                    | Formula                                                    |
| ----------------------- | ---------------------------------------------------------- |
| Parameters              | `out * 2 * in + out`                                       |
| Bytes (int6, packed)    | `ceil(6 * out * 2 * in / 8) + ceil(6 * out / 8)`           |
| Multiply-adds per token | `2 * in + 2 * in * out` (the plan's count, an upper bound) |

The plan counts the linear step per token, an upper bound: it runs once per input, so a run costs about `2 * in` per token for the pool plus `2 * in * out` once. For example three labels after `dense(20, 'relu')`: 123 parameters, 93 bytes, 160 multiply-adds per token in the plan.

Initial values: weights uniform in `±sqrt(6 / (2 * in + out))` (Glorot); biases zero.

When it helps:

- When the answer is one class or a few numbers for the whole input: the kind of a note, a language, a score, a date as numbers.
- Mean pooling sees how much of something the input holds (a share of words); max pooling sees whether something appears anywhere, wherever it is. The head gets both, so a marker word at any position counts the same.

When it does not:

- When the answer belongs to tokens, such as spans or roles of words. Use the `tag` head.
- Pooling drops order: "dog bites man" and "man bites dog" pool to the same vector after `embed`. Put `conv` or `biScan` before the head when order decides the answer.
- It is linear after the pool: put `dense` with `relu` or `tanh` before it when the classes need a non-linear boundary. The values form has no activation, so it can return any number; clamp or round in your decode step.

WGSL: two passes. The pool is one invocation per channel, `ceil(in / 64)` workgroups of 64, each looping over the `n` tokens and writing mean and max to `temp`; the linear step is one invocation per output, `ceil(out / 64)` workgroups. A model with `input.max` 1, whose buffers cannot hold `2 * in` floats, runs only the linear step on the one token. Argmax and decode stay on the CPU. Matches the CPU within 1e-5 on the software adapter (llvmpipe) for both forms with `n` = 1, 2 and the maximum token count, and for a one-token model (`test/gpu/pooled.test.ts`). Real GPU: not run. Real browser: not run.

## retrieve head

```ts
head: { kind: 'retrieve', size: 16 }
// eval: table: 'data/table.jsonl', train: { ..., negatives: 8 }
// app:  loadModel(weights, { backend: 'auto', table: [{ id: 'dark-mode', text: 'toggle dark mode' }] })
```

What it does: scores the input against every row of a table of `{ id, text }` rows and ranks them. There is one tower, used twice: the `net`, then the head's op, which is the `pooled` pool and linear step with `out = size`, turns any text into a vector of `size` floats. The table's rows go through the same tower once, at load, and each input's score against row `r` is the dot product of the two vectors. `run` returns one `{ id, score }` per row, the highest score first (table order on a tie); the table's ids type `id`.

Training: gold is the array of ids of the rows that answer the input. Each example is compared with its gold rows and `negatives` other rows drawn from the seed's stream, with softmax cross-entropy over those candidates' scores, every gold row counted as right. The score step's backward pass sends gradient to the input's side and to every candidate row's side, and both walk back through the same weights, so the tower learns from both. A step costs one tower pass per candidate, `2 + negatives` for one gold row.

| Cost                    | Formula                                                                     |
| ----------------------- | --------------------------------------------------------------------------- |
| Parameters              | `size * 2 * in + size`                                                      |
| Bytes (int6, packed)    | `ceil(6 * size * 2 * in / 8) + ceil(6 * size / 8)`                          |
| Multiply-adds per token | `2 * in + 2 * in * size` (the plan's count, an upper bound)                 |
| Score step, per run     | `rows * size` multiply-adds, not in the plan's count; no parameters         |
| At load                 | one tower pass per table row; `rows * size` floats kept, uploaded to WebGPU |

For example `size` 16 after `dense(16, 'tanh')`: 528 parameters, 396 bytes, 544 multiply-adds per token in the plan, and 960 multiply-adds per run for a table of 60 rows.

Initial values: weights uniform in `±sqrt(6 / (2 * in + size))` (Glorot); biases zero.

When it helps:

- When the answer is one of a known list that changes more often than the model: commands, help articles, settings. The list is data, not labels, so a row can be added or reworded without retraining, and a row the model never trained on still gets a score.
- When the list is long or its rows share words: the scores rank every row, so the decode step can show the top few.

When it does not:

- When the list is short and fixed. A `pooled` class head with one label per row is smaller and needs no table.
- When the input and the rows share no tokens, such as a misspelt word: a token no training text holds reaches only the embedding bias, so it adds the same vector to every input and can pull the query away from its row. On a synthetic task (`test/heads/retrieve.test.ts`, 20 rows of three made-up words, queries are noisy copies) held-out recall at 1 is 0.925 against 0.04 for the untrained tower, 1 on queries without a misspelt word, and every miss has one. Train on the misspellings and paraphrases users type, or add a rule baseline for exact matches.
- It pools, so order is lost unless `conv` or `biScan` comes before the head.

WGSL: the tower runs the `pooled` passes. The score step is one pass, one invocation per table row, `ceil(rows / 64)` workgroups, reading the input's vector from the tower's output and the row vectors from the weights buffer, where they follow the model's parameters. Ranking stays on the CPU. The score step matches the CPU within 1e-5 for 1, 20 and 130 rows, the head op for `n` = 1, 2 and the maximum token count, and a whole retrieve model's scores against 100 rows within 1e-3 on 30 inputs, all on the software adapter (llvmpipe) (`test/gpu/retrieve.test.ts`). Real GPU: not run. Real browser: not run.

<!-- vitehub-docs:section-links -->
## In this section

- [Layer catalog](https://gpu-engine.dev/raw/docs/reference/layers.md) (this page)
- [CLI](https://gpu-engine.dev/raw/docs/reference/cli.md)
- [Diagnostics](https://gpu-engine.dev/raw/docs/reference/diagnostics.md)

Next: [CLI](https://gpu-engine.dev/raw/docs/reference/cli.md)

All pages: [llms.txt](https://gpu-engine.dev/llms.txt)
