Every task is a known composition of primitives — the structure — instantiated with concrete parameters and data — the surface.
We generate the tasks, so we know the structure. The model is never told it. There is no docstring. It sees input/output pairs and must induce the rule, then write transform(xs).
What the model sees
That is the whole prompt. No docstring, no list of primitives, no hint that there are two steps.
The hidden structure
? → ?
Try it first. Look at the pairs on the left and say the rule out loud. That guess is exactly the induction problem the model is handed.
One demo, pushed through the composition
How verification works
Exact match, element by element. No LLM judge, no partial credit. The model sees the 4 demos; the executor also runs 4 hidden tests it was never shown. Fail one case and the task is unsolved.
| Held-out set | What it measures | Base |
|---|---|---|
| HO-surface | Same structures as training, unseen surface. Specialisation: did it learn the transformation, or memorise these particular lists? | 0.100 |
| HO-composition | Novel compositions of the same primitives; templates never trained on. Systematic generalisation — the one the whole project cared about. | 0.083 |
| FORGET probe | MBPP+, ordinary unrelated Python problems, never trained on. What the update destroyed. | 0.847 |
The generator draws every structure from this vocabulary. The model is not shown this list either.
Honesty note.
Scale note.