Proofs
Can a model find a matching identity, recover a hidden subtotal, or preserve enough information through a nested summary? These experiments isolate one relationship at a time. Controls remove the signal, change the pairing, or deliberately discard information so that a useful prediction has a clear explanation.
Each scenario opens with a short Insights summary: what the proof shows, why it matters, and where the conclusion stops. A model tree, contrasting records, and expandable code sections explain the experiment together. Each page comes from the same Python file you can download and run. The collection covers 50 scenarios in 23 families, including retention, adaptation, reset, deactivation, and deletion after schema mutations.
Read The Status
Statuses summarize recorded evidence in proofs/results.yaml. Each experiment has a stable ID, such as P014, independent of its title or script name. Some experiments are designed to confirm a limitation, such as losing count when averaging identical values. Their expected outcome is useful evidence about model design. Each page provides commands to reproduce the experiment. Acceptance thresholds are labeled as gates. Each page shows the latest full run’s measurements, with behavioral checks collapsed by default. Expand them to inspect the individual criteria. Short smoke runs never replace full experiment results.
| Status | What the evidence establishes |
|---|---|
| Provisional pass | The reported experiment clears its gates, usually for one deterministic seed. Broader stability or generalization remains to be established. |
| Expected limitation | The experiment reproduces a specific loss of information or unsuccessful modeling route. Its checks are met when that limitation appears. |
| Partial evidence | Internal behavior passes a mechanistic check, while a claim such as held-out prediction or partition recovery remains untested. |
| Gates not met | The latest full run completed but did not meet every stated behavioral criterion. |
| Execution error | The latest full run could not complete. Its error is recorded separately from model quality. |
| Not run | No full experiment run has been recorded. |
These are empirical checks of learned behavior. Training speed and memory use require separate measurements. Learned arithmetic is approximate; use exact preprocessing for fixed business rules that must be exact.
Explore The Proofs
Sort by ID, Proof, Family, or Status, using the column headings or the order menu. Filter by an ID, task, family, or status, then select a proof to explore its insights, examples, and evidence.
| ID | Proof | Family | Status |
|---|---|---|---|
| P022 | Retrieve Through Learned Summaries | Argmax retrieval | Provisional pass |
| P023 | Retrieve the Winning Payload | Argmax retrieval | Provisional pass |
| P024 | Copying by Position | Associative recall | Provisional pass |
| P025 | Recall by an Unseen Key | Associative recall | Provisional pass |
| P042 | Combining Weekday and Hour | Calendar reasoning | Provisional pass |
| P045 | Day-of-Year Cannot Identify Weekday | Calendar reasoning | Expected limitation |
| P044 | Learning Month from Annual Phase | Calendar reasoning | Provisional pass |
| P043 | Resolving the Leap-Day Boundary | Calendar reasoning | Expected limitation |
| P017 | Signal and Noise | Calibration | Provisional pass |
| P003 | A mean that survives bag duplication | Cardinality generalization | Provisional pass |
| P005 | Extending a visible-count sum | Cardinality generalization | Provisional pass |
| P004 | Restoring sums with a visible count | Cardinality generalization | Provisional pass |
| P002 | Sums beyond trained collection lengths | Cardinality generalization | Provisional pass |
| P001 | Sums within the trained length range | Cardinality generalization | Provisional pass |
| P026 | Average the Requested Group | Category-conditioned reduction | Provisional pass |
| P027 | Choose a Group and an Operation | Category-conditioned reduction | Provisional pass |
| P019 | A Plain Cluster Input Stays Dormant | Clustering | Expected limitation |
| P018 | Cluster Commitment with Hidden Regimes | Clustering | Partial evidence |
| P020 | Cluster Commitment with Stable Labels | Clustering | Partial evidence |
| P029 | Compare Two Unseen Identities | Collection overlap | Provisional pass |
| P028 | The Collection Overlap Boundary | Collection overlap | Expected limitation |
| P008 | Averaging supplied centered products | Distribution statistics | Provisional pass |
| P009 | Averaging supplied squared deviations | Distribution statistics | Provisional pass |
| P006 | Covariance from aligned pairs | Distribution statistics | Provisional pass |
| P007 | Variance from raw values | Distribution statistics | Provisional pass |
| P011 | Selecting and summing supplied contributions | Grouped weighted aggregation | Provisional pass |
| P010 | Weighted sums for a requested group | Grouped weighted aggregation | Provisional pass |
| P041 | An Average Does Not Preserve Count | Hierarchical statistics | Expected limitation |
| P039 | Learning a Fixed-Length Sum | Hierarchical statistics | Provisional pass |
| P040 | Preserving Counts Across a Hierarchy | Hierarchical statistics | Provisional pass |
| P021 | Equality for Unseen Identities | Identity | Provisional pass |
| P030 | Keep Each Subtotal with Its Item | Item alignment | Provisional pass |
| P049 | Deactivate a Learned Input and Restore Its Contribution | Mutation ablation | Provisional pass |
| P050 | Delete a Learned Input and Adapt to the Remaining Information | Mutation ablation | Provisional pass |
| P047 | Learn an Added Target Without Losing Existing Tasks | Mutation adaptation | Gates not met |
| P048 | Reset One Learned Target and Teach It Again | Mutation reset | Provisional pass |
| P046 | Learned Predictions Survive Neutral Edits | Mutation retention | Provisional pass |
| P031 | Choose a Numeric Operation | Operation-conditioned reduction | Provisional pass |
| P012 | Requesting a value by rank | Order statistics | Provisional pass |
| P032 | Compare Each Item with Its Group | Peer-relative inference | Provisional pass |
| P035 | Compare Each Item with Its Peers | Peer-relative inference | Provisional pass |
| P033 | Share One Summary Across Grouped Targets | Peer-relative inference | Provisional pass |
| P034 | Subtract a Supplied Peer Mean | Peer-relative inference | Provisional pass |
| P036 | Transfer Values by Category Identity | Sibling entity transfer | Provisional pass |
| P037 | Transfer Values by an Unseen Identity | Sibling entity transfer | Provisional pass |
| P038 | Null Is Not Zero | Value state | Provisional pass |
| P015 | Summing supplied contributions | Weighted aggregation | Provisional pass |
| P016 | Weighted means across varying lengths | Weighted aggregation | Provisional pass |
| P014 | Weighted sums from raw pairs | Weighted aggregation | Provisional pass |
| P013 | When Mean loses contribution count | Weighted aggregation | Expected limitation |
What Strengthens A Result
The shared protocol calls for three paired data/model seeds before promoting a core claim, plus at least ten seeds to calibrate a new threshold. Held-out splits, matched baselines, and controls must continue to support the claimed relationship. Each page identifies the additional work for that scenario.
Regression errors are generally reported as nRMSE: model RMSE divided by the RMSE of a baseline that predicts the training mean. Lower is better; 1 matches that baseline. Other baselines, such as an oracle with information removed, are identified on the relevant page. Classification pages report accuracy or ROC AUC, together with the appropriate chance or corruption control.
Run uv run python proofs/run.py --list to list proof IDs, or uv run python proofs/run.py P014 --accelerator gpu to run one on a GPU. Each script uses seeded synthetic records and records its metrics, controls, seed, environment, and source fingerprint. make proofs runs all scripts serially.