Seed 3615 on gpu: 2 of 2 behavioral checks met. See the measurements below. The experiment code has changed since this run; these measurements describe its earlier version.
The learned sum extends beyond trained collection lengths and responds approximately to duplication. Longer held-out bags test length extrapolation. Duplicating complete bags checks that predictions roughly double, while equal-value probes isolate the need to distinguish different repetition counts.
Spare branch capacity only permits longer inputs; it does not establish their meaning. These behavioral checks provide the evidence that the model uses accumulated mass. The equal-value probes themselves use familiar lengths, so they complement rather than replace the unseen-length test. The result supports a bounded extension beyond training, not arbitrary-length or exact summation; missing values and nested totals require separate checks.
"""Characterize sum extrapolation beyond every trained collection length.
Require unseen-length accuracy and exact complete-bag scaling together.
Run: uv run python proofs/run.py P002"""
from __future__ import annotations
from collections.abc import Iterator
from copy import deepcopy
import lightning.pytorch as lit
import numpy as np
import torch
from reporting import report
import relflow as rf
PROOF_ID = "P002"
TRAIN_MAX = 6
CAPACITY = 12Targets use mask=True; the labels below are supervision hidden from the encoder.
items:
- {amount: 0.5}
- {amount: 0.5}
- {amount: 0.5}
- {amount: 0.5}
- {amount: 0.5}
- {amount: 0.5}
- {amount: 0.5}
- {amount: 0.5}
total: 4.0Eight items require a total of 4.0, although training bags contain at most six.
items:
- {amount: 0.3}
- {amount: -0.1}
total: 0.2This shorter bag sums to 0.2 and fits inside the trained length range.
items:
- {amount: 0.3}
- {amount: -0.1}
- {amount: 0.3}
- {amount: -0.1}
total: 0.4The empirical distribution is unchanged, but the total doubles to 0.4. The proof compares predictions before and after duplication, in addition to its separate unseen-length accuracy gate.
def random_records(*, rows: int, seed: int, minimum: int = 1, maximum: int = TRAIN_MAX) -> Iterator[dict]:
"""Draw variable-length numerical bags and their mean and sum."""
if not 1 <= minimum <= maximum <= CAPACITY:
raise ValueError(f"length range must satisfy 1 <= minimum <= maximum <= {CAPACITY}")
rng = np.random.default_rng(seed)
for _ in range(rows):
length = int(rng.integers(minimum, maximum + 1))
values = rng.uniform(-1.0, 1.0, size=length)
row: dict[str, object] = {
"items": [{"amount": float(value)} for value in values],
"mean_amount": float(values.mean()),
"total": float(values.sum()),
}
yield row
def duplicate(observations: list[dict]) -> list[dict]:
"""Duplicate every complete bag and update its algebraic targets."""
rows = deepcopy(observations)
if any((len(row["items"]) * 2 > CAPACITY for row in rows)):
raise ValueError(f"duplicated collection exceeds configured capacity {CAPACITY}")
for row in rows:
row["items"] = [*row["items"], *row["items"]]
row["total"] *= 2.0
return rows
def equal_value_probes(*, value: float, lengths: tuple[int, ...]) -> list[dict]:
"""Create matched bags that differ only in repetition count."""
if not lengths or min(lengths) < 1 or max(lengths) > CAPACITY:
raise ValueError(f"probe lengths must be within 1..{CAPACITY}, got {lengths!r}")
rows: list[dict[str, object]] = []
for length in lengths:
row: dict[str, object] = {"items": [{"amount": value}] * length, "mean_amount": value, "total": length * value}
rows.append(row)
return rows
def prediction(model: rf.Model, observations: list[dict]) -> np.ndarray:
inputs = [{"items": row["items"]} for row in observations]
output = model.predict(inputs)["predictions"].to_pylist()
return np.asarray([row["record/total"]["content"] for row in output], dtype=np.float64)
def rmse(actual: np.ndarray, predicted: np.ndarray | float) -> float:
return float(np.sqrt(np.mean(np.square(actual - predicted))))
def score(*, train: list[dict], test: list[dict], predicted: np.ndarray) -> dict[str, float]:
"""Compare held-out RMSE with the constant training-target mean."""
actual = np.asarray([row["total"] for row in test], dtype=np.float64)
baseline = rmse(actual, float(np.asarray([row["total"] for row in train], dtype=np.float64).mean()))
measured = rmse(actual, predicted)
return {"rmse": measured, "baseline_rmse": baseline, "nrmse": measured / baseline}
The schema matches the in-range sum: learned rf.Attention reductions receive raw amounts without a supplied count. Training lengths are one through six, while the unseen test set uses seven through ten. Branch capacity is twelve, allowing these inputs without overflow.
A second intervention duplicates complete bags of at most five items and checks that predictions approximately double. Equal-value probes at lengths one, three, and six isolate sensitivity to multiplicity. The capacity bound permits these shapes; the accuracy and intervention gates establish the learned behavior.
def run(seed: int, steps: int | None, accelerator: str) -> tuple[dict, dict]:
lit.seed_everything(seed, workers=True)
# Split seeds are independent; rerunning a generator reproduces the same records.
train = list(random_records(rows=1536, seed=seed + 1))
in_range = list(random_records(rows=512, seed=seed + 3))
unseen = list(random_records(rows=512, seed=seed + 4, minimum=7, maximum=10))
short = list(random_records(rows=256, seed=seed + 5, maximum=5))
model = rf.Model(
d_model=32,
n_layers=2,
n_heads=4,
reduction=rf.Attention(n_layers=2),
batch_size=64,
optimizer=lambda module: torch.optim.Adam(module.parameters(), lr=0.001),
items=rf.Branch(
length=CAPACITY, attention="mha", n_layers=2, reduction=rf.Attention(n_layers=2), amount=rf.Number
),
total=rf.Number(mask=True, objective="mse"),
)
datamodule = rf.SyntheticDataModule(
model=model,
train=lambda: random_records(rows=1536, seed=seed + 1),
validate=lambda: random_records(rows=384, seed=seed + 2),
seed=seed,
)
trainer = lit.Trainer(
accelerator=accelerator,
devices=1,
max_epochs=-1,
max_steps=1100 if steps is None else min(steps, 1100),
logger=False,
enable_progress_bar=False,
enable_model_summary=False,
enable_checkpointing=False,
deterministic=True,
num_sanity_val_steps=0,
)
trainer.fit(model, datamodule=datamodule)
# Evaluate held-out answers and retain their original labels in corruption controls.
in_range_score = score(train=train, test=in_range, predicted=prediction(model, in_range))
unseen_score = score(train=train, test=unseen, predicted=prediction(model, unseen))
short_prediction = prediction(model, short)
duplicated_prediction = prediction(model, duplicate(short))
duplication_error = rmse(2.0 * short_prediction, duplicated_prediction) / float(
np.std(np.asarray([row["total"] for row in unseen], dtype=np.float64))
)
probes = equal_value_probes(value=0.65, lengths=(1, 3, 6))
probe_prediction = prediction(model, probes)
probe_target = np.asarray([row["total"] for row in probes], dtype=np.float64)
probe_error = rmse(probe_target, probe_prediction) / float(
np.std(np.asarray([row["total"] for row in unseen], dtype=np.float64))
)
metrics = {
"in_range_score": in_range_score,
"unseen_score": unseen_score,
"duplication_error": duplication_error,
"probe_error": probe_error,
"probe_target": probe_target.tolist(),
"probe_prediction": probe_prediction.tolist(),
"steps": trainer.global_step,
}
checks = {
"All measurements finite": bool(
np.isfinite([in_range_score["nrmse"], unseen_score["nrmse"], duplication_error, probe_error]).all()
),
"Seen nRMSE < 0.35, unseen nRMSE < 0.30, duplication and cardinality errors < 0.20": bool(
in_range_score["nrmse"] < 0.35
and unseen_score["nrmse"] < 0.3
and (duplication_error < 0.2)
and (probe_error < 0.2)
),
}
return metrics, checksSeed 3615, gpu, recorded 2026-09-15T02:17:20.021979+00:00. Outcome: met.
Source fingerprint: 70dfbf74a5344939544a0869e108d29b141f31a431040ac421f69ac2484d3787. Python 3.12.6; Torch 2.12.0.
| Measurement | Value |
|---|---|
| in_range_score | rmse: 0.0177695; baseline_rmse: 1.04012; nrmse: 0.017084 |
| unseen_score | rmse: 0.131842; baseline_rmse: 1.78742; nrmse: 0.0737611 |
| duplication_error | 0.0699551 |
| probe_error | 0.0484787 |
| probe_target | 0.65, 1.95, 3.9 |
| probe_prediction | 0.669512, 1.96077, 3.75162 |
| steps | 1100 |
| Behavioral check | Outcome |
|---|---|
| All measurements finite | Met |
| Seen nRMSE < 0.35, unseen nRMSE < 0.30, duplication and cardinality errors < 0.20 | Met |
Repeat the gates across seeds, capacities, and nested branches. Heavy tails, high duplication, empty collections, missing values, and overflow semantics need separate coverage. The doubling gate allows approximation error; it is not an exact-sum contract.
The family’s promotion target is at least three core seeds and ten lightweight calibration seeds.
Run by stable ID from the repository root:
uv run python proofs/run.py P002Or run the self-contained script directly:
PYTHONPATH=proofs uv run python proofs/aggregation/attention_sum_unseen_lengths.pyAdd --accelerator gpu for CUDA or --seed 42 for another seeded experiment. --steps 2 checks execution with a short training budget; it is recorded as a smoke run.
if __name__ == "__main__":
report(PROOF_ID, run, seed=3615)