Signal and Noise

A held-out evaluation should recognize an obvious signal and leave independent noise at chance.

Before interpreting a difficult learning task, check that the evaluation can separate an easy relationship from noise. This proof changes one Boolean input while keeping the model, distractors, and training budget the same.

Seed 7 on gpu: 3 of 3 behavioral checks met. See the measurements below.

Insights

A meaningful learning check should recognize a real signal and remain at chance when that signal is absent. The two models have the same schema and training budget; only the relationship between the visible Boolean and the target changes.

The positive-signal and independent-noise checks distinguish useful learning from a model that appears successful regardless of the data. Separate train, validation, and test streams help distinguish learned relationships from memorized observations.

Use this as a basic check before interpreting harder proof failures. Copying a Boolean does not establish aggregation or relational reasoning, and the one-seed result does not measure training efficiency.

Setup

"""P017: distinguish an obvious Boolean signal from independent noise.

Only the relationship between ``leak`` and ``target`` changes. Both models use
independent train, validation, and test streams and the same training budget.
"""

from collections.abc import Iterator
from functools import partial

import lightning.pytorch as lit
import numpy as np
import torch
from reporting import report

import relflow as rf

PROOF_ID = "P017"

Examples

These records illustrate the two generating processes. target is hidden from embedding by mask=True; the displayed labels are supervision.

Visible positive signal

x: -0.7
segment: segment-3
tags:
  - round
  - hot
leak: true
target: true

In the signal process, leak always equals target. The other fields are independent distractors.

Visible negative signal

x: 0.2
segment: segment-1
tags:
  - red
leak: false
target: false

The rule applies to both labels. Copying the visible Boolean would solve this positive-control task.

An independent-noise record

x: -0.7
segment: segment-3
tags:
  - round
  - hot
leak: true
target: false

This record is possible in the separately generated noise process. Here the Boolean and target are independent, so either target can accompany the same inputs. This is a sampled label, not a reliably inferable answer.

Synthetic data and controls

def records(*, rows: int, seed: int, signal: bool) -> Iterator[dict]:
    """Yield balanced labels with matching or independent Boolean inputs."""
    rng = np.random.default_rng(seed)
    target = np.tile(np.array([False, True]), (rows + 1) // 2)[:rows]
    rng.shuffle(target)
    independent = rng.integers(0, 2, size=rows).astype(bool)
    tags = np.array(["red", "blue", "round", "square", "hot", "cold"])
    tag_rows = [rng.choice(tags, size=int(rng.integers(0, 4)), replace=False).tolist() for _ in range(rows)]
    values = rng.normal(size=rows)
    segments = rng.integers(0, 4, size=rows)
    for index in range(rows):
        yield {
            "x": float(values[index]),
            "segment": f"segment-{segments[index]}",
            "tags": tag_rows[index],
            "leak": bool(target[index] if signal else independent[index]),
            "target": bool(target[index]),
        }

Model tree

Figure 1: The two variants change only the leak–target relationship. The target is always hidden from model inputs.

How it works

The positive model can learn to copy the visible Boolean through the shared context. A separately trained control sees the same schema, but leak is independent of the balanced target. That model should have no reliable way to rank positive examples above negative ones.

Each model trains on 1,024 rows, validates on 512, and tests on 2,048. The three splits use independent random streams. Both runs use 12 deterministic epochs, so a difference in the information available explains the contrast.

Training and evaluation

def fit(*, signal: bool, seed: int, steps: int | None, accelerator: str) -> float:
    """Fit one relationship and measure its held-out Boolean AUC."""
    lit.seed_everything(seed, workers=True)
    model = rf.Model(
        name="calibration",
        d_model=24,
        n_layers=1,
        n_heads=4,
        batch_size=128,
        x=rf.Number,
        segment=rf.Category(size=4, p_unavailable=0.0),
        tags=rf.Set(size=6, p_unavailable=0.0),
        leak=rf.Boolean,
        target=rf.Boolean(mask=True),
    )
    model.optimizer = lambda module: torch.optim.AdamW(module.parameters(), lr=5e-3)
    data = rf.SyntheticDataModule(
        model=model,
        train=partial(records, rows=1024, seed=seed + 1, signal=signal),
        validate=partial(records, rows=512, seed=seed + 2, signal=signal),
        test=partial(records, rows=2048, seed=seed + 3, signal=signal),
        seed=seed,
    )
    trainer = lit.Trainer(
        accelerator=accelerator,
        max_epochs=12,
        max_steps=steps if steps is not None else -1,
        logger=False,
        enable_progress_bar=False,
        enable_model_summary=False,
        enable_checkpointing=False,
        deterministic=True,
    )
    trainer.fit(model=model, datamodule=data)
    metrics = trainer.test(model=model, datamodule=data, verbose=False)[0]
    return float(metrics["calibration.target/test.auc.content"])


def run(seed: int, steps: int | None, accelerator: str) -> tuple[dict, dict]:
    signal_auc = fit(signal=True, seed=seed, steps=steps, accelerator=accelerator)
    noise_auc = fit(signal=False, seed=seed, steps=steps, accelerator=accelerator)
    gap = signal_auc - noise_auc
    return {"signal_auc": signal_auc, "noise_auc": noise_auc, "auc_gap": gap}, {
        "Signal AUC is at least 0.98": signal_auc >= 0.98,
        "Independent noise AUC remains between 0.42 and 0.58": 0.42 <= noise_auc <= 0.58,
        "Signal exceeds noise by at least 0.40 AUC": gap >= 0.40,
    }

Evidence

Latest full run

Seed 7, gpu, recorded 2026-09-15T02:23:17.604572+00:00. Outcome: met.

Source fingerprint: 388d96f0024c691fa3c4ea225efa769f77981b7577578bf8077f8e1ca00decb5. Python 3.12.6; Torch 2.12.0.

Measurement	Value
signal_auc	1
noise_auc	0.475399
auc_gap	0.524601

Behavioral checks

Behavioral check	Outcome
Signal AUC is at least 0.98	Met
Independent noise AUC remains between 0.42 and 0.58	Met
Signal exceeds noise by at least 0.40 AUC	Met

Recorded results.

Remaining work

Run three paired core seeds and a calibration panel of at least ten seeds. Record the resulting distributions before freezing the gates, and reconcile the implemented 2,048 test rows with the proof plan’s 4,096-row requirement.

Reproduce

Run by stable ID from the repository root:

uv run python proofs/run.py P017

Or run the self-contained script directly:

PYTHONPATH=proofs uv run python proofs/calibration/signal_vs_noise.py

Add --accelerator gpu for CUDA or --seed 42 for another seeded experiment. --steps 2 checks execution with a short training budget; it is recorded as a smoke run.

Download the complete proof.

if __name__ == "__main__":
    report(PROOF_ID, run, seed=7)