Cluster Commitment with Hidden Regimes

Adaptive cluster state can settle near five generating regression regimes without explicit group labels.

Each repeated identity follows one of five hidden relationships between x and y. The model receives no group label. This experiment asks whether adaptive Cluster state settles near the generating number of regimes.

Seed 0 on gpu: 4 of 4 behavioral checks met. See the measurements below.

Insights

Settling near five clusters does not yet show that the model discovered the five generating functions. Repeated identities follow stable regimes while their numeric inputs vary; reconstruction engages adaptive Cluster state alongside the supervised numeric task.

The recorded gates inspect terminal count and usage. Count is calculated from usage perplexity, so those diagnostics are coupled. Neither tells us whether identities share the correct regime or whether predictions work on independent observations.

The current train and validation tables are identical. Evaluate held-out regression and assignment agreement before treating this as regime discovery. The existing result supports adaptive-state behavior, with the harder behavioral claim still open.

Setup

"""P018: track adaptive clusters across five hidden regression regimes.

Training and validation deliberately reuse observations. Count and usage
perplexity diagnose the mechanism; they do not prove partition recovery or
held-out prediction quality.
"""

from collections.abc import Iterator
from functools import partial

import lightning.pytorch as lit
import numpy as np
import torch
from reporting import report

import relflow as rf
from relflow.tensorfields.extensions.cluster import Embedder

PROOF_ID = "P018"

Examples

These are idealized values before observation noise, with illustrative regime assignments. The seeded generator assigns regimes randomly; these ID-to-regime choices are not reported measurements. y is always hidden from embedding.

One identity at the start of its curve

id: id-0007
x: 0.0
y: 10.0

An identity assigned to sin(x) + 10 has a noiseless target of 10 here.

The same identity at another input

id: id-0007
x: 1.5707963267948966
y: 11.0

At approximately π/2, the same regime gives 11. Its target varies with x, but its generating group stays fixed.

Another regime at the same input

id: id-0042
x: 1.5707963267948966
y: -2.0

An identity assigned to the constant -2 regime gives a different answer for the same x. Repeated identity evidence can distinguish these processes. The examples describe the generating functions, not verified learned assignments or held-out predictions.

Synthetic data and controls

def records(seed: int) -> Iterator[dict]:
    """Repeat 100 identities thirty times across five hidden regression regimes."""
    rng = np.random.default_rng(seed)
    functions = (
        lambda x: np.sin(x) + 10.0,
        lambda x: np.cos(x) + 4.0,
        lambda x: np.zeros_like(x) - 2.0,
        lambda x: 0.5 * x - 8.0,
        lambda x: -0.3 * x - 14.0,
    )
    id_to_cluster = rng.integers(0, 5, size=100)
    identities = np.repeat(np.arange(100), 30)
    clusters = id_to_cluster[identities]
    x = rng.uniform(0.0, 10.0, size=3000) + rng.normal(0.0, 0.05, size=3000)
    y = np.empty(3000)
    for cluster, function in enumerate(functions):
        selected = clusters == cluster
        y[selected] = function(x[selected]) + rng.normal(0.0, 0.05, size=int(selected.sum()))
    for index in rng.permutation(3000):
        yield {"id": f"id-{identities[index]:04d}", "x": float(x[index]), "y": float(y[index])}


class Trajectory(lit.Callback):
    """Inspect Cluster internals; these diagnostics do not establish partition accuracy."""

    def __init__(self, address: rf.Address) -> None:
        self.address = address
        self.rows: list[dict] = []

    def on_train_epoch_end(self, trainer: lit.Trainer, pl_module: lit.LightningModule) -> None:
        embedder = pl_module.nodes[self.address].embedder
        if not isinstance(embedder, Embedder):
            raise TypeError(f"{self.address} requires a Cluster embedder, got {type(embedder).__name__}")
        usage = embedder.usage_ema.detach()
        probabilities = usage / usage.sum().clamp_min(1e-12)
        bounded = probabilities.clamp_min(1e-12)
        entropy = -(bounded * bounded.log()).sum()
        self.rows.append(
            {
                "epoch": trainer.current_epoch,
                "n_committed": int(embedder.committed.sum().item()),
                "perplexity": float(torch.exp(entropy).item()),
                "adherence": float(embedder.adherence_ema.item()),
            }
        )

Model tree

Figure 1: The ID uses a 50% training mask with reconstruction enabled. The supervised y target is always hidden from input.

How it works

The generator supplies 100 identities with 30 observations each. Its five regimes are sin(x) + 10, cos(x) + 4, -2, 0.5x - 8, and -0.3x - 14, with small additive noise. The repeated identity lets the model associate behavior across changing x values.

The ID field uses rf.Mask(rate=0.5, reconstruct=True) to engage the Cluster reconstruction loss. The numeric objective adds pressure to explain y. Cluster assignments belong to identities; row-varying x does not directly condition the query that reconstructs the identity itself.

The model trains for 30 deterministic epochs. Training and validation currently use the same 3,000 observations. An internal-state callback measures the final five epochs.

Training and evaluation

def run(seed: int, steps: int | None, accelerator: str) -> tuple[dict, dict]:
    lit.seed_everything(seed, workers=True)
    model = rf.Model(
        name="obs",
        d_model=32,
        n_layers=1,
        n_heads=4,
        batch_size=128,
        id=rf.Cluster(capacity=128, n_clusters=(3, 15), mask=rf.Mask(rate=0.5, reconstruct=True)),
        x=rf.Number,
        y=rf.Number(mask=True),
    )
    model.optimizer = lambda module: torch.optim.AdamW(module.parameters(), lr=3e-3)
    source = partial(records, seed=seed + 42)
    data = rf.SyntheticDataModule(model=model, train=source, validate=source, seed=seed)
    trajectory = Trajectory(rf.Address("obs", "id"))
    trainer = lit.Trainer(
        accelerator=accelerator,
        max_epochs=30,
        max_steps=steps if steps is not None else -1,
        callbacks=[trajectory],
        logger=False,
        enable_progress_bar=False,
        enable_model_summary=False,
        enable_checkpointing=False,
        deterministic=True,
    )
    trainer.fit(model=model, datamodule=data)

    tail = trajectory.rows[-5:]
    committed = [row["n_committed"] for row in tail]
    perplexity = [row["perplexity"] for row in tail]
    return {
        "trajectory": trajectory.rows,
        "final_committed": committed,
        "final_perplexity": perplexity,
    }, {
        "Final committed counts remain between 4 and 7": all(4 <= value <= 7 for value in committed),
        "Final perplexity remains within 1.5 of five": all(abs(value - 5) <= 1.5 for value in perplexity),
        "Terminal commitment is above the lower bound": committed[-1] != 3,
        "Terminal commitment is below the upper bound": committed[-1] != 15,
    }

Evidence

Latest full run

Seed 0, gpu, recorded 2026-09-15T02:23:27.629749+00:00. Outcome: met.

Source fingerprint: 4035bdca26e7f43662ce98fbb02dc392c9435b4288004b86dd61d42aa111ed8a. Python 3.12.6; Torch 2.12.0.

Measurement	Value
trajectory	30 values; final 8: epoch: 22; n_committed: 6; perplexity: 5.62755; adherence: 0.00186418, epoch: 23; n_committed: 6; perplexity: 5.53046; adherence: 0.00160209, epoch: 24; n_committed: 5; perplexity: 5.43456; adherence: 0.0013695, epoch: 25; n_committed: 5; perplexity: 5.34655; adherence: 0.00119096, epoch: 26; n_committed: 5; perplexity: 5.26736; adherence: 0.00103886, epoch: 27; n_committed: 5; perplexity: 5.20353; adherence: 0.000903981, epoch: 28; n_committed: 5; perplexity: 5.13021; adherence: 0.000823084, epoch: 29; n_committed: 5; perplexity: 5.07396; adherence: 0.00106301
final_committed	5, 5, 5, 5, 5
final_perplexity	5.34655, 5.26736, 5.20353, 5.13021, 5.07396

Behavioral checks

Behavioral check	Outcome
Final committed counts remain between 4 and 7	Met
Final perplexity remains within 1.5 of five	Met
Terminal commitment is above the lower bound	Met
Terminal commitment is below the upper bound	Met

Recorded results.

Remaining work

Create independent train, validation, and test observations. Measure held-out regression against a marginal baseline and require permutation-invariant partition recovery with ARI ≥ 0.80. Add unique-identity and no-regime controls, and replace internal instrumentation with a public diagnostic when available.

Repeat all behavioral and partition gates across three core seeds and at least ten calibration seeds before promoting this proof.

Reproduce

Run by stable ID from the repository root:

uv run python proofs/run.py P018

Or run the self-contained script directly:

PYTHONPATH=proofs uv run python proofs/cluster/hidden_regression_regimes.py

Add --accelerator gpu for CUDA or --seed 42 for another seeded experiment. --steps 2 checks execution with a short training budget; it is recorded as a smoke run.

Download the complete proof.

if __name__ == "__main__":
    report(PROOF_ID, run, seed=0)