Evaluation

Evaluate the task you need the model to solve, alongside its aggregate reconstruction loss. A model can improve its combined objectives while regressing on the output that matters to an application.

The modeling proofs give worked examples of held-out metrics, corrupted-input controls, and deliberately lossy representations. Each scenario includes its model tree and the current evidence for its claim.

Read The Metrics

With a trained model and a configured validation split:

import relflow as rf

metrics = trainer.validate(model=model, datamodule=datamodule)
validation_loss = metrics[0]["loss/validate"]

relflow logs epoch-level metrics. Individual tensorfields determine which content metrics are available.

Metric	Interpretation
loss/train, loss/validate, loss/test	Sum of active leaf losses after each leaf’s weight is applied.
order.returned/validate.loss.state	Reconstruction loss for value state at order/returned.
order.returned/validate.loss.content	Datatype-specific content loss at that address.
order.returned/validate.auc.content	Example Boolean content metric; availability depends on datatype.
throughput/train	Estimated observations per second across the training epoch.

Metric names convert schema address slashes to dots. A leaf can contribute both state and content loss. State metrics describe valued, null, padded, masked, and other states; content metrics describe the value being predicted. Leaf metrics are logged before weighting, so their unweighted sum need not match the total loss.

If no reconstruction coordinate is selected anywhere across distributed ranks, the training batch is skipped. Validation/test return zero loss for that batch. Repeated empty-objective warnings suggest a mismatch among masks, source geometry, and split contents; they do not demonstrate learning.

Compare Models Fairly

Use the same held-out observations, target definitions, preprocessing, and masking policy across comparisons. Keep entity groups and time boundaries consistent with deployment; construct those splits before supplying them to the data module.

Choose an output-specific metric alongside loss/validate. Number errors and categorical accuracy measure different things; their scale and eligible coordinates differ. A state-accuracy improvement alone does not establish better content predictions. Use validation data to choose thresholds and checkpoints, leaving the test split for a final assessment.

Boolean fields accept one threshold or a list of thresholds for metrics:

returned = rf.Boolean(mask=True, threshold=[0.5, 0.7, 0.9])

This changes threshold-qualified accuracy, precision, recall, and specificity metrics. It does not change the training loss or the probability-only prediction payload. Category topk likewise adds datatype-specific metrics; see the relevant datatype reference.

Measure Dependence On An Input

An ablation temporarily removes an input and reevaluates the trained model:

price = rf.where("address") == "order/line_items/price"
baseline = trainer.test(model=model, datamodule=test_data)[0]
with model.override(price, active=False):
    ablated = trainer.test(model=model, datamodule=test_data)[0]
loss_change = ablated["loss/test"] - baseline["loss/test"]

Here test_data is a data module with a held-out test split and the same model. Compare the target’s task metric as well as total loss. Missing-input behavior can be out of distribution for a model that never saw such absence; training with suitable structural dropout can make the comparison more useful. An ablation shows dependence under this experiment, not causation.

override restores schema attributes on exit, not a snapshot of all learned state. See Schema Mutation for its boundaries.