Evaluation

Measure whether the model finds more disputes at the team’s review capacity, and compare each addition fairly.

The payment review team can investigate only a limited number of payments. Evaluate whether the model finds more later disputes at that review capacity, alongside validation loss. Start with the payment-and-history model before adding profile fields or another task. When a model has several targets, an improving combined loss can hide a regression in an individual outcome.

Read the metrics

With a trained model and a configured validation split:

import relflow as rf

metrics = model.validate(datamodule, batch_size=256)
validation_loss = metrics[0]["loss/validate"]
dispute_auc = metrics[0][".trxns.disputed/validate.auc.content"]

Set the evaluation batch size on the method call; see Batch size for its persistent model setting and per-device meaning.

relflow uses Lightning’s native logging. Attached loggers receive training metrics only at the configured step cadence, every log_every_n_steps optimizer updates. Validation and test losses and content metrics summarize each pass. Throughput is logged per step. Each tensorfield determines which content metrics are available. Training options explains how logging, validation, and checkpoint schedules are configured.

Metric	Interpretation
loss/train	Training loss from the latest batch: the sum of active leaf losses after each leaf’s weight is applied.
loss/train_epoch	Internal training loss average for callbacks; not sent to attached loggers.
loss/validate, loss/test	Aggregate reconstruction loss for each evaluation pass.
.trxns.disputed/validate.loss.state	Reconstruction loss for dispute value states at /trxns/disputed.
.trxns.disputed/validate.loss.content	Datatype-specific content loss across transaction dispute labels.
.trxns.disputed/validate.auc.content	Example Boolean content metric over the retained transactions.
throughput/train	Observations processed per second per rank for the latest batch, using its actual row count and reported at the configured step logging cadence.
throughput/validate, throughput/test, throughput/predict	Observations processed per second per rank for each evaluation or prediction batch, using its actual row count.

Metric names join schema-address components with periods, preserve case, and prefix the address group with a period before appending /stage.metric.component. For example, /trxns/disputed becomes .trxns.disputed/validate.auc.content. The leading period distinguishes address groups from aggregate groups such as loss and throughput. The root uses ., as in ./validate.loss.state. A leaf can contribute both state and content loss. State metrics describe valued, null, padded, masked, and other states; content metrics describe the value being predicted. Leaf metrics are logged before weighting, so their unweighted sum need not match the total loss.

If no reconstruction coordinate is selected anywhere across distributed ranks, the training batch is skipped. Validation/test return zero loss for that batch. Repeated empty-objective warnings suggest a mismatch among masks, source geometry, and split contents; they do not demonstrate learning.

Numeric targets and missing values

For a numeric target such as refund amount, Number content loss and error metrics include selected reconstruction coordinates only when the true value is valued. Null outcomes still teach the state objective, but do not act as numeric zero targets. Validation/test MAE and RMSE accumulate error totals and valued-target counts across the full pass and distributed ranks. For example, one eligible target in a batch has one tenth the weight of ten targets in another batch. RMSE takes its square root after the squared errors have been combined; it is not an average of batch RMSEs. See Number.

Compare candidate models on the same source and eligible coordinates before setting early-stopping thresholds. Loss aggregation remains separate from the target-counted content metrics.

Compare models fairly

Use the same held-out observations, target definitions, preprocessing, and masking policy across comparisons. Keep entity groups and time boundaries consistent with deployment; construct those splits before supplying them to the data module.

Choose an output-specific metric alongside loss/validate. Number errors and categorical accuracy measure different things; their scale and eligible coordinates differ. A state-accuracy improvement alone does not establish better content predictions. Use validation data to choose thresholds and checkpoints, leaving the test split for a final assessment.

Hash validation uses fixed salt zero, matching test and prediction; training still resalts each batch. Keep this encoding policy consistent when comparing validation scores. If using EMA, validation and checkpoint selection during fit use the averaged parameters. Record that choice alongside the architecture and dataset split.

Evaluate the additions in the same order as the examples. Change one modeling choice at a time:

Question	Comparison
Does the history help?	Compare the payment-only model, the history model, and a tuned tabular baseline with spending counts, totals, and recency.
Do richer transaction and profile fields help?	Add payment channel and account tenure to the same history model.
Do shared outcomes help?	Compare the joint dispute/repeat model with a separate model for each task, inspecting each outcome’s metric.
Does deeper account context help?	Add purchase line items, then the separate withdrawal history in the account example, keeping the same transaction-level targets.
Does pretraining help?	Compare fine-tuning with the same supervised architecture trained from scratch.
Is the representation reusable?	Keep the exporting model fixed and compare downstream predictors using existing features, exported vectors, and both together.

Keep the prediction time and future outcome window identical across these comparisons. A payment without a recorded dispute yet may still be awaiting its outcome; define when a negative label is ready before building the splits. See Other domains and outcomes for concrete entity, history, and outcome-window definitions.

Preserving individual events has a cost: more preparation, retained records, and model computation. Choose that representation when its task improvement justifies those costs. A good aggregate-feature model remains a useful option and a necessary comparison.

Boolean fields accept one threshold or a list of thresholds for metrics. Configure them on the /trxns/disputed target:

disputed = rf.Boolean(
    mask=True,
    threshold=[0.5, 0.7, 0.9],
)

This changes threshold-qualified accuracy, precision, recall, and specificity metrics. It does not change the training loss or the probability-only prediction payload. Category topk likewise adds datatype-specific metrics; see the relevant datatype reference.

Measure dependence on an input

An ablation temporarily removes an input and reevaluates the trained model:

amount = rf.where("address") == rf.Address("trxns", "amount")
baseline = model.validate(validation_data)[0]
with model.override(amount, active=False):
    ablated = model.validate(validation_data)[0]
loss_change = ablated["loss/validate"] - baseline["loss/validate"]

Here validation_data is a data module with a held-out validate split and the same model. Removing /trxns/amount removes amount evidence at every transaction position, including the latest. Compare the target’s task metric as well as total loss. Missing-input behavior can be out of distribution for a model that never saw such absence; training with suitable structural dropout can make the comparison more useful. An ablation shows dependence under this experiment, not causation.

override restores schema attributes on exit, not a snapshot of all learned state. See Schema mutation for its boundaries.

The modeling proofs give worked examples of held-out metrics, corrupted-input controls, and deliberately lossy representations. Each scenario includes its model tree and the current evidence for its claim.

Next, extend the model with profile context and a second outcome, and evaluate each addition in the order above.