Online vocabulary

Category, Set, and Cluster learn labels during training, grow storage automatically, and handle unknown values.

Category, Set, and Cluster learn their labels from training data. Supply merchant names and tags directly; there is no separate integer-encoding step. Each field keeps its own vocabulary, saved with the model.

A vocabulary belongs to a field

Extend the transaction history with transaction tags. Earlier payments and the latest payment use the same field definitions inside trxns:

trxns:
  - booked_at: "2026-05-21T10:00:00"
    amount: 32.00
    merchant: Station Cafe
    tags:
      - food
      - cafe
    disputed: false
  - booked_at: "2026-05-27T18:15:00"
    amount: 56.20
    merchant: North Market
    tags:
      - grocery
      - in_store
    disputed: true
  - booked_at: "2026-05-28T14:30:00"
    amount: 84.50
    merchant: North Market
    tags:
      - grocery
      - in_store
    disputed: false

import relflow as rf

model = rf.Model.xs(
    trxns=rf.Branch(
        length=32,
        overflow="tail",
        booked_at=rf.DateParts(
            dateparts=["day_of_month", "day_of_week", "hour_of_day"],
        ),
        amount=rf.Number,
        merchant=rf.Category,
        tags=rf.Set,
        disputed=rf.Boolean(mask=True),
    ),
)

Every transaction position uses /trxns/merchant. That field learns one merchant vocabulary shared by the earlier and latest payments; both North Market values use the same mapping. /trxns/tags owns a separate mapping, shared across its transaction positions. There is no duplicate root merchant field. Preprocessing covers preparing this collection from application records.

Keep labels in one compatible Arrow type family and normalize spelling, case, and label types upstream. A null is a separate value state and consumes no vocabulary slot. Set members are labels; the entire set is not one vocabulary entry.

When labels are learned

Stage	Vocabulary behavior
Training	Admit new labels before the batch’s forward pass, growing storage as needed.
Validation and test	Reuse the existing mapping; unseen labels remain unknown.
Prediction and serving	Reuse the loaded mapping; requests do not teach new labels.
Fine-tuning or resumed training	Continue from the saved mapping, admitting new labels and growing storage as needed.

Two lanes. In the training lane, batch one carries Station Cafe and North Market and admits two labels; batch two carries North Market and Harbor Internet and admits one more label while existing IDs keep their rows. In the prediction lane, a request carrying North Market and Pier Bakery reuses the mapping without admission, and Pier Bakery is marked unavailable while remaining a valued input. — Figure 1: Training batches admit labels before the forward pass and grow storage as needed. Prediction reuses the mapping and treats an unseen label as unavailable.

Learning observes the pristine values retained by the schema, after preprocessing and query/overflow selection but before input masking and p_unavailable perturbations. An always-hidden Category target therefore learns its label vocabulary too. Masking does not prevent vocabulary learning.

Admission does not move existing predictions: a newly admitted label embeds exactly as it did while unknown, and its decoder row starts silent until gradients move it.

Training-stage encoding, including model.encode(..., strata="train"), can update this state even without an optimizer step. Validation or prediction on a newly constructed model starts with empty vocabularies; load a trained artifact to reuse its mappings.

Inspect and reuse the mapping

After training the example model:

merchant_address = rf.Address("trxns", "merchant")
merchant_labels = rf.Category.vocabulary(model, merchant_address)
merchant_counts = rf.Category.counts(model, merchant_address)
tag_labels = rf.Set.vocabulary(model, rf.Address("trxns", "tags"))
merchant_label_count = len(merchant_labels)

Vocabulary accessors return immutable tuples in slot order. Category counts are pristine training exposures for populated labels, excluding the smoothing prior and unavailable values. Repeated epochs count again; these are not unique-record counts. DDP exposure counts synchronize at epoch end and before the evaluation-loop warning checks. New training labels are counted from their first consumed batch. rf.Cluster.vocabulary(model, address) provides the same mapping inspection for Cluster fields.

Preprocessors can inspect established mappings through encoding_context. Training preprocessing runs before vocabulary admission: filtering new labels against that snapshot would prevent them from being learned. Normalize labels there, and reserve vocabulary-based filtering for an explicit application policy. See Preprocessing.

Save/load preserves the mapping and associated model state. Fine-tuning continues learning from it; freezing parameter gradients alone does not freeze vocabulary admission. Use prediction stages for a fixed vocabulary.

Cluster also has an explicit assignment API that can add labels and grow storage on an idle model outside training.

Automatic growth needs no schema update. Resetting a field discards its learned vocabulary and runtime state. Review Schema mutation before explicitly rebuilding a trained field.

Unknown values and prediction

At evaluation, a label outside the learned mapping remains valued; it does not become null. Its content representation depends on the datatype:

Type	Unknown content
Category	Zero categorical content plus the value-state embedding.
Set	Known membership plus unavailable-member counts. A learned unavailable contribution distinguishes unknown from empty sets, without recovering unknown identities.
Cluster	A shared unavailable assignment.

These types default to p_unavailable=0.01, which simulates unavailable content during training. All three perturb inputs only, preserving the original reconstruction answers. This changes input content, not membership in the learned vocabulary or pristine exposure counts. See the datatype pages for their respective loss and coverage definitions: a Set counts memberships, where Category and Cluster count scalar targets.

Predictions name only populated vocabulary entries. Category and Cluster label probabilities are normalized over those entries; Set emits their independent membership probabilities. None can name an unseen label. A high probability does not establish that a request belongs to the known vocabulary. Evaluate unknown-label coverage alongside content accuracy, which excludes unknown targets for Category and Cluster.

Cold input identities

Simulated unavailable inputs teach prediction from the context that remains. They cannot teach the identity-specific effect of an unseen arbitrary ID. A model can legitimately remain confident when other fields are decisive; there is no universal OOV confidence cap. Use entity-disjoint or time-respecting validation to assess the actual cold-start population, not just a stratified row split. Evaluate proper scores such as NLL or Brier score separately for known and unseen input groups.

p_unavailable is an explicit input-augmentation policy, not an estimate of how often a category will be missing at deployment. Increasing it can improve the fallback while weakening use of known identity. Select it on a separate development split; do not learn a vocabulary from validation to conceal OOVs. P061 measures both regimes and includes decisive-context controls. Its synthetic missing-ID population does not establish robustness to arbitrary new-population effects.

Vocabulary admission activates an unused or newly allocated representation. For Category that representation starts where the unknown label stood: a zero content row and a silent decoder row, so existing predictions are unchanged until training moves it. Admission does not train the representation. Keep serving vocabularies frozen, and evaluate a separately continued-training artifact before promotion. Counts measure exposures, not representation competence.

At the start of each Lightning validation, test, or prediction loop, RelFlow warns when any learned label has fewer than 10 training observations. The warning identifies the field and exposure counts without logging label values. It changes neither predictions nor losses. This includes sanity validation during fitting, but not direct model.predict(payments) calls. Ten exposures are a diagnostic threshold, not a guarantee of learning.

P066 measures how admitting input labels changes reconstruction quality across successive training cohorts. For pretraining targets, see coverage-aware evaluation.

Automatic storage growth

Category, Set, and Cluster manage vocabulary storage automatically. Neither size nor capacity is a field option, and there is no configured maximum. When a training batch introduces more labels than fit, storage grows geometrically until it can hold them. Users do not need to estimate the number of labels before training. Cluster’s bounds controls latent clusters, not vocabulary storage.

Growth preserves existing label IDs, learned embedding and decoder rows, exposure counters, partial gradients, and optimizer history: Adam/AdamW moments and step counters, or SGD momentum. New parameter rows receive standard initialization and zeroed optimizer moments or momentum. Cluster preserves its unavailable assignment while growing per-label rows; its configured cluster bounds are unchanged. Checkpoints preserve the mapping and grown tensor sizes, so resumed training can continue from that allocation. Vocabulary storage is runtime state, not schema configuration; growing it does not change the schema.

Labels append in encounter order without evicting or renumbering existing ones. With DDP, each batch’s labels are gathered in rank order, preserving encounter order within each rank. Record order, shuffling, and rank layout can change the assigned IDs, but early rare labels do not exclude later common labels. P057 checks admission and learning across encounter orders.

Automatic growth still consumes memory and increases decoder cost. In particular, Set membership and decoder outputs span the current allocation. Spare rows are not labels: reconstruction objectives and written probabilities use only discovered labels. Actual new labels can change the learning objective; additional storage alone does not add classes.

Use Hash for identifier equality without a persistent vocabulary. Cluster shares latent representations between labels but still keeps growing per-label state.

Workers and distributed training

Data workers encode immutable dictionaries local to each batch. The training process binds those dictionaries to the model’s current vocabulary before forward, admits the new labels, and remaps inputs, targets, and exposure counts. Prefetched batches remain valid across growth because their local IDs are resolved when consumed. Labels are available to their first training batch, including with multiple workers.

Under DDP, every rank gathers that batch’s dictionaries and applies the same ordered admission. When parameters grow, RelFlow replaces the DDP wrapper during the current fit, before the affected batch enters forward, and migrates optimizer state to the grown parameters.

Growth also happens before the affected batch during gradient accumulation (accumulate_grad_batches > 1). Before replacing the DDP wrapper, RelFlow synchronizes which parameters have gradients and their partial gradients across ranks. Resizing preserves these gradients, including contributions from parameters used only in earlier microbatches, and accumulation continues.

Automatic growth supports single-device training and standard eager Lightning DDP with automatic optimization and static_graph=False; DDP growth during accumulation requires dense gradients. FSDP, DeepSpeed, other sharded backends, compiled DDP, and DDP configurations with native mixed precision, custom communication hooks, or delayed all-reduce are unsupported for growth.

Checkpoints record the model’s admitted vocabulary and allocated rows. Prefetching alone does not admit labels or record training exposures. Integer slots are local to a trained model; preserve its artifact and inspect its mapping when reusing it.

The Vocabulary and OOV proofs exercise unknown inputs and targets, vocabulary growth, set membership, and checkpoint/continued-training behavior. They distinguish learned fallback from information that a frozen evaluation vocabulary cannot represent.