Set

Use Set for unordered membership in a learned vocabulary. Use Category for one label and Branch when each item has attributes or position.

tags:
  - vip
  - international
  - vip

import relflow as rf

tags = rf.Set(threshold=0.75)

Input and options

An Arrow list holds the labels; a scalar label is treated as a one-member set. Labels may be Boolean, integer, floating-point, string, or binary, with one compatible Arrow type family per field. Duplicate members collapse, and null members are ignored. An empty list is valued; a null list has state null.

Option	Default	Meaning
p_unavailable	0.01	Training probability of hiding each known positive input member. Reconstruction answers remain intact.
threshold	None	Prediction filter in [0, 1]; retain labels whose probability meets it. None returns every populated label.

See shared leaf options for target and embedding configuration.

Vocabulary and learning

Vocabulary storage grows automatically, with no size option to configure. The shared Online Vocabulary contract covers admission order, growth and state preservation, supported training strategies, and checkpoint reuse.

The training process binds batch-local member dictionaries before forward, growing embedding rows, membership width, counters, and decoder rows as needed. Validation, test, and prediction reuse the frozen mapping. For an application-supplied trained model:

labels = rf.Set.vocabulary(model, rf.Address("tags"))

This tuple snapshot lists populated labels in index order. It also accepts a preprocessor’s encoding_context; consult established vocabularies for inference preprocessing, not to decide which training labels may be learned.

Content contains membership bits spanning the current allocation and a count of distinct unavailable members. Unknown identities have no positive bit; duplicates and member order do not affect either representation. The encoder averages known member embeddings and adds a learned contribution when any member is unavailable, plus the value-state embedding. That contribution starts at zero; real or simulated OOV training teaches it. It distinguishes unavailable membership from genuinely empty input without inventing unknown identities.

Reconstruction combines unweighted state classification with binary cross entropy over populated labels in valued target sets. Unallocated slots are not negative examples of future labels. Complete set annotations are assumed: known positives and known negatives remain supervised even in a partial-OOV or all-OOV set. Unknown members cannot supply identity-specific positives. An empty set supplies known negatives; null is a separate state.

Epoch metrics use summed counts, including under DDP, and a fixed 0.5 cutoff:

Metric	Meaning
targets.known, targets.unavailable	Counts of distinct positive memberships per selected valued set, summed across observations.
coverage.content	Known positive memberships / all positive memberships.
sets.valued, sets.complete, coverage.set	Selected valued sets, sets with no unknown members, and their complete fraction. Empty sets are complete.
loss.content, accuracy.content	BCE and bit accuracy over populated membership slots only.
accuracy.set	Exact-set accuracy; any unknown target member makes the complete answer unrecoverable.
precision.content, recall.content, f1.content	Micro membership scores on known labels.
recall.all, f1.all	Micro scores also counting unknown target members as missed positives.

An empty denominator yields NaN, not perfect reconstruction. Bit accuracy alone can still be misleading for sparse sets; examine F1, exact-set accuracy, and coverage together. Output threshold does not change these metrics.

P063 tests probability preservation and coverage; P064 tests learning the empty/unknown distinction.

The unavailable-input contribution adds a parameter to Set artifacts. Older checkpoints without that parameter cannot be loaded unchanged by the strict checkpoint loader; there is no automatic artifact migration in this change.

Prediction

content is a list of value / probability structs, with string labels and independent membership probabilities. Only populated vocabulary entries appear. The configured threshold filters this list; it changes neither training metrics nor the decoder computation over the current allocation.

See the prediction envelope for shared state probabilities, inferred, and repeated shapes.