Serving

rf.Deployment serves a checkpoint or an in-memory model through FastAPI. Install the optional dependency:

pip install 'relflow[serving]'

Create an ASGI application in your application module:

import relflow as rf

app = rf.Deployment(checkpoint="order.ckpt").app()

Or start Uvicorn through relflow with rf.Deployment(checkpoint="order.ckpt").serve(). The default port is 8000. The model is loaded at startup and placed on the selected accelerator.

Requests

POST /predict accepts a JSON object, an array of JSON objects, or an Arrow IPC stream. This YAML illustrates one JSON request for an order model; send its JSON representation over HTTP:

request_id: order-42
amount: 86.50
items:
  - sku: coat-17
    quantity: 1
  - sku: scarf-03
    quantity: 2

curl http://localhost:8000/predict \
  -H 'Content-Type: application/json' \
  --data-binary @request.json

request.json is supplied by your application. A single object returns a single object; an array returns an array in request order. For Arrow, send an IPC stream with Content-Type: application/vnd.apache.arrow.stream; the response is also an IPC stream. GET /health supplies a health endpoint, and /docs exposes the API reference.

Default responses contain predictions plus a model provenance field with version and checkpoint. They omit the interactive prediction table’s inputs column. A postprocessor controls application response fields; provenance is appended afterward, so model is a reserved name.

Validate application inputs

Optional Pydantic signatures validate JSON requests before prediction and responses after postprocessing. For the request above:

from pydantic import BaseModel

class Item(BaseModel):
    sku: str
    quantity: float

class OrderRequest(BaseModel):
    request_id: str
    amount: float
    items: list[Item]

app = (
    rf.Deployment(checkpoint="order.ckpt")
    .forge(request=OrderRequest)
    .app()
)

These signatures apply to JSON. Arrow requests are validated through the preprocessing and model-binding contracts.

Reuse processing

Attach the application’s existing processors to preserve its input and output contracts:

app = (
    rf.Deployment(checkpoint="order.ckpt", retain=("request_id",))
    .preprocess(prepare)
    .postprocess(reshape)
    .app()
)

prepare and reshape are decorated eager Polars processors supplied by the application. Retention exposes processed source columns to the postprocessor. The final result must contain one row per valid request in request order.

Concurrent JSON requests can share a model microbatch. max_batch_size bounds its size; batch_timeout sets the optional collection wait in seconds. accelerator selects auto, cpu, cuda, or mps. An in-memory model can be supplied with model=model instead of a checkpoint.

Use one process for an in-memory model or a deployment configured with signatures, processors, or queued mutations. For Deployment(workers=2).serve(), the configuration must be checkpoint-based and cannot carry those in-process objects or a tuple retention projection.