Streaming & Scaling¶

The .cv.pipe(...) expression is an ordinary elementwise Polars plugin, so the parallelism comes from the Polars engine, not from inside the plugin. This has one important consequence for large workloads.

Why eager runs single-threaded¶

On a plain DataFrame.with_columns(...) / .select(...) call (the eager, in-memory engine), the whole column is processed on a single thread:

# Eager: single-threaded over the whole column
result = df.with_columns(
    processed=pl.col("image").cv.pipe(pipe).sink("numpy")
)

This is fine for small or interactive use. For anything larger, polars-cv emits a one-time warning pointing you here (see Silencing the warning).

Use the streaming engine for scale¶

Run through the lazy streaming engine instead. Polars splits the column into morsels and processes them across its worker pool, and can spill intermediate state to disk when memory is tight:

result = (
    df.lazy()
    .with_columns(processed=pl.col("image").cv.pipe(pipe).sink("blob"))
    .collect(engine="streaming")
)

This is the recommended path for anything beyond small/interactive use. The plugin's per-morsel graph is compiled once and cached process-wide, so per-morsel overhead is just a hash lookup — you get Polars' multi-threaded, larger-than-memory execution with no extra plumbing.

Note

The detection-metrics APIs already collect with engine="streaming" internally, so you don't need to opt in when using them.

Silencing the warning¶

A large batch run under the eager engine prints a one-time notice. Control it with environment variables:

Variable	Effect
`POLARS_CV_SILENCE_ENGINE_WARNING=1`	Suppress the warning entirely
`POLARS_CV_ENGINE_WARN_ROWS=<n>`	Row-count threshold above which the warning fires

Cheaper decoding for curation passes¶

When scaling over large image sets, you often only need a cheap signal (a perceptual hash, a mean, a quality score) to filter before doing full-resolution work. Decode a downscaled thumbnail first with thumbnail(max_size) (JPEG IDCT-scaled decode), compute the signal, filter, then full-decode only the survivors in a second pass.