Changelog¶
All notable changes to polars-cv are documented in this file.
The format is based on Keep a Changelog, and this project aims to follow Semantic Versioning.
Unreleased¶
0.39.0 — 2026-10-18¶
Shapes made explicit, contours made drawable, and CSE that settles. Binary
ops no longer broadcast: operands of different shapes are refused, and the
new squeeze/expand_dims change a rank. channel_select keeps the
channel axis by default, and apply_mask reads a mask on its own dtype's
scale. Contours gain set operations, shaping, stroking and open lines in
pipelines; rasterize can supersample; distance_transform,
coordinate_grid, apply_lut and disk footprints are new. CSE now computes
each distinct value once, fuses linear runs and repeats until the graph
settles. The sdist ships its license files where PyPI looks for them.
Install 0.39.0 rather than 0.38.0 from PyPI: the 0.38.0 wheels there were
built from this release's code, not 0.38.0's, and its sdist was refused (see
Fixed). Upgrading from 0.38: see the
migration guide.
Breaking changes¶
-
channel_select(i)keeps the channel axis:[H, W, 1], not[H, W]. That is the shapegrayscaleandrasterizegive, so a selected channel now combines with them as it is (the binary ops broadcast nothing, so a[H, W]channel against a[H, W, 1]mask was refused). The newkeepdims=Falsegives[H, W]; it is a literal, since it fixes the rank. A[H, W]input is returned as it is either way. Code that relied on the 2-D result passeskeepdims=False(see the migration guide). -
Binary ops no longer broadcast: both operands must have the same shape (breaking).
add,subtract,multiply,divide,blend,maximum,minimumand the bitwise ops used NumPy broadcasting, aligned from the last axis.[H, W, 1]against[H, W]therefore became[H, W, W]on a square canvas (4 GB of f32 at 1024 px) and an error on any other, and a[H, W, 1]operand stretched silently over[H, W, 3]. Operands of different shapes are now refused (OpShape::Elementwise,elementwise_dims): at build where the sizes are known, per row where they are not. Change a rank with the newsqueeze/expand_dims.apply_maskkeeps spreading a single-channel mask over the channels, as its contract, and now does so for a[H, W, 1]mask itself rather than through broadcasting. The broadcast half of the binary kernel (per-operand repeat blocks,broadcast_shapes,ViewBuffer::leading_axes) is deleted with it. Anextract_shape()vector added to an image is now refused at build rather than at execution.
Added¶
-
squeeze(axis)andexpand_dims(axis): remove an axis of size 1, or insert one, as zero-copy views (ViewOp::Squeeze/ExpandDims, shapesOpShape::RemoveAxis/InsertAxis). A negative axis counts from the end, as NumPy's. Unlikereshapethey need no sizes, so they change the rank of an image whose height and width are known only per row;squeezerefuses an axis of a known size other than 1, and either refuses an axis outside the rank, at build. Planned over an input of unknown rank, a negative axis claims no sizes (every position after it moves with the rank). The array sink's shape-mismatch error had long told users to call these two; now they exist. -
A second logo: a halftone polar bear (
docs/assets/logo/halftone_logo.py). The bear's face is a table of polar curvesr(θ)evaluated by a polars query (int_ranges, expressions,group_by), and it is printed as a halftone by one polars-cv graph: a signed distance field sets each cell's dot radius (sampled per cell by aresizedown and anearestresize up), andcoordinate_gridminus itsfloorplaces the dots. Writes a lockup with the existing wordmark and a square mark;tests/test_halftone_logo.pyholds both to the pixels that make them a bear. -
"Drawing with polars-cv" guide, and the logo rebuilt on the drawing ops (CR-101). The guide (
docs/user-guide/drawing.md) covers shapes as data, filling and stroking, the shaping and set operations, fields over pixels, and colouring an index map with oneapply_lutpalette; every example runs in the test suite.docs/assets/logo/make_logo.pyis now one graph: terraces fromdistance_transform, contour lines as strokedoffsets, the wordmark as stroked centrelines and a filledv, stacked bymaximumand coloured byapply_lut. It replaces a per-pixel DataFrame for the distance field and per-channel weighted sums of mask layers. -
coordinate_grid(normalized=False)(CR-100): an f32[H, W, 2]buffer of each pixel's centre(x + 0.5, y + 0.5)(divided byW/Hwhennormalized, per-row capable), for fields over positions. Reads only the input's height and width, so it takes any dtype and channel count, and makes no copy of a view. Its shape isOpShape::Canvas, which now carries a channel count (a rasterized mask's 1, a grid's 2). Held tonp.meshgrid, in the frame of the buffer it follows (a crop before it moves the origin). -
Contour set operations (CR-99):
union,intersection,differenceandsymmetric_differencecombine two contour sets, each read as the union of its regions, into disjoint regions (always a set). Each is one lazy pipeline op (the other side another contour-domain expression), and its.contouraccessor (the other side a contour column) is generated from that op's catalogue entry and runs it; a line on either side is refused. Held to the mask identity — rasterizinga op bpaints the pixelwise|,&,& ~or^of the two sides' masks, exactly on integer shapes — and to inclusion-exclusion andiouon float shapes. -
Contour shaping and line rendering (CR-98). Five contour ops, each a pipeline op and a
.contouraccessor with one definition:offset(distance, join="round")grows (negative: shrinks) a set's regions, read as one union, so the result is always a set;union_all()merges them into disjoint regions;smooth(iterations=1)is Chaikin corner cutting;densify(max_segment_length)splits each edge into equal parts no longer than the bound; andstroke(width=1.0, cap="round", join="round")is the region a set's paths cover when drawn — an open line capped at its ends, a region's rings (and a line returning to its start) as closed paths — as disjoint regions.rasterizestays fill-only and still refuses a line, now namingstroke(): lines are drawn asstroke(...).rasterize(...). (Stroking was first added asrasterize(line_width=, cap=, join=), three parameters a region-only set silently ignored; they never shipped.) New enumsLineJoin(round,miter,bevel; a miter limit of 4) andLineCap(round,square,butt). Round arcs are polygons within 1/64 pixel up to a radius of about 32, aboutr / 2000beyond. Held to distance-band oracles (a round offset or stroke is the set of points within the distance, checked on the mask's own samples), closed-form areas per join and cap, a pure-Python Chaikin, an exact densify construction, andrasterize(union_all(s)) == rasterize(s). -
Open contours in pipelines;
boundary()andclose()(CR-97). The contour domain carries lines (is_closed = false) as well as regions:source("contour")reads open rows instead of refusing them, and each contour op behaves as its.contouraccessor does —perimetermeasures a line's length,bounding_box,translate,scale(not about a centroid) andsimplifykeep it a line,convex_hullmakes a region, andarea,centroid,largestandrasterizerefuse it with the accessors' message.boundary()gives a region's rings as lines (always a set);close()joins a line's ends into a region. Both are also.contouraccessors. -
rasterize(..., supersample=n)(CR-96): anti-aliased masks. Each pixel samples itsn x nsub-pixel centres and is painted betweenbackgroundandfill_valueby the fraction inside (rounded half away from zero), fornfrom 1 to 16 (per-row capable). The default of 1 is the pixel-centre rule, byte for byte. Not theanti_alias=removed earlier, which was accepted and ignored: held to matplotlib's point-in-polygon on the same samples, and the covered area toarea()(exactly on integer edges, within a band that shrinks as1 / notherwise). -
apply_lut(table, out_dtype="u8")(CR-95): each u8/u16 value read as a row index intotable. Rows of one entry map every element in place on any channel count (cv2.LUT); rows ofKentries take a single-channel image toKchannels, a palette. Entries are stored asout_dtype(any of the ten) and must fit it, which is checked when the pipeline is built, as are ragged rows and float input. A value with no row is an error for its row (soon_error("null")nulls just that row); clamp withclamp(0, n - 1, out_dtype=...)first to send larger values to the last row. Each entry accepts an expression. -
footprint="disk"onerode,dilate,morphology_gradient,morphology_openandmorphology_close(CR-94): the pixels within Euclidean distanceksize // 2of the centre, edges replicated, for round rather than square morphology on any dtype. The default stays"square", byte for byte. With a disk,iterationsapplies it that many times (two disks do not compose into one). Held to scipy'sgrey_erosion/grey_dilationwith a disk footprint and, on masks, to the distance transform.footprintaccepts an expression. -
distance_transform(signed=False)(CR-93): the exact Euclidean distance transform of a single-channel mask, as an f32 image of the same shape. Unsigned, each non-zero pixel holds its distance to the nearest zero pixel, centre to centre — whatcv2.distanceTransform(DIST_L2, DIST_MASK_PRECISE)andscipy.ndimage.distance_transform_edtcompute: exactly scipy's distances, correctly rounded to f32 (OpenCV's can be one ulp off).signed=Truegives the distance to the mask's pixel-edge boundary (the oneextract_contourstraces), negative inside, matching.point.signed_distance_to_contour. A class with no pixel at all is infinitely far (±inf), where scipy and OpenCV return arbitrary finite values.signedaccepts an expression. -
A logo, drawn by polars-cv itself (
polars-cv/docs/assets/logo/). A polar bear is sketched from overlapping ellipses, fused withblur+thresholdand traced back to a low-poly outline withextract_contours+simplify;.point.signed_distance_to_contourover a pixel-per-row DataFrame then gives a distance field that thresholds into topographic terraces and contour lines. The wordmark is built from contour primitives.make_logo.pyregeneratespolars-cv-logo.png(--stagesalso writes the bear's construction stages). -
Each output of a multi-output sink takes its own sink parameters. A dict
sink()value may be(format, {params}):merged.sink({"x": ("array", {"shape": [224, 224, 3]}), "thumb": ("jpeg", {"quality": 85})}). Keyword arguments tosink()still apply to every output, so before this anarrayshape and a JPEG quality could not share one graph (each was refused by the other's sink), and an image that fed both had to be decoded twice. A parameter given both per output and shared is refused, as is an output that is neither a format nor(format, {params}).
Changed¶
channel_mergejoins channel axes. Each operand may be[H, W](one channel) or[H, W, C](itsC), joined in order, sophoto.channel_merge(alpha)is an RGBA image; it used to take only single-channel operands, so adding an alpha channel meant splitting the photo into its channels first. Stacking single channels is unchanged.- CSE computes each distinct value once, and fuses linear runs. It used
to share only a leading op run common to every root node on a column, so a
bare decode, two branches with different first ops, or one unrelated
pipeline on the same column left each decode separate, and nothing below a
root was shared. Every op is now numbered by what it computes (its input
and its wire form, operands included), so equal work is shared wherever it
was written, duplicates merge all the way down, and a node that only passes
its input on vanishes. Each linear run of ops that nothing else reads then
runs as one node, so a chain of lazy method calls is one node, and the
spatial-window pushdown can move a crop across what were node boundaries.
The logical passes then repeat until the graph stops changing, so two
pipelines that differ only by a no-op the identity pass removes, or by an
order the pushdown normalises, are shared too.
Outputs are byte-identical;
opt_flags=Falseshows the graph as written. -
README: a branch-and-rejoin example (a cup and saucer cut out to a transparent PNG, with the optimised graph it runs, from
scripts/make_readme_cutout.py); the detection-metrics snippet and the "When not to use polars-cv" section are gone. The halftone logo has a transparent background, a tinted face, inked eyes and nose, and outlined letters, so it reads on light pages too. -
footprint="disk"erode/dilate/morphology are linear in the radius (CR-105). Each half-width's spans were a separate full-size image, built from scratch: O(r²) work per pixel and r + 1 images held at once. The spans now grow a step at a time and the column fold runs row offset outermost, regrouped (the min/max fold is associative) without changing its order, so results are bit-identical, signed zeros and NaNs included. A 31-px disk is 2.2–2.4x faster (release, 256² to 1024²). -
apply_lutreads less and copies less per row. A table of single entries converts and holds only its rows, indexed directly (it padded a table to all 65 536 u16 values on every row, 512 KiB forout_dtype="f64"), and a palette writes its[H, W, K]output in one pass instead of buildingKplanes and merging them (one view-sized allocation instead ofK + 1). Its row errors are prefixedApplyLut:, asHistogram:'s are: the two fallible buffer steps now share one executor path. -
target/debugno longer grows without bound (CR-104, development only). The disk filled twice in one session; measured, three causes were avoidable. The workspace's own crates keep debuginfo inline (split-debuginfo = "off"per package, the polars stack keeping its sidecars): each incremental rebuild had written its.dwofiles under new names and cargo never deletes the old (~79,000 files, ~12 GB). A rebuild after a plugin edit is ~10 s, against ~11.5 s before (and ~21 s with debuginfo inline for the whole stack, which the profile's comment records). view-buffer's 31 integration-test executables (~4 GB, each relinked on every change) are one,tests/it/. The SessionStart hook cleanstarget/whenrustc -vVchanges, since a new stable orphans every artifact (~5 GB here). polars-cv'spolarsdev-dependency is gone, so its Rust tests run against the polars features the plugin ships.test_build_efficiency.pyguards each. -
Documentation restructured. The README and the docs site open with the halftone logo (also the site's logo and favicon). A new Parameters & Errors page is the one place for per-row parameters, null parameter values and error policies, which had been split between the image-operations and sources pages; the image-operations page's hand-kept table of which parameters accept expressions is gone (it had missed every parameter added since 0.36), and the method signatures in the API reference say it instead. The migration guides have their own section with an overview page at
user-guide/migration/(the 0.28 → 0.29 guide, which had that address, is nowmigration-0.29). The domains page covers open lines and the contour-to-contour ops; the quickstart no longer says an eager call runs on one thread; installation no longer repeats the cloud-credentials guide. view-buffer's README described an unpublished crate as on crates.io at version 0.3, with apythonfeature that does not exist and an example that did not compile; it now matches the crate. Leftover planning text inISSUES.md(the finished drawing-primitives plan) is gone; its owner decisions moved to the decisions list. -
The README is shorter and leads with one decode per image. Its first example now reads each file once and sinks both outputs from one graph; the figure's top row is one multi-output graph, so its "one decode" label is true (it was five separate plugin calls, each decoding the image). Thread tuning, the 50k-image results and the test inventory moved out to the linked guides and
benchmarks/.
Fixed¶
-
The sdist PyPI refused. The 0.38.0 publish uploaded its wheels and then had its sdist rejected ("License-File LICENSE-APACHE does not exist in distribution"): maturin auto-discovered
LICENSE-MIT/LICENSE-APACHE, named them at the top ofPKG-INFO, but packed them underpolars-cv/.pyproject.tomlnow declares them (license-files, with the PEP 639license = "MIT OR Apache-2.0"expression), which puts them at the top; the build floor rises tomaturin>=1.9.3, the first that does. Thepublish.ymlsdist job runsscripts/check_sdist.pybefore anything is uploaded, so a release can no longer go up half-published this way (tests/test_sdist_metadata.py,tests/test_sdist_build.py). -
apply_maskrefused a[H, W, 1]mask over a[H, W]image (a rasterized mask over a heatmap, asexamples/05does, failed since the binary ops stopped broadcasting). Such a mask is read as the[H, W]one it holds, as it already was over a[H, W, C]image. apply_maskread the mask on the image's scale. A u8 mask (arasterizeorthresholdresult, 0 or 255) multiplied a float image by 255 and darkened a u16 one 257-fold, and a float mask truncated to 0 or 1 on a u8 image. The mask's own dtype now says its scale — a u8 mask weighsm / 255, a float maskmin[0, 1]— whatever the image's dtype, and the result keeps the image's dtype. A mask of any other dtype is refused (cast or scale it first). A u8 image under a u8 mask is unchanged.- Aliases. An alias given to two different nodes is now refused (it resolved silently to whichever was found last), and a node may carry several aliases, each an output (all but one used to be lost: "Alias 'q' not found in graph").
- Only the nodes an output reads are serialized.
merge_pipe's own node and branches no alias selects used to ride along on the wire and into the compiled-graph cache key (Rust never ran them);show_graphhid them by deleting childless nodes. channel_merge,apply_maskandmerge_pipeapplied the left operand's ops twice. They cloned its pipeline into the composed node, which receives that operand's output, so a node fed by another node ran its ops again on its own output (x.pipe(Pipeline().add_constant(50)) .channel_merge(y)added 100), and one on a column decoded it a second time. They now start from the operand's output, as the binary ops did (LazyPipelineExpr._receiving).-
show_graph()hid a root node's ops and named nodesNone. A source node now lists the ops after its decode (where CSE puts a shared prefix), every node leads with the wire id its operands are named by (maximum( other=n5)), ops render as calls, and the DOT text no longer carries each node's Python repr. -
An op-backed
.contouraccessor runs the pipeline step it is (CR-106)..contour.area,offset,smooth,scaleand the twelve other accessors that are pipeline ops each had their own body calling the op's engine function, beside the step the pipeline runs; every divergence the reviews of this release found between the two surfaces came from that second dispatch. Each now resolves the op as the pipeline resolves a step (once, or per row when an argument is an expression, with the op's check), executes it with the executor's own dispatch, and only presents the result its way (a value per member, or a set). So a refused row reads the same on both surfaces, named once by the step:.contour.area()of a line saidan open contour ... (row 0)where the pipeline saidArea: an open contour ..., andboundary,close,smooth,densifyandscalenamed themselves in lower case on both (smooth: 40 passes ..., nowSmooth: 40 passes ...). An empty contour'sbounding_boxis NaN on both: the pipeline gave a 0x0 box at the origin, the accessor a null. Sixteen accessor bodies are deleted;tests/parity/invariance/test_parity_accessors.pynow holds the two surfaces to the same refusal text as well as the same rows and nulls. -
stroke()of a ring narrower than the stroke is solid. Stroking a region's ring wider than the region left a hole where the band should cover it: a single pixel stroked withwidth=4came back as a 5x5 square with a diamond cut from its middle, so the drawn outline had a gap. Every point of the region is within the half-width of its ring, buti_overlay(4.5) turns the ring's inward side inside out once it passes the middle. The band round a closed path is now also taken as its region grown by the half-width less that region shrunk by it (offset, exact), joined toi_overlay's stroke, which still covers what has no area (a doubled segment). Found by a new law of the parity suite: on contours traced from random masks,stroke(width=2k)paints exactly the pixels withinkof the rings, by L∞ for a miter join and Euclidean for a round one. The same laws now holdoffsetto Minkowski growth and shrinking, the four set operations to|,&,& ~and^of their masks,close(boundary())to the identity,largest,union_all,densify,smooth(0)andsupersampleto the mask they read, and an open line's stroke to its distance band; and an invariance property holds each op-backed.contouraccessor to its pipeline op (rows, refusals, nulls). -
A refused value names its op once, as the user spells it. The planner and the geometry accessors' build check prefixed every
check()message with the op's wire name, which most messages already began with their own:rasterize: rasterize: supersample ...,contour_densify: densify: ...,blur: blur: sigma .... The prefix is gone, and the checks that did not name their op (normalize,histogram's range,perceptual_hash,largest,extract_contours,convolve2d,assert_shape) now do. One convention throughout: a build-time refusal begins with the op's Python name (histogram: range ...), a row error with its engine step's name (Histogram: range ...,Offset:,Union:); the set operations had named their row errors in lower case.assert_shape()'s empty declaration is refused by the op's own check, no longer by a copy in its Python sugar. -
A
histogramrange no row can use is refused when written. A non-finite or inverted literalrange=passed the op's check and failed every row; the check now refuses it at build, through the one range rule the executor also applies per row (named once, notHistogram: histogram:). -
A contour column of the wrong layout fails the query under every error policy. A
source("contour")pipeline, and the set operations (.contour.union(other), ...) built on one, read a non-contour column per row, soon_error("null")turned it into a column of nulls; the column's layout is now checked once per call and refused like any other wrong input type. A set operation'sother=that is not an expression is aTypeErrornaming it, and its docstring states what it returns. -
rasterize(shape=)onto another node's canvas checks its row. The shape-reference form skipped the op's check, so a per-rowsupersampleof 0 panicked in the engine and one past 16 ran; both are now refused with the row, like the other forms..contour.largest(k=)per row likewise applies the op's own rule rather than a second copy of it. -
Contour results publish the canonical contour struct. A
.contourfunction that returns contours declared "the input's own layout" and cast its result to it. On a column polars inferred (holes: List(Null), every hole empty) the set-level functions (largest,boundary,offset,union_all, the set operations) panicked, and on a struct withoutis_closedorholesthe cast dropped what that layout cannot hold:boundary()'s lines read back as regions, a hole a union made vanished. Every contour result is now{exterior, holes, is_closed}asgeom_schemadeclares it; a declaration that differs is an error, never a cast, andcontour_accessor!no longer hands the element type the input dtype to echo. The three identical set-level output types are one. -
smoothanddensifyrefuse a row past 2^22 points before making it. A pass doubles the points and a shortmax_segment_lengthsplits every edge, sosmooth(iterations=40)ordensify(1e-9)— literal or per row — aborted the process on the allocation. Both now fail the row by name. The budget is the row's whole set, not each contour's: thousands of traced contours each within it could together still abort. -
stroke()of a ring with no points no longer panics. An empty exterior or hole underflowed insidei_overlay; a ring of fewer than two points has no edge and strokes to nothing (a one-point line is still a dot, its caps). Every shaping op is now swept over degenerate rings on both surfaces. -
An op-backed
.contouraccessor applies its op's check as it is written..contour.offset(inf),densify(0),stroke(width=0)andlargest(k=0)were accepted at build and failed only at collect, where the pipeline forms refuse at build; the accessor's build-time check now runs the op's owncheck(). -
The round-arc tolerance is stated as it holds. Arcs are within 1/64 pixel up to a radius of about 32 pixels; past it
i_overlay's finest arc step (0.01π) bounds them byr (1 − cos 0.01π), aboutr / 2000(transforms::arc_tolerance). The docs promised 1/64 at every radius; 0.12 pixel was measured atoffset(1000). -
boundary,closeandscale(origin="centroid")refuse a contour the same way on both surfaces, through one engine function each (transforms::boundary_outline,close_outline,scale_outline); their rules no longer live in the plugin's graph encoder, with a second copy of the scale message in the accessor. -
A line that returns to its first point is stroked as a loop.
geostrokes every line string as an open path, sostroke()capped a loop's start instead of turning it like its other corners: abuttcap left the corner missing, aroundone rounded it. Found rebuilding the logo, where each bowl of the wordmark showed a nick; a loop is now stroked as a closed path (i_overlay, undergeo, which view-buffer now names directly). -
A non-finite float literal crosses the wire (CR-103). JSON has no NaN or infinity, so
simplify(float("nan"))— any op or geometry accessor — failed as the decoder's "expected value at line 1 column 41". Every float field now reads and writes"nan","inf"and"-inf"(asboundfields already spelled their infinities), and_types._literalencodes them, so a literal meets the op's own check exactly as the same value passed per row does:apply_lut([[nan]], out_dtype="f32")maps to NaN, andoffset(inf)is refused byoffset's rule. (A first fix refused every non-finite literal in Python, which made a NaN table entry an error whilepl.lit(nan)worked; it never shipped.) Aboundstill refuses NaN. A sweep of every float field of every pipeline op at NaN and ±inf holds each to "refused at build naming the op, or runs without a panic"; it foundclamppanicking on a NaN or crossed bound (f32::clamp), which a per-row pair already reached.clamp's check now refuses both, at build for literals and with the row for expressions. -
Two parity references claimed inputs they do not model, failing the deep lane:
apply_lut's strategy read a channel axis a rank-2 buffer (afterchannel_select) lacks, andcanny's compared a 5-channel image againstcv2.Cannyon its first three channels, while the engine (as documented) takes the strongest gradient over every colour channel. Thecannyreference now reads colour channels by the engine's one alpha rule and is compared only wherecv2.Cannyapplies (one or three colour channels). -
The parity suite's per-row parameters now encode nested lists level by level, so an
apply_luttable varying by row passes one expression per entry; it turned a whole table row (whose length is structural) into one expression, which failed the deep lane. -
How many threads a call uses is measured, not raced. Before a call's first row range (about 300 image rows) finished, the plugin read a row's cost off how fast the host woke a helper thread compared with how fast that range ran. On a host that wakes threads in a few µs (the macOS CI runner) a call of expensive rows stayed on two threads for its whole first range. A call now times its first rows on its own thread (one row, then 1, 2, 4, ... until about 50 µs of rows; a first row only counts by itself when it takes over 1 ms, since it pays the call's one-off costs) and invites one helper per 200 µs of the work that measurement says is left. Expensive rows reach the whole pool after a row or two, and a call of cheap rows that finishes within its probe wakes no thread at all. Results are unchanged. The cost: a call's first row now runs alone, so a call of only a few expensive rows is slower — 8 rows of ~18 ms on 4 threads took ~58 ms against ~39 ms. From 16 rows up, and in the regression suite's pipeline, sink, split and geometry cases at 4 and 32 threads, the two are within noise.
0.38.0 — 2026-10-15¶
Standard pHash, a pool switch, and a README with benchmarks.
perceptual_hash() now computes standard pHash, so its hashes change and
stored ones must be recomputed; blur(sigma=0) is no blur instead of an
all-zero image, and a negative sigma is refused. On many-core hosts a call of
cheap rows no longer wakes the whole plugin pool, and
polars_cv.set_row_parallelism(False) turns the plugin's own row splitting
off. The README is rewritten, with results from a new cross-library benchmark
suite in benchmarks/, and the license texts now ship with the package.
Upgrading from 0.37: see the
migration guide.
Security¶
rustls0.23.45 → 0.23.46, for RUSTSEC-2026-0346 to -0349 (TLS secrets inDebugoutput, ECH, Brotli certificate decompression, a ServerHello key-exchange downgrade). It is the TLS stack of remotefile_pathreads (http(s)://,s3://,gs://,az://).
Breaking changes¶
perceptual_hash()(the defaultalgorithm="perceptual") computes standard pHash, so its hashes change. It thresholded the 8x8 lowest frequencies of a 16x16-input DCT at their mean, which the DC term dominates, so most bits agreed between unrelated images. Distinct COCO val2017 images averaged 18 differing bits instead of ~32, and 236 of the 5,000 (all distinct) shared a hash with another, in 64 groups. It now does whatimagehash.phashdoes: BT.601 grayscale, a Lanczos resize to 32x32, a 2-D DCT-II and the 8x8 lowest frequencies against their median, with the bits row-major, most significant first. The grayscale and the resize are the engine's own ops, which round differently from Pillow's, so a 64-bit hash is typically within 0-2 bits of imagehash's rather than identical. A coefficient that is mathematically 0 is compared as 0, not as rounding noise. Hashes stored from an earlier version must be recomputed.average,differenceandblockhashare unchanged.
Fixed¶
-
Many-core hosts no longer spin the plugin's pool on cheap rows. A plugin call invited up to the whole pool to help before every row range, however little work its rows were; on a 72-thread host the woken threads spent their time searching for work, and a contour table took 19.5 s and 1,356 s of CPU against 0.4 s and 4.9 s at 16 threads (worst on hyperthread siblings). A call now invites one helper per 200 µs of work it estimates is left: from its finished ranges' measured cost, or, before one has finished, from how long the first has been running, and helpers invite too. So cheap rows stay on a thread or two, and expensive rows still reach the whole pool within a few wake-ups.
-
blur(sigma=0)returned an all-zero image. The kernel wasexp(-0/0), i.e. NaN, which casts to 0.sigma=0is now no blur, as in PIL and SciPy. A negative, NaN or infinite sigma is refused, when the pipeline is built for a literal and when the row runs for a column value. A negative sigma used to be silently treated as no blur.
Added¶
polars_cv.set_row_parallelism(enabled)/row_parallelism()turn the plugin's own row parallelism off (or back on) for the process. Off, every plugin call — pipelines and the.contour/.point/.bboxaccessors alike — runs its rows on the thread polars called it on, so the streaming engine's morsels are the only parallelism and the plugin's pool is never woken. A process setting, not asink()argument: it does not change what a graph computes, so it stays out of the wire form, CSE and the compiled-graph cache.
Documentation¶
- The README is rewritten for new users: a figure generated by polars-cv
itself (
scripts/make_readme_figure.py, from a CC0 image), a 30-second example on files with a per-row parameter, scaling and threading guidance checked against the code, the operations list in a collapsible block, when not to use polars-cv, and a project-status section with the test, CI, release and versioning facts. Its images use absolute URLs, so PyPI renders it as GitHub does. - A cross-library benchmark suite in
benchmarks/(its own uv project, run against the published wheel): PIL, OpenCV, torchvision, Daft and Hugging Facedatasetson COCO val2017 over four workloads, with parity tests against PIL, a subprocess per cell with pinned thread counts, peak PSS sampling and a memory cap. polars-cv runs eager, streaming, and streaming with its own pool off (set_row_parallelism(False)). The README's Benchmarks section shows the 0.38.0 results, including the two workloads where OpenCV is faster (thumbnails, and perceptual hashing at 5,000 images). - License files.
LICENSE-MITandLICENSE-APACHEat the repository root, linked intopolars-cv/so the sdist and wheels carry them asLicense-File. The project declaredMIT OR Apache-2.0but shipped no license text outsideview-buffer/LICENSE(MIT only). tests/test_readme.pyexecutes every Python block of the README, each on its own in a scratch directory, in the per-push lane. The previous README's quick start referenced undefined names and could not run.
0.37.0 — 2026-10-11¶
Detection metrics: streaming, null groups, and carry=. Every detection
metric now runs on polars' streaming engine except group_objects' object
lists, a null group key is evaluated as a group of its own, and every matcher
takes carry= to compute per-image measures in its own pass. Two results can
change: a null group's FROC/LROC curve is no longer empty, and
match_detections(images=...) no longer checks that images has one row per
image. Upgrading from 0.36: see the
migration guide.
Added¶
carry=on every matcher and onmatch_detections. Columns of the frame that definesimage_metadata(or expressions over it, named by their output name) are copied into it, one value per (image, class) row, and are then grouping keys likegroup_id. They are computed in the matcher's own pass, so a per-image measure of the same images (e.g.segmentation_measures(...).struct.field("dice")) needs no second reader of the image rows, which a streaming cache would otherwise buffer whole. A missing column, an unnamed literal, a repeated name and a nameimage_metadatadefines itself are refused.
Changed¶
- A null group key is a group of its own throughout the metrics. Curves,
operating points, AUCs and the weighted bootstrap already grouped nulls in
their
group_bys and windows, but about thirty joins on the keys did not match nulls, so a null group's FROC/LROC curve came out empty (sensitivity null, AUC 0 or null) while the same images under any name were evaluated. Every keyed join inpolars_cv.metricsnow passesnulls_equal=True(tests/test_join_ratchet.py), andtests/test_null_groups.pyholds every grouped entry point to "a null group equals the same group named". The interpolation's null-key group, whichjoin_asof(by=)matched to nothing, is interpolated too. Inmatch_detections, a nullimage_idis one image. grouped_scanrefuses an ungrouped reverseCumMax. Over the whole frame a reverse running max must buffer it in memory; no caller made one (every caller passes a key). Group by a constant column instead.
Fixed¶
- The detection metrics stay on the streaming engine. Under polars 2 three steps ran on the in-memory engine, which collects its whole input, and the guard meant to catch that did not see them:
- an unwindowed
pl.int_range(pl.len())row position in every ungroupedgrouped_scan(FROC/LROC/partial AUC/sensitivity-at) and in the weighted bootstrap's weight clusters, now awith_row_index; join_asof(by=...)ininterpolate_curve_lazy(froc_summary_table,froc_sensitivity_at_fp,lroc_sensitivity_at_fpf, partial AUCs) and in 11/101-point AP, now_grouped_scan.bracket: queries and knots sorted together and each knot filled forward and backward per group. Linear in the rows, and the bracketing knot'syis copied rather than matched back by float equality;match_detections(images=..., weight=/group=)'svalidate="m:1"join, which polars validates only in memory. It is now a plain streaming join. One row per image inimagesis the caller's contract and is no longer checked: a repeatedimage_idrepeats that image's objects instead of failing the query. Checking it cost a pass overimages. The image columns are now joined insidegroup_objects(newimage_columns=), before its final sort: a streaming join after the sort handed the matcher one morsel and serialized it (1.5 s vs 0.9 s at 80k images); before the sort, a call withimages=costs what one without it does.group_objects' per-(image, class) object lists stay the one in-memory step: polars 2.0 builds a list only in memory (aggregation,implode().overand a joined window all fall back). It holds object geometry only (an instance mask is outlined elementwise first); callers with one row per image call a matcher directly and skip it.evaluate_segmentation's default image ids are a row index for the same reason. Results are unchanged (bit-identical, or to the last bit of a weighted float sum, which varies run to run either way).- The streaming guard reads polars' own classification.
tests/_streaming_guard.pymatchedin-memory-mapand plugin/UDFcolumnar-functionlabels, soin-memory-joinand polars' own columnar functions passed as streaming, and a fixture asserted a frame-widerank()streamed. It now flags every node the graph's legend marks as an in-memory engine fallback, raises if the legend stops naming that class, scans every plan the eager metrics collect, and covers every public lazy entry point's grouped, partial, extrapolated and weighted paths.
0.36.0 — 2026-10-11¶
Performance, and one new parameter. Ops that walk their input a row at a time
read a crop or a vertical flip where it lies, convolve2d, convert_color
and the binary ops read any view without packing it first, and a tensor sink
copies a view at most once. sink(..., compact=True) writes only each row's
elements, for columns you persist. Nothing is removed. One result can change:
equalize_histogram and perceptual_hash of a u64 or i64 image saturate into
u8 instead of wrapping. Upgrading from 0.35: see the
migration guide.
Added¶
sink("numpy"|"ndarray"|"torch", compact=True)makes each row'sdataexactly its elements, row-major, at offset 0. By default a tensor sink hands a view over with the storage it reads from (a crop, transpose or flip of a decoded or in-place row, a blob's payload behind its header), so Parquet and IPC write that whole storage: 16 rows of a 200×200 crop of 256×256 images wrote 3.15 MB for 1.92 MB of pixels. A compact row is copied only when it is not already exactly its elements. A packed run of an input column (a passthrough, a crop of whole rows) is sliced in place. The other sinks refusecompact.
Performance¶
- Blur, erode, dilate, morphology gradient, pad and pad_to_size read a crop
or a vertical flip where it lies. These ops walk their input one row at a
time. They used to declare
RequiresContiguous, so the planner packed a view first, even one whose rows are each packed and that only a row stride sets apart. They now declareRequiresDenseRows, a newMemoryEffect. The planner packs only a layout without packed rows (a transpose, a horizontal flip), throughViewBuffer::to_dense_rows, and the kernels read rows at the view's own stride. One view-sized copy fewer per crop→op, held by the copy census and bytests/dense_rows_ops.rs(byte-equal to the packed copy).equalize_histogramandchannel_swapread rows the same way. convolve2dandconvert_colorread any view without a planned pack. Their arithmetic starts with a widening cast (to the accumulator, or f64 for the ranged color spaces), which walks a view once. Packing first only copied the view before casting the copy. A u8 crop, flip or transpose now costs them no extra copy.- Binary ops (
add,subtract,multiply,divide,blend,maximum,minimum,bitwise_*,apply_mask) read their operands where they lie, in any layout.zip_withpacked both operands, whatever their layout: two view-sized copies per call on any view. Each operand, broadcast to the output, is now read a block at a time through its walk's cursor (Walk::cursor): a block inside a packed unit (a contiguous operand of any rank, a row of a crop or a vertical flip) in place, any other (a transpose, a horizontal flip, a broadcast) packed into a stack scratch, never the whole operand. An operand broadcast along its trailing axes (a[h, w, 1]mask) is read once per element and repeated in the block. 1024×1024×3 u8add: crop 0.90 → 0.24 ms, vertical flip 0.84 → 0.33 ms. Measured withbenches/kernels.rsat 1024 against this cycle's first in-place reader: u8 + u16 over a transpose 3.4 → 2.9 ms, over a vertical flip at 65536×4×3 0.59 → 0.20 ms (32 ms before this cycle), an f32blendwith a[h, w, 1]mask 9.5 → 5.0 ms, with a[3]channel vector 7.6 → 2.3 ms, with a[h, 1, 3]column 8.6 → 2.7 ms. A contiguous rank-1 or rank-4 operand is not copied: 3M-element u8add0.83 → 0.23 ms.apply_maskwrites an inverted mask, or a 2-D mask expanded across the channels, once from the view instead of packing it first. - Convolution resizes of f64 and the 32/64-bit integers,
convert_colorof f64,convolve2dof f32 and f64, andequalize_histogramof any dtype but u8 read a view of any layout without packing it. Each converted its input before reading it, and packed it when the conversion was the identity (f64 to f64), or after it (equalize_histogram's conversion to u8, behind a planned pack of the wide input). They now read the view once, in the conversion: the resampler converts straight into its working buffer (one allocation for every dtype, where the integer resizes made two), andequalize_histogramdeclaresPacksOwnRows, so only a u8 input without packed rows is packed, by the kernel. A nearest resize still packs a transpose or a horizontal flip, as fast_image_resize does.convolve2dreads its accumulator-dtype input a row at a time, top down: rows in place where they are packed, a transpose's or a horizontal flip's each packed once, through its walk, into a ring of theksizerows a tap reaches. 1024×1024×3 f32, 3×3, timed with a throwaway harness (warm memory; thekernels.rscases allocate fresh inputs and read ~3x higher): crop 9 → 6.3 ms, vertical flip 27 → 7 ms, horizontal flip 27 → 12 ms, transpose 31 → 17.5 ms; contiguous input and u8 input unchanged. - A tensor sink that has to copy a view copies it once. When a numpy,
ndarray or torch row could not hand its storage over (a slice below the 50%
rule, a buffer another output shares), the view was packed with
to_contiguous()and the result copied again into the column's buffer: two view-sized copies for a strided view. Every such path now packs straight into the buffer Polars receives (ViewBuffer::packed_polars_buffer), held to one allocation bycopy_counts.rs.
Fixed¶
equalize_histogramandperceptual_hashof a u64 or i64 image saturate into u8 instead of wrapping. Their conversion to u8 sent the 64-bit integers through a plain cast, so an i64 300 became 44 and a -5 became 251, while every narrower integer saturated (300 to 255, -5 to 0). The 64-bit dtypes now saturate too, and the conversion matches every dtype explicitly rather than through a fallback arm.
Changed¶
scripts/verify.shruns the slow lane only with--slow. By default it now runs what CI runs on every push and PR: about 5 minutes on 4 cores instead of about 14.5. The slow lane was 65% of a full run, and it is CI's weekly search rather than a per-change check: its floor is one randomised 2,000-example parity search (449 s) that no worker count splits. Run--slowbefore a release (CONTRIBUTING's release process now says so) and after engine changes.--fastis removed and fails with a message, since its behaviour is now the default; any other unknown argument fails too.
Internal¶
- Benchmarks: pyvips, the newer ops, multi-branch workflows and an HTML
report. The comparison suite gains a
pyvips(libvips) adapter, with its operation cache disabled and every result materialised, so a timed pass is neither a cache hit nor only the building of libvips's lazy graph. Nine newer ops join the single-op benchmarks (letterbox,adjust_gamma,to_hsv,laplacian,morphology_open/close/gradient,warp_affine, a 5x5convolve2d), four pipelines are built from them, and aworkflowsscenario times graphs that branch and rejoin (one decode feeding a tensor, a thumbnail and a statistic; unsharp masking; mask-gated statistics; mask to contours to areas), written once per library and held to polars-cv's result bytest_benchmark_workflows.py.test_benchmark_adapters.pynow holds pyvips to polars-cv as it does OpenCV and Pillow, and pyvips joins thedevgroup so CI runs those rows.run_benchmarks --save-json/--htmlwrites a run and a self-contained report,python -m benchmarks.reportre-renders and merges saved runs, andrun_benchmarksrefuses a debug extension asrun_suitedoes. The Pillow adapter now decodes when preparing pre-decoded images:Image.openis lazy, so the first timed pass over each image paid its PNG decode. First run:benchmarks/reports/2026-10-10-library-comparison. - Per-value conversions run through
core::map. The u8 conversion of the image ops working in u8 (ImageU8),apply_mask's mask inversion and its 2-D mask expansion (a stride-0 broadcast, packed or mapped), and the rangedconvert_color(PixelMapnow has an output width:Rangedmaps 1 or 3 channels to 1 or 3) are element and pixel maps: dispatched, reading any view in its walk's runs.for_each_pixel_run, their side channel, is gone. MemoryEffect::PacksOwnRowsfor a kernel that reads packed rows in place and packs any other layout itself: the resizes and letterbox (which declaredStridePreserving, though they pack a transpose or a horizontal flip) and the binary ops (which declaredRequiresDenseRows, though they run outside the planner). No copy count changes.- The copy census holds every dtype to each op's declared memory effect.
An op pays no copy on any dtype for a layout its declaration reads in place,
and (on u8) pays one where it packs; on no dtype does it pay more than its
exact u8 count in
KNOWN. An op that runs on u8 must run on every dtype its contract accepts (accepted_input_dtypes), and an engine panic fails the census rather than reading as a refusal. Copies are counted from one byte per element, so a narrow copy of a wide input (a u8 working copy of f32) counts. It is what found the f64 and non-u8 copies above.copy_counts.rsadds that a contiguous buffer of any rank is never copied to be read, that a binary op orapply_maskallocates the same number of times at any size (the reader that first read views rebuilt every row's slice per 1024-element block: 58x slower than packing on a 65536-row view), and that a mask with packed rows costsapply_maskno copy.binary_views.rscovers rank 1, 2 and 4, lower-rank broadcasts, empty axes and float dtype pairs. - fast_image_resize 6.1 (from 5.6.0, which upstream yanked: it added a
default
stdfeature, a break). 6.0's only change is that opt-instdfeature, which default features keep. 6.1 speeds up the AVX2 vertical pass foru8pixel types. Resize output is unchanged: the bit-exact resize tests and the reference parity suite pass as before. - A census of what a view costs each op (
polars-cv/src/graph/copy_census.rs). Every catalogue op runs on a crop, a vertical flip, a transpose and a horizontal flip, and on a contiguous image of the same shape. The extra view-sized allocations are the copies the view cost. Each must be listed with its reason (KNOWN), and a listed copy that no longer happens fails too. At introduction: element-wise and view ops pay nothing on any layout. EveryRequiresContiguousop pays one pack. Binary ops pay two, packing both operands. Resize packs only a transpose or a horizontal flip. scripts/verify.shprints each check's duration, and drops work it repeated. Profiled on 4 cores (warm tree, ~14.5 min): the slow lane 562 s, the fast lanes 124 s and 85 s,cargo test -p view-buffer62 s. Of that 62 s, 49 s was one test,sixteen_bit_ops_match_the_legacy_code_through_the_lookup_table, looping over two dtypes and two layouts serially. It is now four tests over the same cases, which libtest runs side by side (lib tests 51 s → 24 s). The structural lane runs on every core (18 s → 11 s), and the in-memory lane no longer re-runs the structural guards, which read the source tree rather than an engine's output and have already run twice by then.
0.35.0 — 2026-10-09¶
Fixes and performance. A call's rows now spread over the thread pool whatever
ran before it, remote reads stream through a fetch window on Polars' own async
runtime, and every bootstrap plan streams. No method or parameter is added or
removed. Two results can change: nearest-neighbour resize at exact pixel-centre
ties, and DetectionTable.collect(), which now respects
pl.Config.set_engine_affinity. Upgrading from 0.34: see the
migration guide.
Fixed¶
-
Nearest-neighbour resize took the wrong pixel at an exact tie. Output pixel
itakes the source pixel under its centre,floor((i + 0.5) × src / dst); when that centre falls exactly on a pixel boundary (2 → 21 rows: row 10 at exactly 1.0; 2 → 7: row 3), the u8/u16/f32 path (fast_image_resize's nearest, stepping in floating point) could take the pixel before it. Nearest of every dtype is now one exact integer gather, reading a crop or vertical flip where it lies. Output changes only at those tie rows and columns. Kernel benchmarks (u8 RGB, interleaved A/B): downscale to 224 unchanged to 13% faster, 2× upscale 1.6–1.9× faster. The parity oracle's allowance for tie pixels is removed: nearest is now compared exactly (found by the randomized deep parity lane, CR-88). The golden corpus is re-recorded for the 7expr/*.filtercases, whose nearest row (16×16 → 7×7) has a tie at row and column 3. -
A query could make the next run of the same pipeline single-threaded. Whether a call spread its rows over the plugin's thread pool was decided by history: whether the previous call of the same compiled graph (cached process-wide, and since 0.34 shared by rebuilt expressions) had overlapped another. After a streaming run, an eager
with_columnsof the same pipeline ran on one thread (3.9× slower on 4 cores, debug build), and a streaming scan whose Parquet row groups differed in size ran its large row group on one thread on every run after the first (2.3×). A call now runs its rows on its own thread and on whichever pool threads are idle, under one plugin-wide budget ofPOLARS_MAX_THREADSthreads, re-checked before each row range; nothing outlives a call. Concurrent streaming calls still keep their rows on their own threads once the budget is used. Benchmark profile, 4 threads:split_streaming_then_eager2.2× faster,split_streaming_uneven_row_groups+11%, eager pipelines unchanged, streaming pipelines −3.8% to +6.5% (mediumthe one consistent dip, under the 7% gate); seebenchmarks/reports/2026-10-08-row-split-budget. - Remote reads started a second async runtime. The plugin built its own
tokio runtime (one worker per CPU, ignoring Polars' settings) beside
Polars'
ASYNCruntime, which the 0.55 object stores already spawn onto. Every remote read now runs on Polars' runtime alone, sized byPOLARS_ASYNC_THREAD_COUNT(defaultmin(POLARS_MAX_THREADS, 32)), and a rootclippy.tomlrefuses building another (test_async_runtime.py). - A call held every remote file it read before decoding any. Remote
paths (
file_pathsources,.cv.read_bytes(), the header-only.cv.width()and friends) were all fetched before the call's first row. Under Polars 2.0's streaming default a call is a morsel, and a Parquet morsel is a whole row group, so thousands of encoded images could be resident at once, with no download overlapping a decode. Each row now fetches its own path and the next rows' up toPOLARS_CONCURRENCY_BUDGETahead, and a body is freed after the last row naming it has read it. Each distinct path is still fetched once per call. Benchmark profile: remotefile_path+51–61% at 1 thread, and +43–45% / +6% at 1 / 4 threads with 20 ms latency. At 4 threads over loopback with no latency it is −4 to −6%, where decoding now shares the cores with the downloads. Seebenchmarks/reports/2026-10-08-fetch-window. - Bootstrap intervals ran two steps on Polars' in-memory engine. The
weight cells of each sampling unit (
unique().sort()inside a group-by) and the entity-to-image expansion forsample_col=(a list per entity, exploded per draw) each collected their whole input. They are now a nativeunique()aggregate sorted outside it, and a join, so every bootstrap plan streams;KNOWN_FALLBACKSloses both entries. Bounds are unchanged to the ulp (the pinned bounds hold exactly). - The docs claimed polars-cv's fetches shared Polars' concurrency budget.
The plugin links its own copy of Polars' I/O layer, so its budget, store
cache and runtime are its own:
POLARS_CONCURRENCY_BUDGETbounds polars-cv's requests and, separately, Polars' scans.sources.mdand thecloud.rsdocs now say so.
Changed¶
- The metrics collect with Polars' default engine. They passed
engine="streaming"at 14 call sites, which since Polars 2.0 restates the default and overrode a user'spl.Config.set_engine_affinity.DetectionTable.collect(engine=)defaults to"auto"(was"streaming"). Results are unchanged; a test refuses any literal engine choice in the package (test_engine_choice.py).
Docs¶
-
Release docs pass. Migration guide 0.34 → 0.35. The API reference now covers every public export:
OptFlags,build_info,dtype_name_for,IMAGENET_MEAN/IMAGENET_STD, the remaining geometry schemas (RING_SCHEMA,POINT_SET_SCHEMA,ANNOTATED_POINT_SCHEMA,CORRESPONDENCE_SCHEMA) andBoxFormat, and in metricsReportMetric,Matcher,COCO_IOU_THRESHOLDSandFROC_RATES. The README's operation list gains the element-wise math, binary and mask ops,flip,pad_to_size,assert_shapeand the contour ops. Its threading paragraph described the pre-budget behaviour ("about twicePOLARS_MAX_THREADS") and linked a renamed anchor; it now matches the streaming guide. The streaming guide and example 06 no longer passengine="streaming", the default. The panic-audit page (September 2026) is removed: the fix it recommended shipped as CR-34 andtest_engine_no_panics.pyguards it. Git history keeps the page. -
One issues ledger:
ISSUES.md. The open items of seven root documents (CODE_REVIEW_FINDINGS.md,FOLLOW_UPS_0.32.md,PERFORMANCE_PLAN.md,PERFORMANCE_HANDOVER.md,EXTENSION_TYPES_PLAN.md,POLARS_2_PLAN.md,UPSTREAM_ISSUES.md) were re-checked against the code and moved into one file, continuing theCR-NNids; the rest had shipped. CR-11 and CR-50 were already resolved and FU-05 (yanked numpy) was fixed, so they close; the PR #124 regressions found after merge are CR-67 and CR-68. Every closed id keeps a one-line entry, so ids cited in code still resolve. 22 benchmark report directories that measured superseded code, cited only by those documents, are removed (git history keeps them). The two metrics defects still open (CR-76, CR-77) are nowxfail(strict=True)pins intest_known_gaps.py. -
The streaming guide says how big a call is and how to bound it. A streaming call is one morsel and holds its whole output. Polars sizes morsels by rows, and a Parquet scan gives a call a whole row group: 20,000 images at 224×224×3
f32is about 12 GB in one call. The two knobs are row groups sized by bytes, orpl.Config.set_streaming_chunk_size(process-global, which is why polars-cv does not set it).tests/test_streaming_morsels.pychecks these claims at the plugin. - Feeding a training loop:
LazyFrame.collect_batcheswith a fixed-shapearraysink gives one(rows, *shape)NumPy tensor per batch, viewing the batch's memory.
Internal¶
-
scripts/verify.shruns its pytest lanes on every core. They were 89% of its time, single-threaded: on 4 cores the full run took ~37 minutes, 25 of them the slow lane. Withpytest-xdist(-n auto, a new dev dependency) the two fast lanes take 1.7 and 1.1 minutes instead of 4.8 and 3.4, and the slow lane ~8 instead of 25, bounded by its longest single test. The coverage gate holds across workers (pytest-cov combines them). -
The parity oracle rounded the largest double below 0.5 up. Its float → integer rule was
floor(|v| + 0.5), whose addition rounds: 0.49999999999999994 became 1 (the engine, correctly, 0) and an odd integer above 2⁵² the even one after it. It is now exact, with fixture tests (CR-89, found by the randomized deep parity lane on acast("u8")of a chroma channel that resampled to one ulp below 0.5). - The parity framework carried a bound through a wrapping cast. An
integer cast that can wrap (
u8 → i8) turns a one-level difference upstream (bilinear 127 vs 128) into 255, but cast declared a Lipschitz gain of 1, so the chain check failed on a correct engine. An op's gain now sees its input as its tolerance does (OpSpec.gain_for(x, p)), and cast's is infinite exactly whennp.can_cast(..., "safe")refuses (CR-90). -
Two more parity-oracle defects, both on correct engine output. A colour conversion's float bound was scaled by the whole image, alpha included, though alpha passes through: an infinite alpha made
to_hsv's bound unbuildable andto_ycbcr's infinite (so it accepted anything), and a large one inflated all three (grayscaletoo). They now scale by the colour channels (color_magnitude, CR-91). Theto_labreference was OpenCV's float Lab, whose sRGB curve is approximate: 0.57 off in a* on a dark pixel, where the engine matches the definition. It is now Lab by its definition in float64, and the bound tightens from 0.5 to 0.01 (CR-92). -
Benchmarks: a change to the row splitter is measured on a parallel pool. The splitter does nothing on one thread, and
relevancemapped it topipelines, the geometry and thesplit_cases only; PR #124's rewrite passed those while the eagerlistsink lost 15.6% and streamingsobel_x11.5% at 4 threads.row_split.rsnow also selectssingle_ops,e2eand thesink_cases, with a new@threads=Nselector marker (@threads=4) that travels inSELto both sides;targeted'ssplit_cases declaremin_threads=2themselves.run_suiterefuses fewer threads than a selection needs (and a smaller actual pool), andcomparerefuses results run on different thread counts. _lib._take_max_split_rows()reports the most rows one plugin call ran since it was last read (and resets).- Four Polars issues reproduced on 2.0.0 alone, drafted to file (now
CR-83 to CR-86 in
ISSUES.md). They are list aggregation in a streaming group-by, a per-plugin morsel-size hint, zero-widthArrayacross the C interface, and the out-of-core spill livelock. _plugin.calland_PluginNamespace._pluginlose theiris_elementwiseparameter, which every caller passed or defaulted asTrue. Every plugin function is elementwise, and_plugin.calldeclares it once. The streaming guard now treats a whole-column plugin call (columnar-functionover the.so) as a fallback, andtest_streaming_plans.pycovers the plugin's own entry points (cv.pipe,read_bytes,width, a contour and a point accessor). With the declaration flipped toFalse, those and every metrics plan built on the plugin fail, 30 in all; before this nothing did.- The direct
tokiodependency andCloudError::RuntimeErrorare removed (nothing builds a runtime any more). fetch::prefetch/FetchedBatch/row_bytes/row_header,cloud::read_file,cloud::read_files_concurrentandCloudError::UnsupportedSchemeare removed.fetch::Fetcherreplaces them, andcloud::read_remote_budgetedis the one remote read. Gone with them: the "fetch inline when a row is missing from the batch" fallback, which no row can reach any more, andread_file's second list of remote schemes (is_remote_pathis the one)._lib._last_fetch_peak_resident()reports the most fetched bodies the most recent call held at once, read bytest_fetch_window.py.row_split::CallTracker,Call::spreadsand thegeom_calls!macro are removed, with thecallsparameter every geometry row loop took. The budget replaces them (row_split::split).- New
targetedbenchmark casessplit_streaming_then_eagerandsplit_streaming_uneven_row_groups(run with--threadsabove 1). _lib._last_split_workers()reports how many workers (the calling thread and each pool thread that helped) ran the most recent call's rows, read bytest_parallel_rows.py.
0.34.0 — 2026-10-07¶
Polars 2.0 support. The plugin is built against the polars 0.55 Rust crates
(pyo3-polars 0.28, pyo3 0.29) and requires polars>=2.0.0,<3.0. Equal
pipelines in one query now run once (Polars' plugin CSE), and polars-cv's
outputs and memory take part in Polars' new out-of-core spilling. The
polars-cv API is unchanged; seeded bootstrap intervals differ from 0.33 for
the same seed. Upgrading from 0.33: see the
migration guide.
Changed¶
- Requires polars 2.0 (
polars>=2.0.0,<3.0; was>=1.44.2,<2.0). Every plugin call passesregister_plugin_function(is_deterministic=True), which polars 1.x does not accept. - Seeded bootstrap intervals differ from 1.x for the same seed. The draw is
hash(slot) % n, and polars 2.0 changedExpr.hash's values (polars guarantees hash stability only within a version). Intervals stay reproducible for a given seed and polars version; the method is unchanged. - A Parquet file with a tagged column, read without importing polars-cv,
now loads as polars' generic
pl.Extension("polars_cv.point", ...)rather than its plain struct (polars 2.0's default for unknown extension types). No data is lost:.ext.storage()gives the struct, or setPOLARS_UNKNOWN_EXTENSION_TYPE_BEHAVIOR=load_as_storage. - Performance: no regression across the 83-case regression suite
(median +0.7%);
.pointaccessors are 12–43% faster with ~25–30% less peak memory under polars 2.0 (see the upgrade report). - Metrics windows stream natively. Polars 2.0 runs
.over()windows in the streaming engine, sogrouped_scan(every per-group scan in the metrics) no longer falls back to the in-memory engine; its allowance in the streaming guard is removed, andtest_grouped_scan_stays_streamingholds it to no fallback at all.
Added¶
- Plugin CSE. Two equal pipeline expressions in one query (the same
pipeline and sink, built separately) run as one plugin call. Graph node ids
were random per build and went into the plugin's kwargs, which polars
compares, so no two builds were ever equal; the wire now names nodes by
position, including the nodes an op reads (
Plan.to_specrenames everyNodeRefand refuses one the graph does not name). The same change lets the compiled-graph cache reuse an entry across rebuilt expressions.
Fixed¶
- The extension segfaulted on import on macOS (every Python version),
introduced by this release's pyo3 0.29 bump; no released wheel was affected.
pyo3-polars'
PolarsAllocatorlooks up polars' allocator underPython::attachon the first allocation; under pyo3 0.29.3 that attach locks a mutex whose first lock allocates on macOS, so the lookup re-entered itself until the stack overflowed (pola-rs/polars#29731). The plugin's allocator now does the lookup through the C API alone, which cannot allocate through it, and aborts with the reason should a lookup ever re-enter. Linux and Windows were not affected. froc_curve_lazy(thresholds=[1, ...])raised under polars 2.0, whoseis_inrefuses to compare Float64 data with an Int64 list. Thresholds are now passed as Float64.- Debug builds undercounted their memory to polars. The SIMD dispatcher's
debug parity check made a buffer on a helper thread and freed it on the
caller's; polars' allocation count (which its out-of-core engine spills by)
saw the free but not the allocation, so each check pushed the count down,
to zero after a few calls. The check now frees aside as well. Release wheels
were not affected.
test_polars_counts_the_plugins_memoryholds the count to the truth.
Removed¶
POLARS_OOC_SPILL_POLICYfrom the out-of-core tests: polars 0.55 no longer reads it, so the tests that set it exercised nothing. The canary pinning the 0.54 stub spill backend (no spill files) is replaced by a test that forces a real spill (32 MB budget; lower can hit a polars 2.0.0 livelock in its spill loop, reproduced without polars-cv), reads the spill count fromPOLARS_OOC_LOG_METRICS, and checks every sink against the in-memory engine.read_ipc(memory_map=False)in the benchmark harness (removed in polars 2.0;read_ipcno longer maps).- The
RUSTSEC-2026-0176/-0177(pyo3 < 0.29) advisory ignores: fixed by the pyo3 0.29 bump.
Internal¶
- CI pins uv through a root
.tool-versions(uv 0.12.23, read by everyastral-sh/setup-uvstep, now v10.2.0) instead of resolvinglatestper job;tests/test_ci_tool_pins.pyholds every step to it. uv.lockmoves numpy off the yanked 2.4.0 (2.4.6 on Python 3.11, 2.5.3 on 3.12+).
0.33.0 — 2026-10-06¶
Weighted bootstrap intervals are corrected: 0.32 fixed each weight cell's
positive count in every replicate, which made intervals for sample-estimated
weights too narrow. The new weight_scheme= says what the weights are
("reestimate", the default; "stratified", 0.32's draw; "fixed", for known
or continuous weights). Unweighted intervals and every point estimate are
unchanged. Requires Python >= 3.11. Upgrading from 0.32: see the
migration guide.
Changed¶
- Python 3.11 or newer is required (
requires-python = ">=3.11"; was 3.10). Python 3.10 has reached end of life. Wheels are taggedcp311-abi3, and the CI matrix, the dependency-floors job and ty/ruff's target versions move with it; a structural test now holds all of them torequires-python. Thetomliandtyping_extensionsfallbacks for older Pythons are gone.
Added¶
weight_scheme=on every bootstrap interval (bootstrap_ci,froc_auc_ci_lazy,lroc_auc_ci_lazy,average_precision_ci_lazy,Report.ci), saying what the weights are and so how the resample treats them.Report.cialso takesweight_rtol=."reestimate"(default): weights estimated from the sample (p / q̂over target distributions, conditional ones included). Each(group, weight cell)is redrawn to its own size, so the weights given are the per-replicate re-estimates."stratified": 0.32's draw, which also fixes each cell'sgt_labelcounts. Use it when those counts were fixed by the study design."fixed": known weights (design weights, a continuous weight), carried unchanged on agt_label-stratified draw with no weight cells. A continuous weight now gets an interval instead of null bounds; 0.32 had no way out of weight cells short ofweight_rtol=1e300."fixed"raises if givenstrataorweight_rtol, which describe cells it does not form.- An unknown scheme raises.
Fixed¶
- Weighted bootstrap intervals were too narrow. 0.32 stratified the
image-level draw on
gt_labelcrossed with the weight cells, so every replicate kept each cell's positive count. When the weights were estimated over all images that count is random, and the weighted statistics depend on it: the positives' mix across cells follows each cell's observed prevalence. In simulation (and in a downstream report) the bootstrap SE came out 7-17% below the estimate's actual spread, and 95% intervals covered 0.88-0.93."reestimate"(the default) no longer crosses the label with the cells, which brings the SE within a few percent; targets conditioned ongt_labelalready give each label its own cells. A group with a single cell (unit weights) keeps thegt_labelstratum, so unweighted intervals are bit-identical. - A replicate that drew no positive image scored a value. A draw that does
not fix the
gt_labelcounts (sample_col, and now"reestimate") can draw no positive (or, for Mann-Whitney, no negative) image in a small group. Its statistic still has a value (NaNfor AP,0.5or0.0for the AUCs) that describes no resample; it was scored, pulling the bounds. Such a replicate now nulls its group's bounds, like an undefined one. - A weight-0 image no longer nulls its group's bounds. An image outside the target forms a singleton weight cell, and the singleton rule nulled the whole group although the image contributes nothing to any statistic. Cells whose weights are all zero are exempt.
0.32.0 — 2026-10-03¶
Detection evaluation in one call: evaluate_detections (boxes, polygons or
instance masks, COCO-style by default), evaluate_heatmaps (FROC) and
evaluate_segmentation, built on object-table matching, IoU sweeps, COCO
101-point AP and one bootstrap engine for any Statistic. Precision-recall
metrics are now weighted, and bootstrap memory is bounded by batching the
replicates. Several metric values were wrong and are now computed as
documented: multi-class matching, AP over tied scores, and multi-class or
grouped bootstrap draws. Requires polars >= 1.44.2. Upgrading from 0.31: see
the migration guide.
Added¶
-
.contour.single(label=): each row's one contour, from a contour-set column. A set of any other size fails the query, and the error quotes that row'slabel(an image id, say) rather than a morsel-local row number. It is howmatch_detectionsrefuses an instance mask of several regions. That failure is now apolars.exceptions.ComputeErrorraised by the plugin ("'img1' holds 2 contours, not one"). It used to be aValueErrorraised by a Python UDF. -
One-call evaluation:
evaluate_detections,evaluate_heatmaps,evaluate_segmentation. Each goes from predictions and ground truth straight to a report. evaluate_detections(preds, gts, box_format=...)takes long object tables of boxes, polygons or instance masks. By default it follows COCO: it re-matches at 0.50:0.05:0.95, uses 101-point AP, keeps 100 detections per class, and leaves classes without ground truth out of the means. It agrees with pycocotools'COCOevalto 1e-12 on mAP, AP50, AP75 and AR@100 (tests/reference/test_coco_parity_ref.py, which runs when pycocotools is installed). Area ranges and crowd regions are not modelled.evaluate_heatmaps(df, heatmap=, gt=, **ContourMatcher options)adds a FROC summary: sensitivity at each rate and their mean, the CPM.- The
DetectionReportboth return has:.summary,.per_class,.per_threshold;.ci(metric, by=...), which isbootstrap_ciof the statistic behind each number;.matches(t), which gives each prediction input row its TP/FP outcome and matched ground-truth row, for error analysis;.pr_curve,.froc,.confusion, and.tablefor anything lower level.
-
evaluate_segmentation(df, pred=, target=)and the per-row expressionsegmentation_measures(...)give Dice, IoU, ASSD, HD, HD95 and the directed mean distances in one graph per row.- Hole borders count as boundary.
frame="image"leaves the image edge unmeasured.spacing=gives physical units.- Empty masks follow an explicit policy.
SegmentationReport.ciresamples images with the same seeded hash draw as the detection intervals.
-
.contour.set_boundary_distances(other, sample_step=, frame=). Boundary distances between two contour sets, each read as one boundary (the union of its outlines), returned as one struct per row. This is the surface distance of a mask with several regions, as MONAI computes it. The engine'sboundary_distancesnow takes outline sets, and the per-contour accessor is the case of one outline per set, so both share one implementation. -
match_detections(predictions, ground_truth, ...): object tables in, aDetectionTableout. Takes the long format detections usually arrive in, one row per object, and dispatches on the geometry column's dtype: - boxes, as
BBOX_SCHEMAor as four numbers inbox_format=; - polygons (
CONTOUR_SCHEMA); - per-instance masks, whose single outline is extracted. A mask that holds several regions fails the query and names its image.
group_objects is the reshaping step it uses, from long rows to one row per
(image, class) with object lists. It crosses every image with every class,
so images and classes that appear on only one side (or only in images=)
are not lost. It also applies COCO's max_detections. images= can carry
per-image weight= and group= columns.
- bbox_from_coords(coords, format=...). Reads "xyxy", "xywh" or
"cxcywh" boxes, from one list column or four columns, into BBOX_SCHEMA.
The format is required.
- IoU sweeps: matching once per threshold, as COCO does.
BBoxMatcher(iou_threshold=[...]) and ContourMatcher(iou_threshold=[...])
match at each threshold in one plan. The table carries iou_threshold on
both frames, and DetectionTable.iou_thresholds lists them.
- A detection that lost its ground truth at 0.5 can now claim it at 0.75,
which re-thresholding a single match cannot do.
- Evaluations that would pool the thresholds, and so count each ground truth
once per threshold, are refused in one place, DetectionTable.frames. You
can group by iou_threshold, average over it (mean_ap()), or select one
threshold (at_iou_threshold, which now slices a matched threshold
exactly).
- mean_average_precision evaluates a sweep's thresholds on their own
matchings. The bootstrap keeps a replicate's thresholds paired.
Statistic: every detection metric as one grouped, lazy reduction.AP(interpolation),Recall,PrecisionAt/RecallAt/F1At(score),FROCSensitivity(fp),CPM(fp_rates),FROCAUC,LROCAUCandLROCSensitivity(fpf)each expose.by_group(table, keys)(a lazy[*keys, value]frame) and.value(table). They compose the existing authorities and add no second estimator; a test holds every public metric function to agree with its statistic, and fails on a new public metric that has none.MeanOver(statistic, undefined=...)averages one over facets (class_id,iou_threshold).mean_ap()is mAP:undefined="exclude"leaves classes without ground truth out, as COCO does, while"zero"averages them in as 0, which is whatmean_average_precisiondoes.bootstrap_ci(table, statistic, ...): one CI engine for any statistic.froc_auc_ci_lazy,lroc_auc_ci_lazyandaverage_precision_ci_lazyare now this withFROCAUC,LROCAUCandAP, and their outputs are unchanged. A drawn image brings its rows for every class, so the classes averaged by aMeanOverstatistic come from one resample. That makes an mAP interval correct, where it would be wrong with one independent resample per class.- COCO 101-point AP.
interpolation="101_point"onaverage_precision,mean_average_precisionandPrecisionRecallResult.auc. 11-point and 101-point AP now have a grouped lazy form (ap_by_group), so they can be bootstrapped, andmean_average_precisionno longer loops per class for 11-point. group_by=onfroc_sensitivity_at_fp,froc_summary_tableandlroc_sensitivity_at_fpf, through a groupedinterpolate_curve_lazy.
Changed¶
- Bounded bootstrap memory. A confidence interval's replicate frame holds
n_bootstrap × detectionsrows, and the metric's sorts and group-bys hold all of them at once, natively or not. The replicates are now evaluated in batches of 50 within the one lazy plan, and the streaming engine runs the batches one after another, so peak memory follows the batch rather thann_bootstrap. Each batch draws its ownbootstrap_idrange, and a draw hashes its global slot id, so every bound is bit for bit what one whole draw gives (pinned at batch sizes 1, 7, 50 and unbatched).bootstrap_ci(AP), 2000 images and 50k detections, 1000 replicates: peak RSS 4257 MB → 614 MB, 20.4 s → 14.8 s. Nothing is collected to choose the batch, and there is no new parameter. - Metrics stay on the streaming engine, except where it cannot help.
Polars' streaming engine hands any step it cannot run natively to the
in-memory engine, which collects that step's whole input first. In the
metrics, these steps were
.over()running sums, maxima and shifts, sorts insidegroup_by().agg(), a quantile inside a group-by, and a Python UDF. They sat in the PR, FROC and LROC curves, AP, the AUC integrals and the bootstrap;froc_curve_lazyalone had 15 of them. Every one is now native, or is one of the cases thattests/_streaming_guard.pylists with its reason: - A
.over()scan straight after the sort on its keys.metrics/_grouped_scan.py::grouped_scanis the one way the metrics write a per-group scan: one sort, then windows. Such a window reads the frame the sort has just collected, so it adds no memory of its own. An all-native version was built (frame-wide scans restarted exactly at group starts) and measured. On the bootstrap it held as much memory and ran 2.4x slower, so it was not kept. The guard checks in the physical plan that the window's input is that sort, keyed on the window's keys. - Three list aggregations, which polars cannot build in a streaming
group-by:
group_objects' per-image object lists, a sampling unit's weight cells and an entity's images. Each runs on a per-object or per-image frame, never on the bootstrap's replicate frame. - Reductions that must be reproducible are exact (
exact_sums): Int128 fixed point, exact up to one rounding of each term at 2⁻¹²⁵ of the largest magnitude, and one rounding of the result. They are native, and they no longer vary between runs or thread counts. - The bootstrap interval's
quantileis read off ranks, bit for bit as polars'quantile(q, "linear")computes it. group_objects(max_detections=)caps inside the list aggregation it already builds, instead of using a separate window.PreMatchedAdapter'sdet_idxordinals andto_per_imagego throughgrouped_scan. The weight-cell clustering is native: a frame-wide count minus its value where the part starts.- The mask-instance region check (
match_detections(geometry=<masks>)) is no longer a Python UDF that received the whole column of extracted outlines at once. It is.contour.singlein the plugin (see Added), andtests/test_no_python_udfs.pykeeps Python UDFs out of the package. precision_recall_curveno longer collects the weighted detection table just to test whether it is empty. It collects only the curve and the two totals.tests/test_streaming_plans.pyreads every public plan's physical streaming graph and fails on any fallback not listed, with its reason, intests/_streaming_guard.py.- Removing these fallbacks does not by itself bound memory. The bootstrap
holds its whole replicate frame (
n_bootstrap × detectionsrows) in its sorts and group-bys, natively or not; see Bounded bootstrap memory. - The AUC integrals take a frame, not an aggregation.
_auc_expr'strapz_auc_expr/partial_auc_expr/mann_whitney_auc_exprwerepl.Exprreductions forgroup_by().agg(), which sorted inside the aggregation. They are replaced bytrapz_auc/partial_auc/mann_whitney_auc(lf, keys=...), which return[*keys, auc]. These are internal; the public metric functions are unchanged. -
DetectionTable.to_per_image()returnsmax_score,top_is_tpandbest_tp_score, and nodetectionslist. The per-image list of detection structs existed only for LROC to read those three values back out of it, and building it is a list aggregation the streaming engine cannot run. -
polars-cv now requires
polars>=1.44.2(was>=1.43.2). The bootstrap CIs cache their resample lazily (LazyFrame.cache()), because the replicate metric reads it many times and the streaming engine would otherwise re-run the draw at each read (about 4x slower). On polars 1.43.2, a cached frame read under different projections returns wrong rows, which produced AP bounds above - The
dependency-floorsCI job tests the new floor. - Precision-recall metrics are weighted.
precision_recall_curve,average_precision(all-points and 11-point),mean_average_precision,precision_at_threshold,recall_at_threshold,f1_at_thresholdandconfusion_at_thresholdused to ignoreimage_metadata.weight, so a reweighted study's AP silently ignored its weights while its FROC/LROC were weighted. Each detection now carries its image's weight: precision isΣw·tp / Σw·(tp + fp)and recallΣw·tp / Σw·n_gts, matching scikit-learn'ssample_weight(a weight ofkequals the image drawnktimes; a test checks this across every entry point). - Unit weights reproduce every previous value exactly.
- Each function takes
weight_agg=like FROC. - Zero-weight detections add no PR point.
- The PR curve gains
cum_weighted_tp/cum_weighted_fpandPrecisionRecallResultgainsweighted_gts. ConfusionResultkeeps its rawtp/fp/fncounts andto_dict(), and gainsweighted_tp/weighted_fp/weighted_fn. Itsprecision/recall/f1now read the weighted values, so they agree with the threshold functions.- The grouped AP authority (
all_points_ap_by_group) takesweightandgt_masscolumns in place oftotal_gts. - Bootstrap intervals re-estimate sample-derived weights correctly.
froc_auc_ci_lazy,lroc_auc_ci_lazyandaverage_precision_ci_lazycarried each image'sweightunchanged into every replicate. That is wrong for an importance weightp / q̂whoseq̂comes from the same sample, such as a vendor or prevalence mix reweighted to a target. Each replicate draws a different mix, so held-fixed weights let the weighted mix drift off target and widened the interval. - Weight cells. The resample is now stratified on weight cells: units whose
weights agree within the new
weight_rtol(relative, default1e-6) form a cell, and each(group, cell)is redrawn to its own size. - Entity-level draws. Under
sample_col, entities of different sizes still let the image mix drift, so each drawn image's weight is rescaled by(n_c/N) / (n*_c/N*). That restores its cell's full-sample share, which is the re-estimatedp / q̂(exactly 1 for image-level draws). - Why it is exact. The weighted statistics are scale-invariant ratios, so a
weight that depends only on its cell's count (
p / q̂, post-stratification, raking) is then exactly the weight re-estimated in each replicate. This holds for weights computed per group or globally. The cells come from the weight itself, everything stays lazy, and point estimates are unchanged. - Unit weights: bounds stay bit-identical, apart from single-unit groups (below).
- New
strata=columns are crossed into the cells. They separate cells that happen to share a weight. - Behaviour change: a group with a weight cell holding a single unit now
reports null
ci_lower/ci_upper, keeping its point estimate, as degenerate groups already do. That cell has no bootstrap variance: a continuous weight (every image its own cell), or a one-image group, would otherwise report a zero-width interval. - Behaviour change:
average_precision_ci_lazyis now weighted and weight-stratified like the others. - A downstream proposal asked for a
reweightcallback hook. It was not adopted: it re-weighted the replicates but not the point estimate, and had to handle replicates that draw nothing from a positive-target cell. The cell-stratified draw makes the existing weights correct as they stand.
Fixed¶
-
Multi-class matcher input duplicated every detection.
BBoxMatcherandContourMatcherread multi-class data as one row per (image, class), but attachedclass_idto detections by joining onimage_idalone. On an image with k classes, every detection appeared k times, once under each class, which corrupted per-class AP, mAP and FROC for any multi-class evaluation. The matchers now share one tail (_matching/_table.py), and each detection keeps its own row's class. Images with no predictions no longer leave a null-score row in aBBoxMatchertable. -
All-points AP over-counted score ties. The envelope was integrated as a trapezoid,
(Pₖ + Pₖ₋₁)/2, while the docstring named the step ruleΣ (Rₖ − Rₖ₋₁)·Pₖ(Pascal VOC 2010+ / COCO / scikit-learn). The two agree whenever scores are distinct. They differ when a tied block mixes TPs and FPs, which lowers precision while raising recall: TP@0.9 then {TP, FP, FP}@0.5 against 2 GTs read 0.875, and is now 0.75. Tied scores are common with heatmap-derived scores. This affectsaverage_precision,mean_average_precision,PrecisionRecallResult.auc(),average_precision_ci_lazyand the grouped authority. Untied results are unchanged. - The 11-point recall grid is the reference one. It was
i / 10; it is nowi · 0.1, as the VOC devkit (np.arange(0, 1.1, 0.1)) and COCO (np.linspace) build their grids. A recall landing exactly on 0.3, 0.6 or 0.7 now compares the way it does there. - AP and mAP are reproducible bit for bit. Their sums used Polars'
chunked
sum/mean, whose last bits depend on how the values are chunked. They now sum exactly in Int128 fixed point (exact_sums), which does not depend on chunking, order or thread count, and which streams (see Metrics stay on the streaming engine under Changed). -
DetectionTable.to_per_image()'stop_is_tpno longer depends on row order. It took the first detection of an unstable sort by score, so when two detections tied for the top score, either could be read, and reversing the input flipped the result (and LROC'stop_scoringvariant with it). The tie now goes to the lowestdet_idx, the order the matcher ranked them in, which is what the LROC reference reads. -
.contour.boundary_distances(frame=)no longer spikes next to the frame. The frame removed the target's boundary lying on it, but kept source samples a hair inside it. An annotation drawn even slightly inside the image edge, against a prediction cut by that edge, therefore lost its nearest partner and was measured across the region: about 30 px instead of 1 px in the reported case, inflating the GT→prediction mean,hdandhd95. An inset frame only moved where the jump happened. Now the frame only chooses which samples are measured; each one is measured to the other side's whole outline, so distances are continuous as boundary approaches the edge. The engine's clipped-target outline (PreparedOutline::within) is deleted, and the Rust test that asserted the 30 px reading is rewritten to assert 1 px. -
Multi-class bootstrap replicates over-drew mixed-label images. The image-level resample took one base row per distinct
(image_id, gt_label).image_metadatahas a row per(image, class), so an image positive for one class and negative for another sat in both strata: it had two draw chances per replicate, and replicates held more images than the sample. An image now has one draw slot, in the positive stratum if any of its classes is positive. Single-class tables are unchanged. match_detections(images=...)repeated an image listed twice. Theimages=frame'sweight/groupcolumns were joined onto the per-(image, class) object lists, so animage_idappearing twice in it repeated that image's detections and ground truth. The join now requires one row per image and fails the query on a repeat.- Grouped bootstrap draws brought other groups' rows. A draw partitioned by
a group key was joined back to the metadata and detections on
image_idalone. Undergroup_by="class_id", a draw in one class's partition therefore also brought the image's other classes, and every class's replicate gained the other classes' draws. Draws now carry their partition keys and bring only their own group's rows. Groups that never share an image, such as a per-studygroup_id, are unchanged.
0.31.0 — 2026-10-03¶
Open polylines become a first-class contour (is_closed=False is read),
with boundary distances, coverage matching for line-shaped ground truth,
closing a line along the image frame, and coordinate-list constructors.
FROC/LROC curves are read past their end by one explicit policy, null by
default. Contour labelling and label_reduce region scans are orders of
magnitude faster. Upgrading from 0.30: see the
migration guide.
Security¶
allowed_rootscould be escaped with afile://URL naming a host. The policy check strippedfile://and judgedhost/etc/passwdas a path relative to the working directory, while the read parsed the URL and opened/etc/passwd— so with the working directory inside an allowed root (e.g.allowed_roots=["."]) any local file was readable, through.cv.read_bytes(), thefile_pathsource and the new header-metadata path reads alike. One resolver (cloud::local_file_path) now serves the check and every local read, so the file judged is the file read. Afile://URL naming a host other thanlocalhostis refused (it names no local file), and afile://URL is percent-decoded, as its comment always claimed.
Removed¶
extract_contours(mode="tree"). It ran exactly asmode="all"— the kernel told onlyexternalfrom the rest — while documented as "full hierarchy", which nothing builds (borders come back as a flat list). Usemode="all";"tree"is now refused by the wire like any unknown value.
Changed¶
- FROC/LROC read a curve past its end by one policy, null by default
(breaking for
froc_auc).froc_auc(fp_range=...)filled the curve flat past its last operating point whilefroc_summary_tableandfroc_sensitivity_at_fpreturned null there, so one table reported an unknown sensitivity at 2 FP/image and a defined AUC over 0–2 FP/image — for detections from a thresholded mask, which exist at a single operating point, an AUC that was nearly the curve's maximum sensitivity. The two curve readers (partial_auc_expr,interpolate_curve_lazy) now share oneextrapolateargument,"none"(default) or"flat"(extend the endpoints — the LUNA16np.interpconvention), threaded throughfroc_auc,froc_sensitivity_at_fp,froc_summary_table,lroc_auc,lroc_sensitivity_at_fpf,froc_auc_ci_lazyandlroc_auc_ci_lazy. Anfp_rangereaching past the curve now gives a null AUC; passextrapolate="flat"for the previous value. LROC values are unchanged (its curve always spans[0, 1]). A bootstrap replicate whose curve stops short of the window nulls the bounds instead of being scored as an empty draw's 0.0.
Added¶
on_error("raise" | "null")on the geometry accessors. One invalid row — e.g. aclose_along_borderline ending beyondmax_snapfrom the frame — failed the whole query, and the only remedy was re-implementing the check in Polars expressions.pl.col(...).contour.on_error("null")(and on.point/.bbox) nulls just the rows whose data a function refuses, mirroringsource(on_error="null"); it composes withon_null. An error about the operand column (a contour set where one contour is expected, an unreadable dtype) still raises: those are nowSchemaMismatcherrors internally, which the policy never nulls."null_with_message"is refused (an accessor has no struct output to carry the message).-
.contour.boundary_distances(frame=...). A region cut off by the image edge has a frame segment in its outline that no annotation traces, so every frame sample was measured as error (ASSD/HD/HD95 inflated even for a perfect prediction).frametakes a bbox per row (literal or column): boundary not strictly inside it is dropped from both directions, and each side is measured to the other's boundary clipped to the frame. Insetting the bbox drops boundary near the edge too. -
ContourMatcher(score_reduction=, score_region_mode=). Detections were always scored by the peak heatmap value inside them (label_reduce's"max"/"interior", hard-coded), so on a heatmap that saturates most detections tied at the peak and FROC/LROC collapsed to a few points. Both pass through tolabel_reduce; the defaults are unchanged. - Per-row
ContourMatcherparameters.iou_threshold,extraction_threshold,min_contour_area,coverage_toleranceand the newmin_contour_area_fractioneach take a Polars expression, read per row by the op they reach (correspond/correspond_by_coverage,threshold,extract_contours) — e.g.coverage_tolerance=5.0 / pl.col("spacing_mm")for a physical tolerance across devices. A literal is still checked at construction; a per-rowiou_thresholdstores nomatching_iou_threshold. - Contour predictions in
ContourMatcher.pred_colmay be a contour or contour-set column, scored byscore_col(List[float]aligned with a set, a float for one contour per row; a count mismatch fails the query). The contours are the detections as they are — no extraction, no heatmap scoring, and a caller's 0.0 score is kept — and share the matching, duplicate policy and table build.score_colwith a heatmap prediction is now refused instead of silently ignored. -
DetectionTable.filter_images(ids | predicate). Keeps the named images (or theimage_metadatarows a predicate selects) and their detections, with the stored matcher settings, for stratified evaluation without rebuilding the table by hand. -
Line-shaped ground truth in
ContourMatcher.match_by="coverage"(withcoverage_tolerance) pairs predictions with GT by coverage, so GT annotated as polylines — skinfolds, muscle edges — can be scored, which IoU cannot;gt_colmay now be a contour or contour-set column (withauto_resize=False) instead of a mask.duplicates="ignore"drops a prediction whose only hit was an already-matched GT rather than counting it as a false positive (default"false_positive", as before). -
.contour.correspond_by_coverage(other, tolerance, threshold=0.5, order=None, sample_step=None)pairs contours by coverage rather than IoU: the fraction of each target's boundary samples inside the candidate (when it is closed) or withintoleranceof its edges. It scores line-shaped targets — polyline annotations, which IoU cannot (a polyline has no area) — through the same greedy, exclusive rule ascorrespond. -
Correspondences report duplicates.
CORRESPONDENCE_SCHEMA(from.contour.correspond,.correspond_by_coverageand.bbox.correspond) gainsduplicate: List(Boolean): an unpaired element that cleared the threshold against a target another element had already claimed — a repeated hit, which LUNA16/CAMELYON-style evaluation ignores rather than counting as a false positive. -
Header-only metadata from path columns, and
.cv.image_info()..cv.width(),.height(),.channels()and.image_dtype()take aStringcolumn of paths as well as image bytes, with the samecloud_options,on_errorandallowed_rootsas.cv.read_bytes(): a local file is read only as far as its header needs (a growing prefix, since a JPEG's frame header can sit behind large EXIF/comment segments), instead ofread_bytes()reading every file whole first. Remote objects are still fetched whole (a ranged read would nest the object store's concurrency permits)..cv.image_info()returnsStruct{width, height, channels, dtype}from one header read per row. Path options on a bytes column are refused. -
.contour.close_along_border(width, height, arc="shortest", max_snap=2.0)closes an open line whose ends lie on the image frame into the region it bounds — a pectoral muscle edge, a skin line — by snapping each end onto its nearest frame edge and walking the frame back from the line's end to its start through the corners it passes (arc: the shorter way, or"clockwise"/"counterclockwise"as displayed). A straight chord would drop a corner region entirely. An end farther thanmax_snapfrom the frame is refused rather than joined; a closed contour is refused. -
Geometry from coordinate lists, and back.
polars_cv.geometry.point_from_coords(expr, order="xy"|"yx"),contour_from_coords(expr, order, closed=True)andcontour_set_from_coords(...)build points, contours (open polylines withclosed=False) and contour sets from[x, y]or[y, x](row, column) pairs —ListorArray(_, 2), integers or floats — instead of hand-built structs..point.to_coords(order)and.contour.to_coords(order)are the inverse (Array(Float64, 2)per point; a contour with holes is refused). A pair that is not two non-null numbers is an error naming its row. -
largest(k=1)keeps theklargest contours of a set by area, largest first, equal areas in input order — as a pipeline op (.extract_contours().largest()) and as.contour.largest(k)on a contour column, both from one definition. The result is always a set (a lone contour is a set of one), so downstream set-level code is unchanged;kmay be per row, andk = 0is refused. -
extract_contours(min_area_fraction=)filters contours by area as a fraction of the image's own height x width, resolved per image as it runs, so a speck threshold scales with resolution without reading each image's size first. In (0, 1]; a literal outside it is refused when the pipeline is built, a per-row one when its row runs. Withmin_areatoo, a contour must pass both. -
.contour.boundary_distances(other, sample_step=None)— point-to-edge boundary distances, both directions, asStruct{mean_a_to_b, mean_b_to_a, assd, hd, hd95}: the directed means, the average symmetric surface distance, the Hausdorff distance and the larger directed 95th percentile (MONAI's convention). Each boundary is sampled at its vertices, plus everysample_stepalong each edge when given, and measured to the other's edges, so two tracings of one outline at different vertex densities are 0 apart (hausdorff_distance, which stays vertex-to-vertex, called the reporter's pair 50 apart). Open contours are measured as polylines; for physical units.contour.scale()both sides by the pixel spacing first. Broadcasts a set against a single likeiou. -
.pointaccessors take point sets. APOINT_SET_SCHEMAcolumn (List(point)) gives one value per point, in input order, as aList— the.contourarity rule:pl.col("pts").point.distance_to_contour(pl.col("c"))measures every point against the row's contour without an explode/group-by. A contour or bbox operand broadcasts against the set; two point columns broadcast either way, and a set on both sides is refused. A null point in a set gives a null in its place.x()/y()are typed functions now, so they take sets too and read theX/Yspellings the other accessors accept. -
froc_operating_range(table, group_by=...)reports each curve'smax_fp_per_imageandmax_sensitivity, so a truncated curve is visible next to the AUC.
Performance¶
extract_contourslabels runs, not pixels. Every call flood-filled the whole background through a per-pixelu32label array (34 MB for a 3328 x 2560 mask) before tracing anything — even for an empty mask, and inmode="external". Components are now found by union-find over each row's runs (word-at-a-time run detection, an all-zero mask returns at once), work proportional to the runs. Finding the borders of a 3328 x 2560 blob went from ~140 ms to under 1 ms in a release build; output is unchanged, held to the old flood fill (kept as a test oracle) on randomized masks.label_reducescans regions by scanline.region_mode="interior"/"boundary"tested every pixel in a contour's bounding box against every edge, so cost grew with vertices x pixels: a 4,096-vertex disc of radius 1000 took ~11 s per image. A scanline winding scan now settles each row from its edge crossings, leaving to the exact predicate only the pixels whose centre lies within rounding distance of an edge and the rows through a vertex — so the selected pixels, their order, and every score are unchanged (held to the per-pixel predicate on randomized, self-intersecting and half-pixel contours). The same disc now takes ~23 ms.
Fixed¶
ContourMatcherrefuses a null score for contour predictions. A null-scored contour took part in matching — and could claim a GT — and was then dropped from the table, so the GT counted as missed and the detection that should have matched it as a false positive. A null score list beside present contours dropped the row's detections. Both now fail the query, as a score count mismatch does; a null contour set is still an image without predictions.close_along_borderrefuses a NaN or negativemax_snap(a NaN passed every distance test, so a line in the middle of the image was "snapped" to the frame and closed) and a non-finite frame size.- Densifying an edge of overflowing length no longer loops forever.
boundary_distances/correspond_by_coveragewithsample_stepappended samples without end along an edge between finite vertices ~1e308 apart; such an edge is sampled at its vertices. point_from_coords/contour_from_coordsrefuse a non-finite coordinate, naming the row, rather than building a point every geometry reader then refuses.- The streaming guide no longer claims the plugin stays within
POLARS_MAX_THREADS. The plugin's pool is its own (sized by the same setting), and overlapping calls run inline on Polars' threads while a lone call spreads over that pool, so up to about twice the setting can be runnable. The guide and README now say so, and when to lower the setting. -
A length-1 geometry operand broadcasts. Every
.point/.contour/.bboxaccessor failed with "geometry row 1 out of bounds" when an operand was one row — a bbox, point or contour built from literals, or a column the optimiser folded back into a literal — whatever the arity of the other side (single points and contour sets included, not only the new point sets). The three geometry readers now locate rows through one shared locator that broadcasts a one-row column, as a per-row parameter already did, andcorrespond'sorderreads through the same broadcast-aware accessor. -
Open contours are measured as polylines;
is_closedis read. The field was writtenTrueand never read, so an open polyline was measured as the ring its closing edge would make: the point (0, 10) was 7.07 from the L (0,0)->(10,0)->(10,10) instead of 10.is_closed = Falsenow makes a contour an open polyline: perimeter,point.distance_to_contour,point.nearest_point_on_contour,hausdorff_distance,bounding_box,simplify,convex_hulland the point-wise transforms (translate,scale,flip,normalize,to_absolute) measure or keep it as a line; region functions —area,centroid,is_convex,winding,ensure_winding,contains_point,iou,dice,pairwise_iou,correspond,label_reduce,point.signed_distance_to_contour, and the pipeline'scontoursource — refuse it, naming the row, instead of silently closing it.scale(origin="centroid")refuses one (a polyline has no centroid). An open contour with holes is an error; an unspecifiedis_closed(absent, or null) reads as closed, as before. view-buffer models the polyline as its own type (Outline::Open), so a region measure cannot be handed one: every.contouraccessor declares which it reads. -
PreMatchedAdapterrefuses keys and columns it used to drop silently. Withclass_colomitted, every detection was re-keyed to the default class whileimage_metakept its ownclass_id, so the(image_id, class_id)joins matched nothing andlroc_aucreturned 0.0 for a perfect detector. Aclass_idcolumn on the detections or onimage_metanow requiresclass_col, andclass_colrequiresclass_idonimage_meta. A namediou_col,det_idx_col,n_gts_col,weight_col,gt_label_colorgroup_colthat is missing now raises, as the docstring said, instead of falling back to its default (a misspelledweight_colleft every weight at 1.0). Withoutiou_coltheioucolumn is null rather than 0.0, andDetectionTable.at_iou_threshold— somean_average_precision— raises on such a table: it compared the placeholder and turned every TP into an FP. -
A sourceless
Pipeline()no longer assumes it continues a buffer.contours.pipe(Pipeline().simplify(2.0))raised "expects contour input but pipeline is currently in buffer domain" whilecontours.simplify(2.0)worked: an unsourced plan started from a placeholder buffer state and checked its first op against it, although nothing yet said what it follows. Its first op now anchors it in the domain that op reads (a buffer whenever the op reads one, so every buffer pipeline plans as before), and.pipe()plans every op again from the real upstream node, so a continuation onto the wrong domain is still refused there..pipe(Pipeline().op())now plans like.op()on the lazy node for every chainable op (test_a_sourceless_continuation_plans_like_the_lazy_method).
0.30.0 — 2026-10-02¶
Every op now computes and stores in the dtype it declares, following NumPy and OpenCV where they define an answer: colour ranges per dtype, NumPy promotion, IEEE division, NaN propagation, NumPy histogram binning and pixel-edge contours. Kernels are substantially faster across the board. Upgrading from 0.29: see the migration guide.
Breaking changes¶
- Colour conversions use each dtype's value range, as OpenCV does. They assumed 0–255 for every dtype. Now:
- floats are in [0, 1]: HSV gives H in degrees and S, V in [0, 1];
YCbCr centres chroma at 0.5; Lab reads RGB in [0, 1], and converting
from Lab gives
f32RGB in [0, 1] (it was 0–255). A carried alpha is rescaled with the image (u8 RGBA -> Lab gives alpha 1.0, not 255.0). Scale 0–255 float images by 1/255 first; - u16, u32 and u64 span 0..MAX: YCbCr chroma is centred at
(MAX + 1) / 2 (a u16 gray pixel had Cb = Cr = 128), HSV's S and V span
0..MAX with the hue over the whole range for one turn (S was scaled to
255; OpenCV has no HSV above 8 bits, so this extends its 8-bit
HSV_FULLscheme and is polars-cv's own), and Lab reads RGB / MAX (u16 mid-gray had L = 5393); -
signed integers are refused by
to_hsv,to_lab,to_ycbcrandconvert_colorto or from those spaces, which have no signed range (RGB, BGR and gray conversions accept every dtype). u8 keeps OpenCV's 8-bit conventions; its values move by at most one unit where the math, now f64 throughout, rounds a tie differently. -
Aspect-preserving resizes derive sizes exactly, and never as 0.
resize_to_height,resize_to_width,resize_max,resize_min,resize_scaleandletterboxcomputed the derived size in f32, which landed some exact halves just below .5:resize_to_height(21)of a 14x31 image gave width 46 where 7.5 and 1.5 round up. They also had no floor, soresize_max(1)of a 3x1 image was 1x0,resize_scale(0.25)of a 1x1 image 0x0, andletterboxof a 1x3 image into 1x1 fitted the content to zero rows and returned only padding. A derived size is nown * t / drounded half up in exact integer arithmetic, and at least 1, at plan time and at execution (one rule,shape_rule::scaled_size, which letterbox's fit also reads). Sizes change only for those inputs. -
relu,clamp_minandclamp_maxpropagate NaN. They usedf32::max/min, which return the other operand for a NaN, sorelu(NaN)was 0 andclamp_min(NaN, 0.5)was 0.5, whileclamp,abs,signandroundpropagated it. NaN in now gives NaN out from every scalar op, as in NumPy (np.maximum) and PyTorch. Replace NaN first (fill_nan) where a bound was wanted. -
The ordering reductions propagate NaN too.
reduce_max/reduce_min(whole-buffer and per-axis) return NaN when any input is NaN, andreduce_argmax/reduce_argminreturn the first NaN's index, as in NumPy. They used to compare with>/<, so whether a NaN survived depended on its position.reduce_percentileof input that contains a NaN is NaN. Before, it sorted with a comparator that is not a total order, so the engine panicked ("comparison function does not correctly implement a total order"), which the deep parity chain hunt hit throughsqrt. -
Every op that orders values propagates NaN, from either side. Binary
maximum/minimumgavemaximum(NaN, 1) = 1butmaximum(1, NaN) = NaN;dilate/erodekept a NaN centre but skipped a NaN neighbour;morphology_gradientturned NaN into 0; MinMaxnormalizeskipped NaN (ZScore propagated it); HSV'smax(r, g, b)skipped a NaN channel. They, the scalar clamps and the reductions above now read one rule,ops::util::displaces(maximum/minimum): a NaN anywhere is the result. -
Non-finite geometry is refused, not warped. A NaN or infinite
rotate_and_scaleangle, centre or scale,rotateangle orwarp_affinecoefficient passed the singularity check (NaN < epsis false) and warped every pixel to the border; a NaN, infinite, zero or negativeresize_scalefactor became a 1-pixel orusize::MAXaxis. Each is now refused — at plan time for a literal, per row for a column — by one matrix rule,AffineParams::matrix_is_invertible(finite and non-singular), and a finite-positive factor rule. -
histogramfollows NumPy for NaN. A NaN pixel is in no bin. It is not counted,normalized/bucketsshares sum to 1 over what was counted, and itsquantizedindex is one past the last bin (np.digitize). Before, it was counted in bin 0 with equal-width bins and panicked the engine with explicit edges. An auto-detected range over NaN or infinity, or a suppliedrangethat is not finite, now raises "... is not finite" as NumPy does; before, auto range silently skipped NaN. Explicit edges must increase monotonically, checked at plan time. -
histogramdrops values outside its bins, as NumPy does. A value below the first edge or above the last — of an explicitrangeor of explicit edges — was clamped into the end bin, sorange=(50, 200)counted every pixel. It is now in no bin, like NaN: not counted, and itsquantizedindex is one past the last bin. Pass infinite outer edges to keep everything (bins=[-inf, 50, 200, inf], see Added). Tied edges follownp.histogram([0, 1, 1, 2]puts 1 in the last bin left-closed, the first right-closed); a binary search over the tie used to pick either. Arangewhose min exceeds its max is an error. An auto-detected range still holds every pixel: its ends widen outward where a 64-bit integer rounds inward to f64. -
NaN propagates through
label_reduce, and geometry refuses non-finite coordinates.label_reduce(reduction="max")folded withf64::max, which drops a NaN, so a region holding a NaN pixel scored its largest number; it now scores NaN, asmeanandsumalready did (the whole-image statisticsreduce_sum/mean/std,normalizeandadjust_contrastalready propagated). A contour, point or bbox with a NaN or infinite coordinate is now an error naming the row, as a null coordinate already was, in every geometry function, thecontoursource andlabel_reduce: a NaN vertex panicked the plugin inconvex_hull,iou,diceandsimplify, madehausdorff_distance1.8e308, and left the bbox, winding and convexity finite but wrong. -
Mixed dtypes promote as NumPy's do, and 32/64-bit integers promote to f64. Two operands of different dtypes met in "the larger integer", which lost values:
u8200 plus ani8came backi8,u64withi64wasi64,u32withi16wasu32, and ani64with anf32was computed inf32. They now meet in NumPy'sresult_type(DType::promote, derived from the lossless-containment rule):u8+i8isi16,u32+f32isf64, andu64with a signed integer isf64— where a bitwise op is refused, there being no common integer. The float-promoting ops (scale,sqrt,divide,convolve2d, gamma, the scalar math family, ...) givef64foru32/i32/u64/i64input, computed from the exact value; they gavef32, reading each pixel as f32 (2**24 + 1became2**24), and convolve and divide computed in f64 only to store f32. 8/16-bit integers still givef32. One rule decides both what an op computes in and what it stores (OutputDTypeRule::PromoteToFloatresolves toDType::accumulator). Mixed operands are converted a block at a time rather than each copied whole. -
polars-cv requires
polars>=1.43.2(was>=1.41.1). Older polars exports a slicedArraycolumn that has nulls across the plugin FFI with its offset applied twice (theFixedSizeListArrayexport reported the validity's offset while its values were already sliced), so the plugin's import panicked: anarraysource over two chunks, or one chunk split into streaming morsels, failed with "the offset of the new Buffer cannot exceed the existing length". Fixed in polars itself; 1.43.0 and 1.43.1 are yanked. The floor's fast suite was run at polars 1.43.2 / numpy 2.0.2. -
divideis IEEE true division;ratiois removed.dividealready promoted integer operands to float (f32, orf64with anf64operand) and divided, but a zero divisor gave0, even for floats, and its docstring promised integer division for u8/u16. A zero divisor now follows IEEE 754, as NumPy'strue_divide:x / 0isinf(-inffor negativex) and0 / 0isnan. Writea.divide(b)and replace non-finite values afterwards if0was wanted.ratiowas documented as(a/b)scaled to the dtype's range but executed exactly asdivide; it is deleted rather than kept as a second name. Usedivide(and.scale(255)for the documented scaling). -
extract_contourstraces along pixel edges, not pixel centres. Pixel(x, y)is the unit square[x, x+1] x [y, y+1], and every extracted outline runs along those squares' edges, so it bounds exactly its region's pixels. The centre-traced outline sat half a pixel inside its region: - A
w x hblob came back bounding(w-1) x (h-1); it now comes backw x h. - A single pixel came back as a one-point contour and a
1 x nline as a segment, both with area 0; they are now the unit square (area 1) and a1 x nrectangle. mask -> extract_contours -> rasterizeeroded the mask by a pixel per round trip; it now returns the mask unchanged.
Vertex coordinates, area, perimeter, bounding_box and IoU of
extracted contours all change accordingly, and so does min_area, which
now filters on the pixel count. Connectivity is unchanged: regions are
8-connected, so pixels touching only at a corner are one region, whose
outline passes through that corner twice. mode="external" is now decided
from the region labelling rather than a quadratic point-in-polygon scan.
Added¶
- Open-ended histogram bins. Explicit edges may start at
-infand end atinf, as polars'histbreaks do:histogram(bins=[-math.inf, 0, math.inf])splits every value at 0. The wire carries an infinite edge as"inf"/"-inf"(JSON has no infinity), through aboundcatalogue type (naming::Bound) that only histogram edges have, so no other float parameter can receive an infinity it does not define.
Performance¶
-
Broadcasting two-buffer ops are ~29x faster. An operand broadcast to the other's shape (the
[H, W, 1]maskapply_maskblends into an[H, W, 3]image, or any binary op over differently shaped inputs) was read by decoding every output element's coordinates afresh, allocating aVecand dividing once per axis per element. It is now read by stepping through the output and carrying the source index along: 1024x1024 RGB under a mask drops from ~104 ms to ~3.6 ms. Results are unchanged. -
Kernels use AVX2 on CPUs that have it, whatever the wheel was built for. The published wheels target the x86-64 baseline (SSE2), where much of the engine's arithmetic could not vectorise. One runtime dispatch mechanism now compiles a kernel's whole body a second time with AVX2 and picks it per call. Only AVX2 is enabled, never FMA, so the output is byte-identical on every CPU. Blur (the one kernel that already dispatched) moved onto it.
- Float → integer casts are ~2x faster on the wheels.
castand a fused scalar chain's integer output share one conversion rule and one dispatched loop; on SSE2 rounding was aroundfcall per element (f32 → u8 at 1024²: 9.1 → 5.1 ms). - u8
grayscaleis 5–10x faster and u8threshold1.4–3x. Both are vectorised kernels over the image's rows. A contiguous, cropped or vertically flipped input is read where it lies instead of being copied first. Non-u8 grayscale is 1.4–2.9x faster. - Per-value ops run through one engine that picks how per call.
invert,adjust_gamma,adjust_contrast,normalize,scale,clamp, the scalar math ops and fused chains of them share one element-wise engine, and results are unchanged. Work that cannot vectorise (apowfper pixel, as in gamma) or that differs per channel (presetnormalize) runs once per possible value of an 8-bit input (16-bit once an image has 65,536 values per table), and each pixel becomes a table read: u8 gamma is 22–27x faster and u8 ImageNet-stylenormalize5–12x. An integer map such asinvertruns in integer arithmetic (1.3–1.9x). Everything else streams vectorised, in cache-sized blocks for an integer result, with no image-sized f32 intermediate (u8adjust_contrast2–5.5x, z-score 2.2–4x, a fused u8 → u8 chain 1.8–2.5x). An op whose input nothing else reads writes it in place when the output dtype is the input's (u8invert, u8 → u8 chains, u8threshold). - Converting floats to 8/16-bit integers vectorises. Rounding half away
from zero and saturating is now written so it compiles to vector
instructions, with identical results: f32 → u8
castis another 1.6–2.3x faster on top of the previous entry. - Materialising a strided view copies runs, not elements. A crop, flip or
transpose used to be copied a few bytes at a time with an N-d offset
recomputed per copy. One walk now coalesces the layout into the longest
packed runs it has (a whole buffer, a row, a pixel) and copies each run
once. This covers
to_contiguousand every list/array sink. Acastof a view converts straight from those runs, with no packed copy first, and so does a per-value op's read of a strided view. At 1024² RGB u8 on the wheels, materialising a vertical flip is ~22x faster, a horizontal flip ~5x, a transpose ~5.5x, grayscale of a flipped image ~4.7x, crop then resize ~4.5x and a cast of a flipped view ~3.4x. - Resize reads a crop or a vertical flip where it lies. It used to pack any view into a new image first. Rows packed within themselves, whatever their spacing or order, now go to the resizer as they are, so the output is the only image-sized allocation. At 1024² RGB u8 → 224² on the wheels, crop then resize is 1.3x faster and a vertical flip then resize 1.5x. Output is byte-identical. Other layouts (a transpose, a horizontal flip) are still packed first, as before.
grayscaleof a crop or vertical flip is no longer copied first. The u8 grayscale entry above already said so, but the planner packed such a view before the kernel could read it. u8 grayscale after a vertical flip is 1.65x faster at 1024² on the wheels (CR-57).- Cheap rows cost about half as much. A buffer's shape and strides used
to live on the heap, so copying a buffer or asking whether it was
contiguous allocated, several times per row:
inverton an 8×8 image made 27 allocations per row and spent more than half its time in the allocator. Layouts of up to four dimensions are now stored inline, a cached plan is replayed without copying it, and row results are no longer gathered into one call-sized vector before the column is built.inverton 8×8 rows is 1.3x faster eager and 1.4x streaming (CR-58). - The plugin allocates through polars' allocator. It used the system
malloc, which gave a call's freed row buffers back to the OS for the next call to fault in again. It now shares polars' allocator (jemalloc on Linux), as pyo3-polars intends: 64×64 pipelines are 1.15–1.66x faster, a repeated largeblobsink 2x faster, and peak memory ~10% higher (CR-60). erode/dilatewithiterationsandblurare faster, with identical output: integer images erode or dilatentimes in one wider pass (1.1–1.7x for 3–4 iterations; float images keep the passes, which NaN and signed zeros depend on), and blur converts its input a row at a time instead of copying the whole image to f32 first (1.1–1.3x).- Rotation and
warp_affineare 1.4–2.2x faster, with bit-identical output: the warp takes a bounds-check-free path for pixels whose neighbours are all inside the image and is compiled per channel count and for AVX2. A rotation by 0° (or 360°) now returns its input instead of warping it. Listcolumns are read in place. Alistsource row whose values are already the declared dtype is now a view of the column, as anArrayrow already was, instead of a per-row copy through several intermediate Series: 3.6x faster for 64×64 u8 rows, 2x for flat f32 rows, and a row of another dtype is converted in one pass (1.5x). Raw and blob rows are read in place whenever they are aligned for their own dtype, rather than only at 8-byte boundaries.- Every per-value op,
thresholdandgrayscaleread any view where it lies. Whether a crop, flip or transpose was copied first used to depend on the op, the dtype and which strategy ran it: u8invertorgammaof a horizontal flip packed it,normalizeandadjust_contrastwere always packed by the planner, and grayscale read a crop in place for u8 but not for u16. One traversal now runs all of them, reading a view in the runs its layout has and writing into place when nothing else holds the buffer. The output is the only image-sized allocation, and results are unchanged. On the wheels: per-value ops on a flipped or transposed view 1.0–1.2x, non-u8 grayscale 1.5–2.1x, u8 presetnormalize1.5–1.9x, u8adjust_contrast1.1–1.5x, an f32 scalar chain written in place 1.1–2x (CR-64). - The half-precision tensor sink is 3–6x faster and runs in parallel.
sink("numpy"|"torch", dtype="f16")converted every row one element at a time after all rows were computed, on one thread. Each row is now converted on its own thread, in one pass with the CPU's F16C instructions when it has them. The bytes are unchanged (CR-65).
Changed¶
ContourMatcher(min_contour_area=)defaults to0.0, down from1.0. The1.0default dates from the boundary tracer that collapsed regions into degenerate walks. Since the smallest extracted region now has area 1, both values keep every region;0.0says so directly.ContourMatcher(gt_min_contour_area=)defaults to1.0and no longer followsmin_contour_area(Noneis no longer accepted). A caller who set onlymin_contour_areanow gets1.0for ground truth; passgt_min_contour_areaexplicitly to change it.- Detection metrics move for any
ContourMatcherevaluation. Regions are measured on their full pixel extent, so IoU against ground truth rises, most for small lesions. Under the defaultiou_thresholdsome former misses become true positives, and one-pixel-thick detections now count as false positives. source("list", require_contiguous=True)accepts rectangularListrows. It refused everyListrow, although it documents refusing only jagged ones; rectangular rows are now read in place. A row that would need converting to the declared dtype is still refused, as forarray.normalize(method="zscore")computes its mean and standard deviation exactly. They used to be running f32 sums, which drift at image sizes; 8/16-bit images now use exact integer sums and other dtypes f64 sums. The normalized values can differ from 0.29.0 in the last bits.
Internal¶
- One declaration that 2- and 4-channel images are gray + alpha and RGBA
(
ops::color::has_alpha), read by the resamplers' premultiply (fir's is now set from it explicitly rather than by its pixel-type default), the shape rule's carried alpha, grayscale and the colour conversions; a structural guard (test_alpha_authority.py) refuses a restated channel test. - tests: the parity generator draws NaN/infinity content for float images,
64-bit integers beyond 2**53, and sweeps every op on empty images
(
invariance/test_parity_empty.py: refused or processed, never a panic). What it found is fixed above; the references it showed to be inexact for 64-bit integers (threshold, blend, grayscale's bound) now compare exactly, the YCbCr reference expands its chroma so an infinite channel gives its limit, and the morphology ordering law skips NaN positions. - tests (review follow-ups): the dtype sweep fails an op its contract admits
on no swept layout (it had never swept
reshape, whose sample fitted no layout);rotate_and_scale's lowering and the binary float ops are checked against independent references (geometry; f64 rounded once and exact integers), not restatements of the kernel; the parity harness keys the quarter-turn rule onrotateitself and keeps literal-only list forms (histogramedges) literal on every parameter axis; the u8 resample bound against Pillow is 2 for every filter, as two correct fixed-point resamplers round to opposite sides of the exact value. - tests: a generative parity suite,
tests/parity/(Hypothesis). Every chainable op runs against an independent reference (NumPy, OpenCV, Pillow, SciPy) over drawn dtypes, sizes, channel counts, pixel content, null rows, mixed-size rows, sources and engines, with per-row expression arguments. Random chains and two-branch graphs are checked step by step, each step against its reference on the engine's own previous output, and end to end with the error bound carried through each op's gain. The same case must agree byte for byte across every source, sink, engine, parameter style (literal,pl.col,pl.lit, computed), composition (one pipeline,.pipe(), aliased prefixes, blob-materialized steps), optimizer setting (all, none, and each registered pass alone and removed) and chunking. Algebraic laws cover inverses, commutation, resampling identities, morphology ordering and the lossless contour round trip. Ratchets hold its tables to the op, I/O and dtype catalogues. Each property runs a small derandomized budget per push and a 25x randomized one in the weekly slow lane. - tests: divergences the suite found are registered in
tests/parity/framework/known.py, each pinning the exception and message its repro fails with (known.still_reproduces), so a repro broken for another reason no longer reads as the defect. Every divergence the suite found has since been fixed (see Fixed and Breaking changes) and its entry deleted; the registry is empty. - tests: the parity suite's binary ops are drawn on every dtype their
contract admits (
BinarySpecsplitsacceptsfromref_accepts, asOpSpecdoes); add/subtract/multiply/blend were drawn on u8, u16 and floats only, which is how the claim that they were exact on wide integers went unchecked. - tests:
tests/property/is folded intotests/parity/laws/. TheTestResizeScaleReference/TestAspectRatioResizeReferenceclasses intest_resize_enhancements.pyare removed: they resized with Pillow and asserted Pillow's output size without running polars-cv. The parity suite's resize entries compare polars-cv's pixels with Pillow's. - view-buffer:
ViewBuffer::try_apply_fused_kernel_inplaceis removed. Whether a kernel may write its input is decided in one place,ViewBuffer::unique_contiguous_mut, which the element-wise engine and the u8 threshold ask;apply_fused_kernelremains and never writes&self. - view-buffer: one walk over a view's memory,
core::strided::Walk, replaces the packing loop incopy_elements_into, the element-wise engine's strided f32 read and grayscale's per-pixel strided fallback (a non-dense layout is now packed, then takes the dense kernel). - view-buffer: the affine warp lives in
execution/warp.rs, and M5'sCastFrom<f64>for 8/16-bit targets isround().clamp() as T(the same values as before, in a form the compiler vectorises across a pixel's channels). - view-buffer:
LayoutandLayoutFactsstore shape and strides ascore::layout::{Dims, Strides}(inline up to rank 4), and the constructors (from_vec_with_shape,reshape,from_polars_buffer*,new_contiguous) takeimpl Into<Dims>, so existingVecarguments still work.ExecutionPlan::execute_stepsreplays borrowed steps. - view-buffer: the resizes (
resize,resize_scale,resize_to_*,resize_max/min,letterbox) andgrayscaledeclareMemoryEffect::StridePreserving, so the planner no longer inserts aMaterializeContiguousbefore them. They read their input throughinterop::fir::FirViewAdapter, theExternalLayout::FastImageResizeadapter, which accepts any layout with packed rows (LayoutFacts::is_dense_rows), and one genericresize_pixels::<P>replaces the u8/u16/f32 copies of the resize kernel. - view-buffer:
core::mapis the one traversal of a per-value (ElementMap) or per-pixel (PixelMap) kernel. The element-wise engine's strategies,cast,threshold,grayscaleandto_f16_bitsare maps; their per-kernel in-place / into / pack code,map_pixel_rowsand the typed threshold and grayscale kernels are deleted. The engine's f32 "pass" strategy is merged into "blocked".normalizeandadjust_contrastdeclareMemoryEffect::StridePreserving.Walk::for_each_runtakes aRunSinkand a grain (runs never split a pixel), andclamp_for_dtypeis test-only (the warp and blur oracles). - view-buffer:
ViewBuffer::to_f16_bitsreturns a buffer's elements as IEEE-754 half-precision bit patterns (aU16buffer);halfis a view-buffer dependency. polars-cv'sNumpyRowOutput::from_buffer_f16is replaced byfrom_f16_bits.
Fixed¶
- An empty image is refused cleanly, not a panic.
reduce_argmax/reduce_argminover an empty axis panicked the engine, andpadwithmode="edge","reflect"or"symmetric"panicked extending an empty axis; both are now refused by their contracts, as NumPy refuses them (padding only the other axis of an empty image gives an empty image, as NumPy's), as arereduce_max/min/percentileover nothing (sum,meanandstdstay defined). AnArraycolumn with a zero-size dimension is refused at planning, naming the column: polars' FFI import of one sets its length to 0 and then slices it to the real row count, which panicked the plugin. convert_color(rgb -> gray)isgrayscale(). BT.601 was written out in five places, andconvert_colorcomputed it in f32 for i8, u16, i16 and f32 wheregrayscalecomputes in f64 and rounds once, so the two disagreed (f32 in the last bit). Both now run one luma (ops::color::Luma), as does the YCbCr encoder.- Histogram edges meet integer pixels exactly. A u64/i64 pixel was
compared with an edge through f64, so
2**53 + 3landed in the bin above the edge2**53 + 4. Histogram binning now reads the threshold's exact integer cut (core::cut, the one pixel-against-boundary comparison). - Resizing an empty image is refused, not a panic. An aspect-preserving
resize (
resize(height=…),resize_max,resize_min,letterbox) of an image with a zero height or width divided by it while deriving the other side and panicked the engine. Every resampling op now refuses an empty image in its contract (at plan time when the size is known, per row otherwise), and the shape rule has no output for one. -
ViewBuffer::reshaperefuses a strided view in every build. The check was adebug_assert!in the runner, so a direct caller in a release build got elements in memory order rather than view order. -
8-bit HSV hue stays in [0, 180). A hue in [359°, 360°) rounded to 180 (
RGB(255, 0, 1)-> H = 180), outside the range; it wraps to 0, as OpenCV's does. Integer hues wrap at their period on every dtype. - A gray + alpha image round-trips through TIFF. The
tiffsink expanded it to RGBA, so it came back as four channels, and a two-sample TIFF from another writer (Pillow'sLA) was refused as "Unsupported TIFF color type". It is now written as two samples with the second declared alpha (ExtraSamples), and decoded as two channels. - A histogram counts a value on a bin edge in the bin its edges give.
The uniform-bin index was
floor((x - lo) / width), which for a value exactly on an edge can land a bin low: -7142.857142857143, edge 1 of [-10000, 10000] in 7 bins (and so whatoutput="edges"reports), was counted in bin 0. The edges now decide membership for bothclosed="left"and"right"(whose former epsilon test is gone). - The
scalar_fusionpass no longer changes an integer chain's result. It fused an integerinvertwith a following narrowing integercastinto one kernel that stores by the float -> int rule (round, saturate) instead of the int -> int rule the unfused cast applies (wrap): u16invert().cast("u8")of 1000 was 255 fused and 23 unfused. Every pass is byte-identical on and off again.DType::converts_like_floatis now the one rule bothscalar_fusionandcast_chain_collapseread before replacing a conversion with a float one (and it letscast_chain_collapsedrop an f32 intermediate between integers where that is exact, such as u8 -> f32 -> u16). - A blob or raw column sunk to
numpy/ndarray/torchreturns every row, not row 0's pixels for each. Rows read in place are slices of the column's shared buffer, and the numpydatacolumn registered those slices; Arrow's FFI export hands over a buffer's storage pointer, so every row was read from the start of the shared storage (raw rows: from the wrong place). The column now registers each storage once, unsliced, and addresses each row by its offset in it. reshapeandchannel_selectafter a crop read the crop. A crop that starts below the first row (or right of the first column) leaves a view with an offset; a reshape rebuilt its layout at offset 0, socrop(top=1).channel_select(0)of a single-channel image returned the uncropped rows. Reshape keeps the view's offset.reshapeafter aflip,transpose, quarterrotateorcropcopies first, as NumPy's does. It was refused at execution ("cannot reshape a non-contiguous view"), not by the planner, with no way to ask for the copy. Reshape now declaresMemoryEffect::ViewOfContiguousand the planner packs a strided input before it: a no-op when the input already is contiguous.channel_swap,channel_merge, RGB <-> BGR, gray -> RGB and the morphological gradient are exact on every dtype. They had native paths for u8 (and some u16/f32) only and converted every other dtype through f32:channel_swapandchannel_mergethen panicked against their contract ("expected output dtype U16 ... but got F32"), and the colour moves and the gradient silently rounded values f32 cannot hold (u32 16777217 -> 16777216). Each is now one kernel generic over the element type (with_dtype!);channel_swapand the colour conversions' reorder are the same kernel, where they were two.- Resizing, blurring and
rgb->graykeep 32/64-bit integer and f64 images at full precision. These dtypes were cast to f32, resampled and cast back, so any value f32 cannot hold changed, even through a nearest resize that only moves data (u32 16777217 -> 16777216). Nearest resampling is now an exact gather on every dtype, and the convolution resamplers, blur and the luma compute inDType::accumulator(f64 for these dtypes). fast_image_resize has no 64-bit pixel types, so those dtypes are resampled byexecution::resample, which uses fast_image_resize's own filters and coefficient rule in f64 (tested against it: byte-identical for nearest, within 1e-4 for the convolution filters). u8, u16, f32, i8 and i16 are unchanged. - The two-buffer ops are exact on every dtype, and
blendis normalized on every integer dtype.add,subtract,multiply,blend,maximum,minimumand the bitwise ops ran natively only for u8 and u16 (and same- dtype floats); every other operand pair was computed in f32, so a value f32 cannot hold changed (u3216777219 ^ 16777221gave 0, not 6), andblendwas a plain saturating multiply (i881 blend 81gave 127). One kernel generic over the element type now serves every dtype: integer arithmetic saturates to the dtype's range,blendisround(a*b / MAX)(the normalized product u8/u16 already used), and floats use IEEE arithmetic. True division computes inDType::accumulator. - Colour conversions refuse a channel count they cannot read. They
admitted any
Cat least the source space's channel count, but every kernel reads a pixel as that many values:to_bgrof a 5-channel image reordered the first three channels and returned the other two as zeros, and the other spaces read across pixel boundaries. A conversion now admits exactly the space's channels or those plus alpha (3 or 4 for RGB, BGR, HSV, Lab and YCbCr; 1 or 2 for gray), at plan time when the count is known and per row otherwise. convolve2d(andsobel,laplacian,sharpen) on f64 returns f64. It planned f64 and executed f32, which the output guard refused. A convolution now accumulates inDType::accumulator— f64 for f64 and the 32/64-bit integers, f32 otherwise — and stores its declared dtype.- A literal float parameter equals the same value given as an expression.
The graph JSON was parsed without
serde_json'sfloat_roundtrip, so a literal could land an ulp away from what the caller wrote, while the same value from a column was read exactly. Any f64 parameter could differ; the parity suite caughtwarp_affine's matrix (the last bits of an f64 or wide integer image differed betweenmatrix=[...]and per-row expressions) andthresholdnear a u64 pixel. Literals now parse to the nearest f64. rotate_and_scalewith a per-row angle, centre or scale matches the same values given as literals. Its matrix was computed in Python: through therotation_matrix_2dFFI for literals, but as polars expression arithmetic for any expression argument, and polars'radians()(x * (pi/180)) rounds differently from the engine'sx * pi / 180, so the two disagreed in the last bits of an f64 image.rotate_and_scaleis now a typed op the engine lowers per row throughAffineParams::rotation_matrix_2d, so there is no second copy of the trig; its signature and outputs are unchanged. Thepolars_cv._lib.rotation_matrix_2dFFI, which existed only to feed the Python copy, is removed.thresholdcompares integer pixels exactly. A non-u8 integer pixel was rounded to f64 before the comparison, so above 2**53 it answered for a neighbour (2**53 + 1 > 2**53came out false). Every integer dtype now comparesp > floor(t), as u8 already did.rotate/warp_affinefill pixels off the image with the border value rounded and saturated, like every other pixel. The fill truncated it: a u8border_value=7.5filled 7 while pixels at the edge blended toward 7.5, and a value the dtype cannot hold (300 for u8) filled 0. It now fills 8 and 255. Integer border values inside the dtype's range are unchanged (CR-66).ContourMatcherno longer silently drops one-pixel-thick regions. Two causes, both fixed:- Traced through pixel centres, a single pixel, a two-pixel pair or a
1 x nline had no area, so the defaultmin_contour_area=1.0dropped it. Even kept (min_contour_area=0), it overlapped nothing, so it could never be a true positive. label_reduce(bothPipeline.label_reduceand.contour.label_reduce) returned 0.0 for a zero-area contour in every region mode. Its zero-width or zero-height bounding box returned early, before the documented "no pixel centre" fallback ran, soContourMatcherremoved the region as a zero-score detection.
Edge tracing gives every region an area equal to its pixel count, so thin regions are now scored, measured and matched on their own pixels: over a matching lesion they are true positives. A contour supplied with no area, or whose region catches no pixel centre, is now scored on the pixels its outline passes through, in every region mode. That rule replaces the centroid fallback sub-pixel contours had.
rotate(0)no longer spreads NaN or infinity. It ran a full bilinear warp, which turned a NaN or infinite pixel's left and upper neighbours into NaN. It now returns the image unchanged (CR-63).-
A warped u64/i64 image keeps its maximum values.
rotateandwarp_affinestored a value at the top of the 64-bit range as 0; it now saturates, as every other conversion does (CR-62). -
A
list/arrayvalue the declared dtype cannot hold is an error, not 0. Converting a row to its declared dtype stored 0 for any value out of range (300or-1asu8) or undefined (NaN). It now fails naming the value, or nulls the row underon_error="null". Float values declared as an integer dtype are still truncated (CR-61). -
sink("blob")of a whole-row crop wrote the rest of the image after it. A crop that keeps every column is a contiguous view of part of the image, and its blob's payload was the whole image's length from the crop's first byte on: too long, and for a crop below the top row a read past the end of the image's memory. The blob now holds exactly the crop (CR-59). -
grayscaleof a non-u8 gray + alpha image mixed the alpha into the intensity. A[H, W, 2]u16/f32 pixel came out as0.299·gray + 0.701·alpha(u16 (1000, 65535) → 46239). It is now the gray channel, as it always was for u8 (CR-51). inverton an integer dtype other than u8/u16 failed. i8/i16/u32/i32/ u64/i64 input computed1 - xas f32 against a schema promising the input dtype, so the query raisedplanned dtype … but execution produced F32. Every integer dtype now inverts asMAX + MIN - xin its own dtype:255 - xfor u8 (unchanged),-1 - xfor a signed dtype, which is NumPy's~x(CR-53). Signed 8/16-bit invert now also fuses with neighbouring scalar ops.
0.29.0 — 2026-09-27¶
Every operation, source, sink and geometry accessor is now one typed Rust definition from which its Python method, docstring and wire form are generated, and the planner that computes a pipeline's schema runs in Rust over the same definitions execution uses. Plan-time optimizations are one explicit, toggleable phase. Many silent wrong results are now errors. Upgrading from 0.28: see the migration guide.
Breaking changes¶
Each item is described in full under its section below.
- Builder signatures. One rule for every generated method: an op's only
required parameter is positional-or-keyword, every other parameter is
keyword-only (e.g.
.normalize(method="minmax")).source()keywords default toNone.perceptual_hash's parameters are keyword-only. - Removed parameters and methods.
convolve2d(ksize=),sobel(ksize=),laplacian(ksize=);source()'s contour canvas keywords (width,height,shape,fill_value,background— write.rasterize(...));Pipeline.output_encoding();PipelineGraph.set_root_column()(to_graph(column)requires its column);PlanState.DIM_NAMES;_graph.LIB_PATH; theaffine_fusionoptimization pass. - Changed defaults and results.
.contour.scale()scales about the centroid;.point.interpolate(t=)is keyword-only;cannymatchescv2.Canny(more edges, no pre-blur); an image with no contours yields[], not null; a nullnumpy/torch/ndarraysink row is a null value;output_dtype()can return"auto_float";PipelineGraph.to_expr()needs an optimized graph (.sink()produces one). - Stricter input checks.
cropwindows past the edge or with negative values,rawbyte lengths that are not whole elements,reshapes that change the element count, image ops on anything but[H, W]/[H, W, C], shorttransposeaxis lists, jagged or nullListrows, null contour coordinates, contour structs without anexteriorfield,sobel(axis=)other than"x"/"y", per-row"triangle"/"grey"enum aliases, andlist/arraysink rows that do not match the plan are all refused. - Hand-built graph JSON. Fields follow the typed definitions; an unknown,
misspelled or inapplicable field is rejected by name, as are the old
alias/domain/output_dtypenode fields and theexpr_column_nameskwarg. - Environment and dependencies. Requires
polars>=1.41.1;networkx/graphviz/pydotmove to thevizextra. The single-thread warning andPOLARS_CV_ENGINE_WARN_SECONDS/POLARS_CV_ENGINE_WARN_ROWS/POLARS_CV_SILENCE_ENGINE_WARNINGare gone.
Added¶
- A plan-time optimization phase you can see and control. Building a
pipeline records the graph as written;
.sink()then applies the optimizations in one phase (PipelineGraph.optimize(flags)). Every pass —common_subexpression_elimination,identity_elimination,spatial_window_pushdown, and the engine rewritescast_chain_collapse,cast_identity,view_flip_involution,view_transpose_mergeandscalar_fusion, which used to run unconditionally — has anOptFlagsfield (polars_cv.OptFlags, withOptFlags.all()/OptFlags.none()). Choose them per call with.sink(opt_flags=...)or for the process withPOLARS_CV_OPTIMIZATIONS; all are on by default.Pipeline.explain()shows the chain as written, or withoptimized=Trueafter the passes.sink()would apply (from the same registry, so it cannot drift from execution). Every pass is held to byte-identical output by an on/off differential intest_optimize_equivalence.py. identity_eliminationpass. Deletes operations that are provably no-ops so they never reach the engine: a zero pad (pad(0, 0, 0, 0)), a same-dtypecast, and a full-framecrop(or apad_to_sizeto the current size). A crop is a candidate only with a literal(0, 0)origin (an offset crop the size of its input runs past the edge). A blob's declareddtypeis checked at decode, so the pass never removes a.cast()on the strength of a wrong declaration.spatial_window_pushdownpass. Hoists an H/W crop earlier in a pipeline, past the run of pointwise ops immediately before it, so they process only the window. A neighbourhood, geometric or global op, anassert_shape, and an op that reads another node (apply_mask,channel_merge, binary ops) are barriers.numpy_from_column(column)reads a numpy/ndarray sink column without copying. Every row comes back as a read-only NumPy view of the column's own Arrow memory, keeping its shape, dtype and byte strides; null rows areNone, andcopy=Truereturns owned, contiguous arrays. Reading rows as dicts fornumpy_from_structcopies every row into Pythonbytesfirst (and again with its defaultcopy=True). A view that would reach outside its row's bytes (only a hand-built struct can ask for one) raises.- The planner tracks every dimension, at any rank. A pipeline's planned
shape is one size per dimension rather than three
[H, W, C]slots, so every output whose sizes are all known publishes its shape — a rank-1 or rank-2 output (channel_select,reshape,assert_shape(dims=[h, w])) and a rank-4 one included — and.sink("array")needs noshape=for it. A fixed-sizeArraycolumn's dtype states every size, sosource("array")(orautoover anArraycolumn) publishes the whole shape too; aListcolumn's sizes still vary per row.assert_shape(dims=[...])accepts any rank (it refused more than three).PlanState.dimshas one entry per dimension (the leading declared ones over an unknown rank), andPlanState.DIM_NAMESis gone. source("contour")without a canvas decodes to the contour domain. A pipeline can start from a contour column and measure or transform it (Pipeline().source("contour").area(),.simplify(...), then.rasterize(...)or a native sink). It used to raise: the source was a rasterization.- Arrow extension types:
polars_cv.ndarray,.point,.contour,.bbox.NdArrayType,PointType,ContourTypeandBBoxType(inpolars_cv.extension_types, re-exported frompolars_cv) tag the structs polars-cv already emits, so a column is identified by its type rather than by sniffing field names, and keeps that identity through Parquet/IPC. They are registered atimport polars_cv(pure Python — importing still loads no compiled code), so a tagged column read from Parquet keeps its tag. An instance of one of these classes always has the canonical storage: a column carrying the name over any other storage (or with metadata) reconstructs as polars' genericExtensioninstead. The names and storages are declared once in Rust (ext_types::ExtType) and held to the Python classes by a parity test over the newextension_typesFFI. Polars documents its extension-type API as unstable; polars-cv uses it only inextension_types.pyandsrc/ext_types.rs. sink("ndarray"). The numpy sink struct, taggedpolars_cv.ndarray— same rows, bytes, strides anddtype="f16"downcast.numpy_from_structandshow_images(underformat="auto") read it.sink("numpy")is unchanged: polars' struct operations (.struct.field,unnest, casts,concatwith a plain struct) do not see through a tag, so tagging the existing sink would break code that uses them. Use.ext.storage()for those on a tagged column.- Every expression accepts tagged inputs. Each
.cv/.point/.contour/.bboxaccessor and everyvb_graphsource (e.g.source("contour")) takes a tagged column wherever it takes the plain struct, and computes exactly the same result. Geometry outputs stay plain structs. - An op, source or sink field with a declared default may be omitted from hand-built graph JSON; the wire applies the same default the Python signature shows.
- Benchmarks build in a fraction of the time, and run only what a change
can move.
maturin develop --profile benchmarkis release with thin LTO (the fat-LTO link was most of a 10–18 minute release build). The regression suite's--select scenario[:glob]runs a subset of cases by their result names, and--changed REF(orpython -m benchmarks.regression.relevance REF, to pass the same--selectto the base side) derives it from the files a change touches, through a path → cases map that a guard holds every tracked source file to. A newtargetedscenario times what the adapter scenarios never reach: the geometry accessors, thearray/listsinks, JPEG and re-encoding, and the blob source.run_suite --scenariosis replaced by--select(a bare scenario name is a selector). - The benchmark suite refuses a debug build. Its numbers look plausible and
measure nothing;
run_suiteexits unless given--allow-debug-build(a harness smoke test), andcomparerefuses results marked as such. It also refuses results from two different build profiles (thin LTO runs a few percent off fat LTO per case). The extension reports_lib.__debug_assertions__and_lib.__build_profile__for it.
Changed¶
- Breaking: one signature rule for every generated builder method. An op
with exactly one required parameter takes it positional-or-keyword; every
other parameter is keyword-only. Now positional:
adjust_contrast(factor),adjust_gamma(gamma),channel_select(index),channel_swap(order),simplify(tolerance),label_reduce(contours),convolve2d(kernel). Now keyword-only (pass by name):convert_color(from_space=, to_space=),histogram(bins=, range=, closed=, output=),normalize(method=, mean=, std=, out_dtype=),reduce_max/mean/min(axis=),reduce_std(axis=, ddof=),warp_affine(matrix=, output_size=)andperceptual_hash(hash_size=, algorithm=)— e.g..normalize("minmax")is now.normalize(method="minmax"). Names and defaults are otherwise unchanged. - Breaking:
Pipeline.source()keywords default toNone.require_contiguousandon_errorreadNone(the format's own default:False,"raise") instead of restating it, so passing a keyword is exactly passing a non-Nonevalue —on_error="raise"orrequire_contiguous=Falseon a format that does not read them is now refused like any other inapplicable keyword.Pipeline.output_encoding()is removed (the executor reads histogram buckets off the ops). - Breaking: a contour source only decodes; rasterizing is
rasterize().source()no longer takeswidth,height,shape,fill_valueorbackground; writesource("contour").rasterize(...). A per-rowwidth/height/fill_valuethat is invalid for a row now fails the query (or nulls the node underon_null_param="null") as it does forrasterize();source(on_error=)covers only a contour value that cannot be decoded. - Breaking: the geometry accessors are generated from their Rust
definitions. Each
.contour/.point/.bboxfunction parses its own typed definition (src/geom_fns.rs) — or, for.contour.area,perimeter,centroid,bounding_box,convex_hull,translate,scaleandsimplify, the pipeline operation's — so its fields, defaults and docstring are declared once..contour.scale()now defaults toorigin="centroid", matchingPipeline.scale_contour()(it scaled about(0, 0));tin.point.interpolate(other, t=...)is keyword-only; and.contour.label_reduce(image)'simageis required in the signature (it defaulted toNoneand then raisedValueError). - Breaking:
cannycomputes whatcv2.Canny(image, low, high)computes, pixel for pixel. It ran a 5×5 σ≈1.4 Gaussian blur first, used an L2 magnitude and its own suppression rules, so it found 30–70% fewer edges than OpenCV on the same image. It now does OpenCV's algorithm — 3×3 Sobel with a replicated border, L1 magnitude, no pre-blur, OpenCV's fixed-point non-maximum suppression and 8-connected hysteresis — and a colour image takes the strongest channel per pixel, as OpenCV does, rather than its luminance. Add.blur(sigma=...)beforecannyfor the old smoothing.tests/reference/test_canny_ref.pyasserts exact equality withcv2.Canny, for u8 and for every other input dtype. - Breaking:
PipelineGraph.to_expr()needs an optimized graph..sink()runs the optimization phase and returns an executable expression as before; a graph fromto_graph()must go through.optimize(...)before.to_expr(), which otherwise raisesRuntimeError. - Breaking: an image with no contours yields
[], not null.extract_contours()published an empty contour set as null, the same as a null input or a failed row, solist.len()read null instead of 0 and the.contourtransforms (which keep[]) disagreed with it. Null now means only "no value" (a null input, or a row failed underon_error("null")). - Breaking: a null row of
sink("numpy")/"torch"/"ndarray"is a null value. It used to be a struct whose five fields were all null, sois_null()wasFalseanddrop_nulls()/null_count()did not see it, unlike every other sink. Code that testedrow["data"] is Noneshould testrow is None.numpy_from_struct(None)raisesValueError(CR-39). - Breaking: dependency metadata. Requires
polars>=1.41.1(was>=1.0, which could not import below 1.36.1 and failed the suite below 1.41.1) andnumpy>=2.0.2(was>=2.2.6).networkx/graphviz/pydotmove to avizextra (pip install 'polars-cv[viz]'), needed only byshow_graph(). Wheels are taggedcp310-abi3to matchrequires-python>=3.10. A new CI job tests the declared floors on Python 3.10. (CR-44) - Every
.cv.pipe(...)call runs its rows in parallel. Rows are split into ranges on the plugin's thread pool (sized byPOLARS_MAX_THREADS) and reassembled in order, so eagerwith_columns/selecton a single-chunk column is multi-core: 3–4× faster on 4 cores in the measured cases. A call spreads its rows only when it runs alone and the graph's previous call did too, so the streaming engine's concurrent morsel calls stay on their own threads, and a one-thread pool never splits. Underon_error="raise"the reported error is still the earliest failing row's. Breaking: the one-time "ran on one thread" warning and itsPOLARS_CV_ENGINE_WARN_SECONDS/POLARS_CV_ENGINE_WARN_ROWS/POLARS_CV_SILENCE_ENGINE_WARNINGvariables are removed. (CR-32) - Every
.contour,.pointand.bboxfunction uses every core on the in-memory engine. They split a call's rows over the plugin's thread pool through one row driver (GeomParams::map_rows), as pipelines do, and stay inline under the streaming engine, which is already parallel across morsels (~3.2x for.contour.area()over 200k contours on 4 cores). - Everything the plan knows about an op's input is checked where the op is
written. An op no row could run on its planned input raises when it is
appended, naming the op, rather than failing every row: an image op on a
rank-1 buffer (
source("raw").blur()), a crop window past a known edge, a channel index past a known channel count, atransposewith a repeated axis, and two operands whose known sizes cannot broadcast (x.add(y)over(4, 5, 3)and(2, 5, 3)used to plan(4, 5, 3)and then fail each row). A size the plan does not know is still checked per row. Validation messages show an unknown size as?. - A binary op plans each axis it can. Its operands broadcast axis by axis, so two 8x8 images of unknown channel count add to a planned 8x8 (the planner used to drop every size when one was unknown). A lazy continuation or binary op starts from the upstream state as planned rather than a copy with the sizes dropped.
apply_maskandchannel_mergeare planned against the nodes they read. A mask that cannot broadcast against its buffer, or channels of different known sizes, raise when the op is written; a merge plans the H and W any of its channels knows.- Rank and channels come from the op's shape. A crop, transpose or reshape now plans a known channel count, a size an op replaces is no longer carried across it when the input rank is unknown, and a contour measure plans one value per contour (its length is the set's size) rather than a single value.
assert_shapeis a checked operation. It is planned where it is written and checked against every row there; a mismatch fails the row naming the assertion. Optimizations may rely on the declared shape (it is a fact once checked), and an assertion is never removed.dims=entries may be per-row expressions, and its sizes must be positive ints whichever spelling (dims=orheight=/width=/channels=) carries them.repr()/explain()render theassert_shapecalls that were written, where they were written, instead of every inferred size as anassert_shape(...).- Sinks are checked at
.sink()..sink()validates the graph with the plugin's own compile, planning and sink-schema code (_lib.check_graph), so sink problems raiseValueErrorthere. - The builder's planner and the executor share one dtype lattice. A
float-promoting op over a dtype the plan does not know (
scale,sqrt,sobel, … aftersource("image_bytes")) is planned asauto_float— "a float, which one depends on the decode" — where it wasauto.Pipeline.output_dtype()can therefore return"auto_float". - Planner state is a Rust object.
Pipelineholds one immutablepolars_cv._lib.Plan(its source, typed ops and the state at every op boundary);Pipeline._stateis a frozenpolars_cv._lib.PlanState. The PythonPlanStatedataclass andHINT_DIMSare gone;Domainis generated from the Rust registry like every other enum.repr(pipeline)shows enum arguments by their wire name (origin=centroid). - Operations report errors from their Rust definition. Errors name the
op, the field and the valid values (
operation 'resize': 'filter': unknown FilterType "bogus", expected one of [...],'top': -5 cannot be negative), and a source/sink keyword the format does not read is refused naming where it does apply ('quality' does not apply to the 'webp' sink (it applies to: jpeg)), as is a misspelled one. A per-row enum value outside its table readsunknown Winding "CW", expected one of [...]. - Hand-built graph JSON follows the typed definitions. The wire form of a
field is the value itself (
"height": 224,"filter": "bilinear") or{"$slot": n}, the index of the plugin input column that carries an expression parameter; each distinct expression (byExpr.meta.eq) gets one input, root columns first. Wire fields follow the Python signatures:warp_affine(output_size=)(wasoutput_height/output_width),histogram(range=)(wasrange_min/range_max, where one without the other was silently ignored), the binary ops'other,apply_mask'smaskandchannel_merge'sothers(wereother_node/other_nodes), andrasterize'ssize:[height, width]or the id of the node whose canvas it takes. Breaking: a misspelled, extra, missing or inapplicable field is rejected by name, as are a sinkdtypeother than"f16"/"float16"(which Rust used to accept and ignore), theexpr_column_nameskwarg, and the node fieldsalias,domainandoutput_dtype(the plugin read none of them). The graph JSON (the compiled-graph cache key) no longer depends on which other expressions are alive in the process, and carries no expression text. - One way into the compiled plugin.
polars_cv._plugin.callnow builds every plugin expression. It resolves the extension file the import system loads and passes that to polars; before, polars was handed the package directory and took the first.soiterdir()returned, which beside a stale second build (_lib.abi3.sonext to_lib.cpython-3xx-*.so) could be a different library from the one the planner had just read. It also passes each argument as.ext.storage(), which is how tagged inputs are accepted without any Rust input path knowing about extension types. Breaking:_graph.LIB_PATHand_namespace._LIB_PATHare removed. - Point coordinates are always matched by field name. One contour path
took a point's first two fields by position; a
.pointdistance function given a contour set now says so instead of failing to find anxfield.
Removed¶
PipelineGraph.set_root_column();Pipeline.to_graph(column)requires its column. The two existed for a graph built before its column was known, a route nothing took; the root check still refuses a graph whose root has no column.- Loose contour-struct matching. A contour struct's ring is its
exteriorfield. Apointsfield was read as an alias, and a struct with neither had its first list field taken as the exterior. Such a struct is now refused ("Contour struct has no 'exterior' field"); rename the field. sobel(ksize=)andlaplacian(ksize=). Both accepted only3and raised for anything else; the kernels are the 3x3 ones.sobel(axis=)now refuses a value other than"x"/"y"(it computed the y gradient for any of them).convolve2d(ksize=). The side is the kernel's: its length must be the square of an odd number (9 for 3×3, 25 for 5×5, ...), andkernelis now the op's one positional parameter (.convolve2d(k)).ksizecould only restate that length, and a per-rowksizecould only fail when it disagreed.- The resize family's
filter=no longer accepts"triangle"from a per-row column. It was a parser-only alias for"bilinear"that the Python builder already rejected as a literal, so only a column value could reach it. Use"bilinear". Likewiseconvert_colorno longer has the unreachable"grey"/"grayscale"spellings; use"gray". - The
affine_fusionoptimization pass, and therotate_affine_paramsFFI it used. Collapsing a run of warps into one composed warp folds several interpolation passes into one (and drops the intermediate clip of anexpand=Falserotate), which changes pixels by as much as ~185/255 — so it could not satisfy the on/off output-equivalence guarantee every optimization now carries.rotate/warp_affine/rotate_and_scaleops are unchanged and still execute (unfused); only the plan-time fusion of adjacent affine ops is gone.
Fixed¶
- Inputs an op cannot handle are row errors, and some silent wrong results
are now errors. Every operation now checks its input shape before running,
so a shape the plan could not see (e.g. from a
blobsource) is an ordinary error thaton_errorhandles, rather than a panic. As a result some inputs that used to "succeed" with a wrong or unsafe result now raise: reshapeto a different element count, which produced a view over memory the buffer did not own;- image operations (resize, colour conversion, filters, rotation, affine
warps, grayscale, canny) given anything but
[H, W]or[H, W, C], which either passed the input through unchanged or read a 4-D buffer with an axis silently dropped; transposewith fewer axes than the input;- a reshape after a transpose/flip/crop (a non-contiguous view).
normalize(method="minmax"/"zscore") explicitly accepts any shape, and
perceptual_hash any power-of-two hash_size (CR-34).
- crop rejects windows it cannot honour. A negative top/left/height/
width is an error (a literal when the pipeline is built, a per-row value as
a row error), and so is a window that runs past the image. Previously a
negative offset was clamped to 0 while the extent was kept — returning a
shifted window — and an overrunning window was silently shrunk. height
and width are now independent: giving only one used to discard it. Row
errors follow on_error. (CR-42)
- source("raw") rejects a byte length that is not a whole number of
elements instead of dropping the remainder. (CR-42)
- A jagged or null List row is refused, not misread. The list source
took each level's size from its first element and read the row's values as
that shape: [[1, 2], [3]] read past the end of its values (the missing
slot held whatever memory followed), [[1, 2], [3], [4, 5, 6]] came back
silently re-rowed as [[1, 2], [3, 4], [5, 6]], and a null value read as
0. A row must now be a grid of values; anything else is an error naming
what is wrong. .contour.label_reduce's image operand decodes the same way
(it had its own parser, which dropped null pixels and shifted the rest of
the row); a flat, one-dimensional image is refused rather than read as one
row, and an empty image scores a null row rather than zeros.
- A null contour coordinate is an error, not the origin. A point with a
null x or y, or a null point inside a ring, used to read as 0.0,
silently moving the ring (and changing its area, IoU, rasterization…). Every
contour consumer now refuses it, naming the row.
- Geometry values are read by field name, and a missing value is never
0.0. .contour.contains_point read a point's first two fields by
position (a {y, x} point was tested swapped) and read a missing,
null or non-Float64 coordinate as 0.0 — an integer point was tested at
the origin. A bbox's null or missing field read as 0.0. Every
.point/.contour/.bbox function now reads points and bboxes through one
reader each, by field name, refusing a null field or a non-Float64 one.
- .point.rotate(origin=...) with a null origin is a null row. It
rotated about (0, 0), as if no origin had been given.
- A null input point gives a null result point. Point transforms
returned a struct of null fields ({"x": None, "y": None}) for a null row;
they now return a null row, as every other geometry function does.
- numpy_from_struct refuses a struct describing memory outside its
data. Its strided path built the view with as_strided, which does not
bounds-check, so a hand-built struct whose shape, strides or offset reached
past its data read whatever memory followed. It now builds every array
through the same checked view numpy_from_column uses (the byte range the
strides reach must lie within the row), and so does its no-strides path,
which reported a short data as numpy's "cannot reshape".
- An op reading another node checks that node's domain at build.
img.apply_mask(contours), img.add(contours) and channel_merge over a
contour-domain node built, planned a buffer schema and failed every row;
they are now refused while the pipeline is built, like the op's own input.
- apply_contour_mask() runs the whole contour pipeline. It rebuilt the
mask from the contour pipeline's first op alone, so a translate() or
scale_contour() before the rasterize was silently dropped, and a contour
node continued from another node lost its input. Only a trailing
rasterize()'s canvas is replaced (its paint is kept); a pipeline ending
in neither contours nor rasterize() is refused.
- A per-row parameter is no longer validated as a stand-in value. The
planner resolved each op with a placeholder for every expression parameter
(1 for an integer) to read its rules, so
channel_select(index=pl.col("i")) on a known [H, W] buffer was refused
while building as "channel 1", though every row selected channel 0. The plan
now reads each op as written: a literal is checked when the pipeline is
built, a per-row value when its row runs, and nothing is invented. A per-row
blur sigma or morphology ksize likewise plans a per-row neighbourhood
radius rather than the placeholder's.
- An image of unknown size is no longer planned as square. The plan-time
shape prober stood the same placeholder in for every unknown input axis, so
an aspect-preserving resize made per-row by another parameter
(resize_max(7, filter=pl.col("f"))) published [7, 7] for a 100x50 image
that executes as 7x4. Shapes are now computed symbolically: only what the op
fixes is known (resize_to_height(7, ...) plans [7, ?]). Conversely a
per-row rotation angle over a known square image now keeps its size.
- Distinct expression parameters no longer collapse into one. An
expression parameter was identified by its display text, str(expr), which
is not unique: every pl.lit(pl.Series("f", ...)) prints Series[f], and
every Python UDF prints python_udf. Two such parameters in one pipeline
shared a plugin input, so the second op silently read the first op's values —
.scale(pl.lit(s1)).scale(pl.lit(s2)) applied s1 twice. The same text
identity decided CSE merges and root-column deduplication. Identity is now
Expr.meta.eq, through one slot table per graph (CR-31).
- cast(A)→cast(B) no longer drops a narrowing intermediate cast. The engine
collapsed consecutive casts unconditionally, discarding the intermediate even
when it quantized — a fractional-f32 .cast("u8").cast("f32") returned 0.5
instead of the correct 1.0. It now collapses only when the intermediate dtype
losslessly holds the input (DType::losslessly_contains); a narrowing
intermediate is kept and executes. A float intermediate between an integer
input and an integer target is kept too, although it is lossless: float → int
saturates where int → int wraps, so collapsing u16 → cast("f32") → cast("u8")
turned 300 into 44 instead of 255.
- A malformed VIEW blob can no longer cause undefined behaviour. A blob
whose data_offset or strides were not multiples of the element size built a
misaligned typed slice in release builds (debug builds panicked). Both blob
decoders now read through one validated parser (view_buffer::parse_blob)
that rejects it as a row error, and ViewBuffer::as_ptr checks alignment in
every build. ViewBuffer::from_blob also no longer reads past its own copy
for a strided blob. (CR-41)
- on_error("null") now covers failures the engine raises as panics. Some
data-dependent failures, such as two operands that cannot broadcast, are
raised inside the engine as panics, which were caught once per batch. Under
on_error("null") one such row failed the whole query, good rows included,
and under streaming how much of the query it took down depended on the
morsel size. They are now caught per row and follow the row policy like any
other error (CR-34).
- A row the sink cannot represent is an error, not a null. If the encoder
ever produced a row of a different kind than the sink expects,
build_series_from_spec published it as null. It now raises an internal error
naming both kinds (CR-38).
- list and array sink rows that break the plan are errors. A row whose
dtype differs from the planned one was silently cast (only the first row was
checked), and an array row with the wrong element count was accepted
(CR-33).
- uv run no longer builds the extension at release LTO
([tool.uv] package = false). (CR-43)
- The OpenCV and Pillow benchmark adapters time the operation polars-cv
runs. Several single-op benchmarks compared different work, and the
suite's output validator reported the gap as a correctness failure:
resize ran OpenCV's INTER_AREA against bilinear; sharpen an unsharp
mask (OpenCV) or Pillow's SHARPEN against polars-cv's 3×3 kernel;
erode/dilate thresholded to a mask on the polars-cv side only;
adjust_contrast and adjust_brightness truncated to u8 (Pillow's
contrast also took the luminance mean) against polars-cv's f32 result;
blur used another kernel size and border; sobel_x another border; and
rotate turned counter-clockwise (Pillow), about another centre, and
truncated its expanded canvas. Each adapter now computes polars-cv's
definition, and Pillow's sharpen raises NotImplementedError, as its
missing canny/sobel_x do, instead of timing another filter.
tests/test_benchmark_adapters.py holds every single-op benchmark, per
library, to a stated per-pixel tolerance against both polars-cv engines.
Performance¶
- Less per-row executor overhead. A run of all-literal buffer ops is planned once per input layout per call and replayed, instead of re-planned on every row (a 4-op chain: 2.62 → ~1.85 µs/row), and the row loop addresses nodes by position instead of hashing node-id strings (no-op rows: 1.16 → 0.72 µs). Debug-build figures (CR-37).
cannyon u8 input runs in integer arithmetic over padded planes, as OpenCV does.- Image decode and encode no longer copy every pixel an extra time. Decoding hands the codec's allocation over instead of converting through a new image; PNG, JPEG and WebP encode read the buffer's pixels in place. Only a colour type the encoder refuses as it is (JPEG's alpha layouts) still converts. 16-bit PNG keeps the one image-sized buffer its big-endian byte swap needs.
blobandrawsources read aligned rows in place. A row longer than 12 bytes that already starts on an 8-byte boundary in the column's memory (ablobsink's rows always do) is read where it lies; others are copied once, at memcpy speed (the aligned copy used to assemble 8-byte words one at a time, ~7x slower).- f32 scalar ops write in place in pipelines, as the engine intended. A segment takes its input by value, and a node that is its upstream's only reader (not a graph output, not an operand) moves that output instead of sharing it, so the fused kernel's in-place path applies.
listandarraysinks are built straight into Arrow and copy each row's values once. Alistsink of rank 2 or more, or anarraysink with even one null row, built oneAnyValueper element and ran ~34–40x slower thansink("numpy"); both now cost about what the numpy sink costs (CR-33).- Contour columns are read and written straight from Arrow. Every contour
consumer shares one reader over the column's arrays (6 allocations for a
two-contour row, was 98), and the contour sink and
.contourtransforms build their output directly: encode overhead drops ~10x and.contour.translate()~6x on a 90k-point batch (CR-36). source("array")is no longer quadratic in the batch size. Rows are views into the column (~1.2 µs per row, was ~35 µs on a 100k-row morsel of 8×8 arrays) (CR-40).blur()uses AVX2 when the CPU has it, in the published wheels too. The wheels target baseline x86-64; blur dispatches at runtime to an AVX2 build (1.2–1.3x faster), with bit-identical output on every CPU (CR-35).- Plan-time passes are linear. Identity elimination folds each op's
entering state once (O(n), was O(n²) FFI calls). CSE groups nodes by a
canonical source serialization instead of
hash(source), so a hash collision cannot fuse across different sources.
Internal¶
- One typed op protocol. Every op is a variant of a mode-generic family
enum (
ImageOpKind<M>,ComputeOp<M>,GraphOp<M>, …) listed intyped_ops!; sources and sinks are#[derive(Ops)]families (formats::source::Source,formats::sink::Sink), and the geometry accessors areContourFn/PointFn/BBoxFn.TypedOpisGraphStep<Wire>: the planner (src/plan.rs, oneplan::stepper appended op) reads its rules directly, and resolving it per row gives theGraphStep<Exec>that runs. New cratepolars-cv-macrosholds the derives. Each op declares its shape as data (Op::shape() -> OpShape, evaluated on known sizes at execution and symbolically at plan time), and new requiredOpcontractsidentity_ruleandis_spatial_windowdrive identity elimination and the spatial pushdown with no op name matched in Python. The legacy name-keyed op protocol is deleted. - The Python surface is generated.
scripts/gen_ops.pywrites the enums, every builder method,source(), the geometry accessors andLazyPipelineExpr's chainable methods (_ops_generated.py,_lazy_forwarders.py) from the committed catalogues (tests/golden/{op,io,enum,geom}_catalog.json);lazy.pyiandscripts/gen_lazy_stub.pyare gone andtycheckslazy.pyitself. Every "Domain:" docstring line is generated from the op's domain contract, and a hand-written sugar method's is composed from the ops it declares (@_sugar). - Safety net. A golden behaviour corpus (
tests/golden/op_corpus.json), a frozen builder call surface (tests/golden/signatures.json), a pickle/copy pin, a single test seam for planner state (tests/_plan_view.py) and plan-build benchmarks (benchmarks/plan_build.py). - Source decoding lives in
graph/decode.rs, and geometry functions share one row splitter (row_split.rs) and one reader per geometry type (geom_columns.rs). scripts/verify.shand the pre-commit clippy hook run cargo under the PyO3 environmentmaturin developsets (scripts/with-pyo3-env.sh), so the two no longer invalidate each other's builds;verify.shchecks that cargo finds maturin's build fresh.
0.28.0 — 2026-09-12¶
Added¶
- 14 core elementwise-math
Pipelinemethods.neg,abs,sqrt,square,reciprocal,sign,floor,ceil,round,trunc,clamp_min,clamp_max,add_constantandsubtract_constantjoin the pure-elementwise scalar-op vocabulary. Each promotes to float (integers → f32, f64 preserved) likescale/reluand fuses automatically with adjacent scalar ops into a single kernel pass, so a chain of them costs one pass rather than one per op.clamp_min/clamp_max/add_constant/subtract_constanttake a bound that may be a per-row expression;roundbreaks ties to even (banker's rounding, matching Polars/numpy). A cross-path guard pins the f32 bulk kernel and the f64 cold path to compute the same function for every variant, so the fused and unfused results cannot diverge.
Changed¶
froc_aucnow requires an explicitfp_range, andfroc_auc/lroc_aucnormalize by default (breaking). The trapezoidal AUC previously defaulted to the raw area over the observed x-range, which for FROC is neither bounded nor comparable across models (its false-positives-per-image axis is unbounded and its observed maximum is model-dependent). Now:froc_auc(method="trapezoidal")raises without anfp_range— there is no reasonable default window; pass e.g.fp_range=(0.0, 8.0), or usefroc_summary_table/method="mann_whitney"for range-free summaries.lroc_aucdefaultsfpf_rangeto the full(0.0, 1.0)FPF domain (LROC's axis is a bounded fraction), preserving the classic LROC AUC value.correctionnow defaults to"normalize"for both, so the result is the mean sensitivity over the window (bounded, comparable); passcorrection=Nonefor the raw partial area.froc_auc_ci_lazy/lroc_auc_ci_lazymirror these defaults.-
method="mann_whitney"is unchanged; it rejects a range and ignorescorrection(a rank statistic integrates no curve). -
All AUC integrals are now lazy; the eager
_auc.pyintegrals were removed.trapz_auc,partial_aucand_interp(Series-based reductions to a Python float) are deleted. Every AUC —froc_auc/lroc_auc,MetricResult.aucand all-points/trapezoidal AP — reduces through the single lazy authority in_auc_expr.py(collapse_curve+trapz_auc_expr/partial_auc_expr) and collects streaming._auc.pykeeps only thecorrectionvocabulary (CorrectionMethod+validate_correction). Consequence:partial_auc's below-curveUserWarninghad no lazy equivalent and is gone.
Fixed¶
-
Weighted FROC partial-AUC is weight-scale invariant again.
froc_curve_lazydivided sensitivity and false-positives-per-image by weighted denominators floored withmax(..., 1.0), which assumes unit-scale weights: under normalized (sub-unit) weights the weighted ground-truth mass is< 1and was clamped up to1, inflating the denominator and roughly halving the partial AUC purely by rescaling the weights — a metric that should be weight-scale invariant moved by rescaling alone. The division now guards against the unclamped weighted mass (mirroring the LROC curve), and a zero weighted denominator (no weighted ground truths, or all-zero weights) yieldsnull, surfacing the degenerate input rather than fabricating a value. LROC's own zero-mass branch, which previously substituted the unweighted count and silently masked the same degeneracy, now yieldsnulltoo. Guarded by new weight-scale-invariance and null-on-degenerate-mass regression tests for both FROC and LROC. -
correction="normalize"no longer returns non-deterministic garbage on the in-memory engine. The normalize branch guarded the zero-span case withpl.when(span > 0).then(raw / span).otherwise(0.0), readingspanoff the sorted x. The in-memory engine miscompiles apl.whenwhose predicate embeds a sorted reduction next to a.thenthat also embeds sorted reductions: an ungrouped normalized.collect()(the default engine) returned values that changed run-to-run and were frequently negative, whileengine="streaming"and thegroup_by().agg()path compiled the same expression correctly.spanis now read off the unsorted column (x.max() - x.min(), order-independent and numerically identical), so no sort enters the predicate. Guarded bytest_auc_expr.py::TestNormalizeEngineParityand the engine-stability tests intest_froc_auc.py/test_lroc_auc.py.
0.27.0 — 2026-09-10¶
Removed¶
- The McClish standardized partial-AUC correction is removed (breaking).
correction="mcclish"is no longer accepted byfroc_auc,lroc_auc,partial_auc/trapz_auc, their_auc_exprreductions, orMetricResult.auc; themcclish_correctionhelper and its expression twin_mcclish_correction_exprare deleted. McClish's standardization (Med Decis Making 1989;9(3):190–195) maps a partial area into[0.5, 1.0]against the ROC chance diagonaly = xon the unit square, so it is only meaningful when the x-axis is a probability bounded to[0, 1]. None of the curves this package integrates is that ROC curve: FROC's x-axis is false positives per image (unbounded — for any window withlo+hi ≥ 2, such as the standardfp_range=(0, 8), the old code silently returned the sentinel0.5for every model), LROC's chance line is not the diagonal, and the PR chance line is horizontal at prevalence. Thecorrectionvocabulary is now"normalize"(divide the partial area by the window width — a bounded mean sensitivity that needs no ROC assumption) orNone, and an unrecognisedcorrectionis now rejected withValueErrorrather than silently degraded to the raw area. Migratecorrection="mcclish"→correction="normalize".
0.26.0 — 2026-09-08¶
Changed¶
-
All-points AP is now independent of input row order on tied scores. The all-points average precision (
average_precision(..., interpolation="all_points"),PrecisionRecallResult.auc("all_points"),mean_average_precision,all_points_ap_by_group, and the*_apbootstraps) now applies a canonical tie convention: detections that share a score collapse into a single precision–recall point, using the cumulative TP/FP after the whole tied block (the scikit-learn / COCO convention). Previously the curve visited tied detections in input row order, so the AP — and thePrecisionRecallResult.curve— could differ between runs that shuffled equal- scored rows. This changes AP output on inputs with tied scores (values with strictly distinct scores are unaffected). It also makes the scalar and grouped estimators agree exactly on ties. -
Bootstrap confidence intervals are now lazy and group-aware (breaking). The eager, scalar bootstrap API is removed —
bootstrap_froc_auc,bootstrap_lroc_auc,bootstrap_pr_auc,bootstrap_metric_sequential,BootstrapResult, andMetricResult.bootstrap_ci(with its_bootstrap_groupedhook and thethreshold_counts_by_groupauthority) are all gone. In their place are three free functions —froc_auc_ci_lazy,lroc_auc_ci_lazy,average_precision_ci_lazy— each returning apl.LazyFrame[*group_by, <metric>, ci_lower, ci_upper]and never collecting internally. The whole bootstrap (resample, per-replicate metric, and the percentile bounds) is one Polars plan the caller collects, so a CI can be built at plan time with no data present and joined onto a point-metric frame instead of looping over groups in Python. Migratebootstrap_froc_auc(t, seed=0).ci_lower→froc_auc_ci_lazy(t, seed=0).collect()["ci_lower"].item(), and reach forgroup_by=to get one bound row per group. - The resample is now genuinely collect-free.
_lazy_resamplebuilds its draw skeleton by cross-joining a constant-length reps frame against the base units, so the per-replicate slot count is a window expression rather than theO(1)scalar collect the previous release still paid. Draws are partitioned within each group (a group only redraws its own units) and stratified withingt_label; an empty base or group yields no rows instead of raising. - Degenerate groups null their bounds instead of raising. A group keeps its
point estimate but reports null
ci_lower/ci_upper, so one lazy plan spans viable and degenerate groups — no per-grouptry/except. Viability needs ≥1 positive target; undermethod="mann_whitney"(a two-class rank statistic) it also needs ≥1 negative. mean_average_precision(interpolation="all_points")is now a single lazy plan. It computes every(iou_threshold, class)AP through the shared grouped authorityall_points_ap_by_groupinstead of a Python loop of eager per-cellaverage_precisioncollects. It now applies the same canonical tie convention as the rest of the all-points path (above), so its output is order-independent on ties as well as bit-identical to the previous loop on distinct scores. The VOC"11_point"method is unchanged (it keeps the per-class loop).shearandrotate_and_scalerequire their sizing parameters (breaking).output_size(both) andcenter(rotate_and_scale) are now required keyword-only arguments instead of defaulting toNoneand immediately raising "auto-size not yet implemented".output_sizeis structural — it sets the plan-time output H/W — and an image source's dimensions are unknown at plan time, so auto-sizing cannot yield a plan-time shape; the requirement is the principled end state, not a placeholder. Callers that already passed both are unaffected; omitting one now raisesTypeErrorat call time rather thanValueErrorat build time.
Removed¶
PrecisionRecallResult.detection_table— it existed only to feed the removedbootstrap_ci; nothing reads it now.MetricResult.interpolate/MetricResult.summary_table— unused eager wrappers overinterpolate_curve_lazy; nothing called them (the FROC/LROC helpers build oninterpolate_curve_lazydirectly and stay lazy). To interpolate a result's curve, callinterpolate_curve_lazy(result.curve.lazy(), …).- The
heatmap=back-compat alias on the.contour.label_reduce()accessor — it had no callers and duplicatedimage=. Passimage=instead. (ThePipeline.label_reduce(contours=...)method is unaffected.)
Fixed¶
grayscaleon non-u8data no longer corrupts downstream kernel fusion.ViewExpr::grayscale()tracked its output dtype asu8regardless of input, contradicting the op'sPreserveInputcontract. The published schema stayed correct (it reads the contract over FFI), so the divergence was silent — but a fused compute block after grayscale (e.g.grayscale → invert → scale) read the mistrackedu8and inverted normalizedf32as255 − xinstead of1 − x, an off-by-254 wrong result. The four remaining image ops that still routed through hand-written builders (grayscale,threshold,resize,blur) are folded intoapply_op's singleViewDto::Imageconstruction arm, which derives shape/strides/dtype from each op's contract; the image builders are now thin wrappers over it, so no builder can track a dtype its contract does not declare. The dtype-tracking guard was extended to coverGrayscale/Threshold/Blur/Resizeand the builder methods, and a fusion-execution regression was added.
0.25.0 — 2026-09-05¶
Changed¶
- Bootstrap resampling is now fully lazy and streaming. The resample frame
was the last eager step — a Python
for b in range(n_bootstrap)loop callingpl.Series.sampleand building an_bootstrap × n_unitsDataFrame in memory before any lazy plan ran. It is replaced by_lazy_resample, a position-independent hash-expression draw over a lazypl.int_rangeskeleton (hash(slot, seed) % stratum_size), so the whole bootstrap — draw included — runs as one streaming plan and the intermediate frame never leaves the engine. The only materialization is anO(1)scalar collect of the base-unit count.bootstrap_froc_auc/bootstrap_lroc_auc/bootstrap_pr_aucinherit this;bootstrap_pr_aucalso dropped the two eagercollect()s it did on the source frames. - The draw is invariant to the runtime thread count. Because each draw is a
hash of its own global slot id (never of row position), a given
seedyields a bit-identical confidence interval regardless ofPOLARS_MAX_THREADSor how the streaming engine morselizes — closing the join-order nondeterminism the old path fought (it once produced negative AUCs on macOS CI). MetricResult.bootstrap_ci(PR/Confusion) is vectorized. The sequential per-replicate reconstruct loop (a full collect per replicate per metric) is gone; one shared lazy resample feeds_bootstrap_grouped, which computes each metric grouped bybootstrap_idin a single plan.bootstrap_ci(metric="auc")is now bit-identical tobootstrap_pr_auc(same stratified resample, same all-points estimator, read from the sharedall_points_ap_by_group/threshold_counts_by_groupauthorities). Supported metrics:auc(all_points),precision_at,recall_at; any other method raises rather than silently degrading.
Removed¶
MetricResult._reconstruct(and the PR override). It rebuilt and collected a whole result per replicate; the vectorized_bootstrap_groupedhook replaces it. Guarded bytest_removed_surfaces.py::test_bootstrap_reconstruct_hook_is_gone.MetricResult._resolve_sampling_entities— the last eager step in the entity-level path. It collected the metadata and iterated rows in Python to build theentity → [image_id]map;_resolve_bootstrap_samplesnow expands entities to images with a lazygroup_by/explodeinstead, so nothing in the bootstrap flow leaves the streaming engine. Guarded bytest_removed_surfaces.py::test_eager_sampling_entity_resolver_is_gone.DetectionTable.image_ids_and_strata— the eager image-id list plus{image_id: stratum}dict the old bootstrap sampled from. Its only caller was_resolve_sampling_entities(also removed);_lazy_resamplenow stratifies directly off thegt_labelcolumn with no Python-side id list or strata dict. Guarded bytest_removed_surfaces.py::test_detection_table_image_ids_and_strata_is_gone.
Note¶
- The lazy sampler changes the RNG stream: a given
seedno longer reproduces the pre-lazypl.Series.sampledraw values (it reproduces its own draws bit-for-bit, on every platform and thread count).seed=Noneis now deterministic (a fixed hash constant) rather than a fresh draw each run.
0.24.0 — 2026-09-03¶
Changed¶
- FROC is fully lazy again; the conflicting-weight guard is gone.
froc_curve_lazy/froc_aucno longer collectimage_metadataat construction time — they build a plan with zero eager collects, so the trapezoidal path streams like the Mann-Whitney and LROC paths already did. The build-time guard that raisedValueErroron disagreeing duplicate weights is removed; a newweight_aggkeyword ("first"default, plus"min"/"max"/"mean"/"sum") resolves each(image[, class])key's weight deterministically for the numerator and the denominators alike (metrics._weights.resolve_key_weights). Conflicting weights now resolve per this policy instead of raising — supplying consistent weights is the caller's responsibility. - The interpolation/summary helpers are lazy.
froc_sensitivity_at_fp,froc_summary_tableandlroc_sensitivity_at_fpfnow return aLazyFrame(breaking: previouslyfloat | None/DataFrame); no metric method collects internally — the caller owns the collect. A scalar isfroc_sensitivity_at_fp(t, fp).collect().item(). They andMetricResult.interpolate/summary_tableshare one lazy authority,metrics._auc_expr.interpolate_curve_lazy(collapse_curve+join_asof).
Added¶
- Weighted Mann-Whitney FROC/LROC AUC.
froc_auc/lroc_aucwithmethod="mann_whitney"are now weighted byimage_metadata.weightat bothlevel="detection"andlevel="image", consistent with the trapezoidal path (previously weights were ignored). Implemented as one weighted-capable authority —metrics._auc_expr.collapse_scores(bucket by distinct score → positive/ negative weight mass) thenmann_whitney_auc_expr(weighted rank-sum) — so the unweighted statistic is literally the weighted one with every weight= 1, not a second implementation.bootstrap_froc_auc/bootstrap_lroc_aucinherit the weighting throughfroc_auc/lroc_auc.
0.23.0 — 2026-09-03¶
Added¶
-
Expression-valued, group-aware FROC and LROC AUC.
froc_aucandlroc_aucreturn aLazyFrame— one row per group — instead of reducing a curve to a Python float, so the integral rides inside a lazy plan and agroup_bybecomes a Polars grouping rather than a Python loop. Both take the trapezoidal family (raw/partial/normalize/mcclish) and the Mann-Whitney family; FROC keeps its image-level Mann-Whitney (level="image") and both expose the detection level. The reusable expression core lives inmetrics._auc_expr, the single authority both the curve builders and the AUC path read. -
Lazy curve and summary helpers.
froc_curve_lazy/lroc_curve_lazyreturn the curve as aLazyFrame, andfroc_sensitivity_at_fp,lroc_sensitivity_at_fpfandfroc_summary_tableare standalone lazy functions built on them, reusing the sharedMetricResultinterpolation and summary geometry.froc_sensitivity_at_fp/lroc_sensitivity_at_fpfreturnNonewhen the operating point is off the curve;froc_summary_tablereturns a null row. -
Vectorized, seed-reproducible FROC/LROC bootstrap.
bootstrap_froc_aucandbootstrap_lroc_aucbuild every replicate as one seeded(bootstrap_id, image_id)sample frame and compute the AUC grouped by replicate in a single lazy plan, replacing the sequential per-replicate Python loop. The confidence interval is now reproducible for a givenseed(the ids are sorted before sampling, closing the.unique()order-dependence that made a fixed seed sample different images each run). Asample_colargument resamples at the entity level (expanding entities to their images), and alevelargument selects the image- or detection-level Mann-Whitney AUC. -
Contour/bbox matchers accept a pre-decoded
LazyPipelineExpr.ContourMatcher.match()(andBBoxMatcher.match()) now take either a column name or a pre-decodedLazyPipelineExprforpred_col/gt_col. When an expression is passed the matcher appends its threshold /extract_contours/label_reduceonto the node the caller already built, so a segmentation graph and the contour extraction share one decode (collapsed by CSE) and stream from a single collect.
Changed¶
- The contour and bbox matchers are lazy — one collect at the boundary.
ContourMatcher.match/BBoxMatcher.matchno longer materialize internally; the eagercollect(engine="streaming")that split the frame and ran an alignment guard is replaced by a cached lazyimage_level, andDetectionTable.collectruns both derived frames through a singlepl.collect_all, so common-subplan elimination executes the shared contour-extraction/correspond graph once (decode-once, collect-once) instead of once per frame. The empty short-circuit is gone: empty and populated inputs flow through the same lazy expressions, so their detection tables can no longer disagree about dtypes.
Removed¶
-
The eager FROC/LROC result API is gone.
FROCResult/LROCResult,froc_curve/lroc_curve, the dead eager curve builders, and the FROC/LROC-onlymann_whitney_u_auc/detection_level_mann_whitneyin_auc.pyare removed — they were a second implementation of the integral the expression path now owns. Callers usefroc_auc(table).collect().item()(or thefroc_sensitivity_at_fp/froc_summary_tablecurve helpers, andbootstrap_froc_auc/bootstrap_lroc_aucfor CIs); the FROC/LROC Mann-Whitney is nowmetrics._auc_expr.mann_whitney_auc_expr. The sharedMetricResultbase and_auc.py'strapz_auc/partial_auc/_interp/mcclish_correctionstay — the PR curve and AP still use them.test_removed_surfaces.pyguards the deletion. -
The multi-metric-dict bootstrap convenience is dropped for FROC/LROC. Call
bootstrap_froc_auc/bootstrap_lroc_auconce per metric.
0.22.0 — 2026-09-01¶
Changed¶
- Detection matching left the CV layers; contour correspondence replaced it.
view-buffer— the tensor engine — implemented greedy one-to-one assignment walked in descending confidence order, which is the COCO/PASCAL evaluation protocol, and returned a struct whose fields were true/false-positive counts. The plugin'sscore_ordercomputed that confidence ranking. Between them the two CV layers owned a policy that belongs to whoever knows what the contours mean.
.contour.match_detections() / .bbox.match_detections() are replaced by
.contour.correspond() / .bbox.correspond(), which run the same rule over
the same overlap matrix but take a visit order — a permutation — instead
of scores. Deriving that order from confidence is now metrics' business.
The engine's half is pairwise::greedy_assign, an assignment over a matrix
that knows nothing about detections.
The rule itself did not change. Every detection row, average precision,
PR AUC (both methods), LROC AUC and Mann-Whitney AUC came back bit-identical
across both matcher paths — 208 of 213 captured values exactly equal. The
other five are fp_per_image and the FROC trapezoidal AUC derived from it,
differing by ≤3e-16; that field is already nondeterministic run to run
(its weights arrive through a join, and joins do not preserve row order), a
pre-existing property confirmed by running unmodified code five times and
getting two different values.
correspondrejects a malformed visit order — wrong length, out of range, or repeated — naming the row. The rule it replaced took scores and silently skipped any index it could not use, so a short or duplicated order quietly left predictions unvisited.
Removed¶
n_tp,n_fp,n_fnandpred_idxare gone from the match result. All four were computed per row, published inMATCH_RESULT_SCHEMA, and documented on two accessors — and read by nothing in the repository. The accessor docstrings argued against reading them: per-row counts undercount false negatives over any population containing ground-truth-bearing images with no detections, which is exactly the population a detection metric evaluates.pred_idxwas worse than unread, it was0..n— a second copy of the position its own list was already indexed by, and removing it collapsed the metrics explode from a join plus two explodes into one explode.
MATCH_RESULT_SCHEMA is replaced by CORRESPONDENCE_SCHEMA
(right_idx, overlap), and unlike its predecessor it is exported at the
package top level — being geometry-only is why the metrics matchers read the
old struct's fields by string literal rather than from its declaration.
-
The tie-break tolerance was unreachable, and is gone. Both the engine and its Python mirror carried an arm treating overlaps within 1e-12 as tied, so the smaller index would win. Under an ascending scan the incumbent's index is always the smaller one, so the arm's guard was never true — in the deleted code as well as the new. The documented rule is now what the code does: exact ties go to the smallest index, and an overlap larger by one ULP is not a tie. Pinned by
test_greedy_assign_does_not_treat_a_near_tie_as_a_tie. -
metricsno longer restates the dtype table._POLARS_TO_CV_DTYPEwas a hand-written Polars-type → engine-name map, kept honest by a core test that imported a metrics private to compare it againstdtype_table!— a three-way coupling that could not survive the two ever being separated. The correspondence is now generated intoNUMPY_TO_SHORTand published aspolars_cv.dtype_name_for; the guard went with the duplicate it existed for.
Fixed¶
- The match-alignment check compared two lists that could not disagree. It
compared
pred_idxagainstgt_idx, both built by the engine from one result. It now checks the payload against the prediction list, which is the alignment the explode actually depends on — and that check immediately caught a real case:extract_contoursreturns null, not an empty list, for a mask with no contours, so predictions on a ground-truth-free image had a null payload. Metrics already reads a null ground-truth column as zero targets; the payload now follows, and those predictions stay the false positives they are.
Added¶
- Greedy assignment is testable on its own. It used to live in a private
match_from_matrixreachable only through a detection-shaped wrapper, and had exactly one test — a self-match at threshold 0.5. Tie-breaking, the effect of the visit order, the threshold boundary andbboxmatching had no coverage at all in either language.greedy_assigntakes a bare matrix and has ten Rust unit tests;tests/test_correspondence.pyadds 30 more that cross-check the engine against a plain-Python reference on every case.
0.21.0 — 2026-08-24¶
Fixed¶
ContourMatchercould not read an image-bytes mask column.metrics/_matching/_contour.pydecided the source format itself and mapped everyBinarycolumn to"blob", so a PNG mask failed with "Invalid blob magic bytes" — while the same column read fine throughsource("auto"), which checks the VIEW magic bytes and falls back toimage_bytes. Metrics now usessource("auto")and contributes only the one fact the Polars schema settles that Rust cannot infer at plan time: a nested column's leaf dtype. The_gt_h/_gt_wderivation likewise branches on the Polars dtype (which decides whether a cheaper native path exists) rather than on a format string.
A String mask column now routes to file_path through auto instead of
raising in Python, so a column of paths to mask files works where it
previously did not.
Pipeline.to_graph()silently discardedon_errorandon_null_param. Three functions built aPipelinefield by field —_clone,_create_sub_pipeline, and CSE's_create_shared_node— and the second carried 11 of the 14 fields.PipelineGraph._to_dictreads the per-row policies off the node pipeline, andto_graph()makes its sub-pipeline the graph's only node, so a publicPipeline().source(...).on_error("null").to_graph(col)executed under"raise". CSE escaped it by accident: the hoist unions over all nodes and the originals kept their policy.
All three now inherit the whole state through one _copy_state_from, driven
by a _STATE_COPIERS table, and override only what they mean to change — so a
new field is carried by default rather than by remembering three call sites.
test_pipeline_state_copy_is_complete fails if the table and __init__
disagree in either direction.
-
A vector-domain reduction was accepted at plan time and rejected at execution. (see also the two notes below on what this deliberately does not extend to, and on fused rotate sizes.)
GraphStep::input_domainsdeclares that binary ops and reductions take[Buffer, Vector], and the Python planner validates against it — but execution re-derived the precondition by hand at ten sites, each with a hardcoded"<Step> requires Buffer"string, andNodeOutput::as_buffer()returnsNonefor a vector. Soextract_shape().reduce_sum()planned clean and failed the row. The ten sites are now onestep_buffer_operandthat reads the declared contract, materializing a 1-D buffer for a vector the step admits — the equivalence the planner's own docstring already claimed. A domain the contract does not admit is still refused. -
An unmappable dtype string is no longer typed
UInt8.dtype_str_to_polarsended in_ => DataType::UInt8, guarded only by a source-scanning ratchet, with a comment conceding that any unmatched string was silently typed. It now parses throughDType::from_short_nameand returnsPolarsResult, and the Polars spelling is an exhaustive match onDType, so an eleventh dtype fails to compile rather than defaulting. -
deny_unknown_fieldsnow covers the nested wire-format structs. serde's attribute does not descend, so closingGraphNodeleftSourceSpec,OutputSpec,UnifiedGraphandGraphKwargsaccepting anything. That mattered most onSourceSpec, which carriesallowed_roots: a misspelled key deserialized toNone, i.e. no path sandbox, silently. -
gcs_bearer_tokenon ans3://path now errors instead of being ignored. S3 signs with SigV4 and cannot use an OAuth bearer token; the value was accepted and silently discarded, which is the bugtoken_commandwas already rejected for one field over. -
S3 object keys are no longer double percent-encoded.
read_s3passedurl.path()— already encoded — toPath::from, which encodes again, sos3://bucket/a b.pngrequested the keya%2520b.png. Reads now use the raw prefix, which also makes polars-cv agree withpl.scan_parqueton the same URL. -
A misspelled storage option is still rejected. polars'
parse_untyped_configsilently drops keys it does not recognise ("Silently ignores custom upstream storage_options"), so delegating naively would have turnedunknown GCS storage option 'x'into a request that quietly did the wrong thing. Keys are validated against each backend's own vocabulary before the options are handed over, lower-cased exactly as polars does so the accepted set is neither wider nor narrower than what polars would act on.
Changed¶
-
Every deserialized plugin-boundary struct is closed.
ContourKwargs,PointKwargsandReadBytesKwargslacked#[serde(deny_unknown_fields)], so a kwarg Python emits and Rust no longer reads was dropped in silence — howsink("jpeg", qualtiy=50)encoded at the default quality with nothing said. Guarded in two halves that do not subsume each other: a source scan (tests/_kwargs_scan.py, with a floor so it cannot pass by matching nothing, and committed fixtures) catches a struct nobody closed; runtime probes throughregister_plugin_functioncatch an attribute present but not rejecting.cloud_auth.rs::Probeis exempt — a deliberate partial parse of a Google ADC file, and not a boundary struct. Exemptions are keyedfile.rs::Nameso a generic name cannot blanket-exempt a future struct. -
The
{x, y}point struct has one declaration. It was spelled out eight times acrosscontour.rs,point.rsandgeometry/schemas.pywith nothing relating them. Rust now readsgeom_schema::point_fields; Python keeps its ownPOINT_SCHEMA(sopolars_cv.geometrystill imports with no compiled extension) held to Rust by a bidirectional parity test over a newpoint_schemaFFI — the wayenum_variantssurfaces the naming registry. A bbox deliberately does not chain off these fields: it beginsx, yby coincidence of meaning, and chaining would drag a point rename into the bbox wire format. -
The CI/verify.sh parity guard reads the
- run:form too. It only matchedrun:on its own line, so a check added as- run: some-checkwould have been invisible to a guard whose entire claim is that a check cannot hide. No workflow here uses that form today; the point is that one could. Its regexes now have committed fixtures (tests/test_verify_script_fixtures.py) covering each branch that exists because a real check once hid. -
ViewExpr::apply_opreads each image op's declared dtype rule. TheCannyandHistogramEqualizearms hardcodedDType::U8andstrides: None— a second copy of a ruleimage.rsalready states asOutputDTypeRule::Fixed(DType::U8), in the module the planner reads the first copy from over the FFI. The dtype-preserving group beside them restatedPreserveInputthe same way. All three collapse into one arm that resolves shape, strides and dtype the way theFilterandComputeneighbours already did. Still exhaustive, so a newImageOpKindhas to say where it belongs. A newexpr.rsunit test asserts the property for every image op rather than for the two names. -
display.pyreads the generated dtype table. It carried a third Python copy of the VIEW wire codes, guarded by a regex over its own source — the weakest guard kind, and one a reformat defeats.gen_dtype_names.pynow emitsNUMPY_BY_WIRE_CODEfromdtype_table!, and the regex guard went with the duplicate it existed for. -
The matcher conformance sweep is derived from
metrics._matching.__all__with a floor assertion, rather than naming its three subjects literally. -
Pipeline.scale_contourtakesorigin=. It hardcodedScaleOrigin::Centroidand exposed no way to choose, while the same-named.contour.scaleaccessor exposedorigin=defaulting to"origin"— so a square scaled by 2 landed at (1,1)-(5,5) through the pipeline and (4,4)-(8,8) through the namespace. Both surfaces now take the parameter and agree for every explicit value.
The defaults still differ: scale_contour keeps "centroid" and
.contour.scale keeps "origin", because each is what that surface has
always done and changing either would silently move existing output.
Aligning them is a separate, deliberate decision; it is pinned in
tests/test_known_gaps.py until it is taken.
-
The generated type stub derives its imports.
gen_lazy_stub.pycarried a hardcodedpolars_cv._typesimport list, so aPipelineparameter annotated with any type it did not happen to name produced a stub referencing an undefined symbol — andtest_lazy_stub_is_current, being a regenerate-and-diff, had the same defect on both sides and could not see it.scale_contour(origin=)is the annotation that proved it. The block is now derived from what the rendered stub references, read from_types' own definitions. -
The rotation matrix has one implementation again. Affine fusion transliterated
AffineParams::from_rotationinto Python line for line, so arotate()got its matrix from Rust when it stood alone and from Python when a neighbouring op made it fusible. The two had already drifted — Python normalisedangle % 360where Rust did not, and rounded the expand bounding box half-to-even against Rust's half-away-from-zero — and nothing compared them: the test that appeared to cross-check compared Python against a third copy of the same arithmetic intest_affine_builder.py. The planner now reads the matrix through a newrotate_affine_paramsFFI, the test helper reads the same authority, andtest_removed_surfaces.pyrejects trigonometry reappearing in the fusion helper. -
GraphStep::input_domainsis exhaustive. It was the one contract method with a_ =>catch-all, so a new multi-domain variant would silently have been given a single domain — in the methodCLAUDE.mdnames as the authority for accepted input domains. -
Staleness of the compiled extension is detectable within a release cycle.
test_compiled_plugin_is_not_stalecompared release versions, and both sides read the sameCargo.tomlliteral, so they agreed throughout the entire window in which Rust gets edited without a rebuild — it could only fire across a version bump. A newbuild.rsbakes a content hash of both crates' sources into the extension,build_info()recomputes it from the working tree, andtest_compiled_plugin_matches_the_rust_sourcescompares them. This matters because 53% of the suite is gated on a.somerely existing, of any age: a stale one does not skip those tests, it runs them against old Rust. -
.pre-commit-config.yamlmoved to the repo root, wherepre-commitactually resolves it. It sat inpolars-cv/, so the hook was never loaded — whiletests/AGENTS.md, thepyproject.tomlmarker description and ten module docstrings all justified themselves with "the pre-commit hook enforces this". The config's own contents gave it away:cargo fmt --allanduv run --directory polars-cvare both written for the root. -
Guards that read as coverage and enforced less.
test_no_local_png_factoriesdocumented two fixture names and grepped for one; the unchecked half had drifted into eleven copies, each having dropped conftest's Pillow-missing skip. Three sweeps intest_schema_parity_chains.pydiscarded theirplan_or_rejectresults, and a fourth asserted only things its own types guarantee.test_verify_script_covers_every_ci_checksearched the rawverify.shincluding comments, so commenting out the structural lane passed both it and its sibling. TheMatcherprotocol was pinned byisinstance, which checks attribute presence and never signatures. Two guard-fixture modules were in the structural lane but pinned by neither lane mechanism. -
The remote source stopped opening one connection per file.
read_httpbuilt areqwest::Client— which is the connection pool — inside the per-file read, andread_s3/read_gcs/read_azureeach built a fresh object store per file. The 0.20.0 benchmark counted the result exactly: 1.00 requests per connection, so every image paid a TCP handshake and, off loopback, a TLS handshake. Under the streaming engine it paid them again on every morsel, because nothing scoped to a plugin call can outlive one.
The three cloud schemes now go through polars-io's build_object_store,
which keeps a process-wide store cache; http(s):// keeps its own pooled
client, because polars deliberately does not cache HTTP stores (that suits a
few large Parquet files, not many small images). Measured over three
successive plugin calls on 64 URLs: 16 connections for the first call — one
per concurrent worker rather than one per file — and none at all for the
calls after it.
Delegating rather than reimplementing is also what makes caching a
credentialed store safe, which is the question 0.20.0 measured but declined
to answer. polars keys the cache on the bucket and the credentials, and
exec_with_rebuild_retry_on_err rebuilds the store with cached credentials
cleared and retries once on any object-store error, swapping the new store in
through the Arc the cache holds.
- Fetch concurrency is a global budget, not a per-call constant. It was 16
per plugin invocation, and the streaming engine invokes the plugin once per
morsel across threads — so the real number of in-flight requests was (morsels
in flight × 16), bounded by nothing and settable by no one. Every request now
takes one permit from polars' own semaphore, so our fetches and polars' own
scans share one bound:
POLARS_CONCURRENCY_BUDGET, defaulting tomax(thread count, 10).
The chunked fan-out went with it. It spawned 16 scoped OS threads and joined
all of them before starting the next chunk, so the slowest file in a wave
stalled the wave; buffer_unordered starts the next file as soon as any one
finishes. fetch::DEFAULT_CONCURRENCY and the max_concurrency parameter are
deleted — a per-call knob beside a global budget would be a second authority
for one number.
- Six
chunks_exact(N)loops becameas_chunks::<N>(), whichclippy::chunks_exact_to_as_chunksrequires on stable 1.98. No behaviour change;slice::as_chunksis stable well below the crate's 1.96 floor.
Removed¶
-
DomainOp(view-buffer). A public trait with zero implementors, a doc example implementing it for a type that does not exist, and an entry inview-buffer/AGENTS.mddescribing it as a live part of the op contract. Domain dispatch actually lives onGraphStepin the plugin. It was also the only reasonops/traits.rsimportedNodeOutput— a graph concept reaching into the engine's trait module — so deleting it closed that too. Pinned bytest_removed_surfaces.py. -
match_detections(strategy=)on.contourand.bbox. TypedLiteral["greedy"], rode the wire asContourKwargs.strategy, and selected nothing — the greedy policy is unconditional in the sharedmatch_from_matrix. The contour entry point checked the value and discarded it; the bbox one did not check at all, sostrategy="hungarian"was accepted, ignored, and matched greedily. Guarded bytest_removed_surfaces.py.
Notes¶
-
Binary ops still refuse a runtime vector, and that is deliberate. The reduction fix above coerces a runtime
NodeOutput::Vectorto a 1-D buffer, but only for steps whose published output shape derives from that operand alone.Binarydeclares it acceptsvector— it must, becauseperceptual_hashplans as one — yet a phash executes as a buffer, so the only operands that ever arrive as a runtime vector areextract_shapeandlabel_reduce. Coercing those would broadcast against the other operand while the planner had already published the left operand's rank:extract_shape().add(image)would plan rank 1 and execute rank 3. It raises instead, naming the reason. -
rotate(expand=True)output dimensions can differ by one pixel from 0.20.0 for a small fraction of inputs (~1 in 5000 over a 200k-case sweep), now that the fused path reads the engine's matrix rather than recomputing it. The cause is precision, not convention:AffineParams::from_rotationtakesangle_deg: f32where the Python transliteration used f64. The new value is the one an unfused rotate has always produced, so this makes the two paths agree rather than changing what a rotate means. -
to_graph()now emitsshape_asserted: truewhere it emittedfalse, because the sub-pipeline it builds inherits_asserted_dimsalong with the rest of the state. It only selects which of two error stringsvalidate_output_schemaprints, and the new value is the correct one — it is what.cv.pipe()already emitted — butgraph_jsonis the compiled-graph cache key, so affected pipelines mint a new cache entry once. -
Ambient AWS configuration is now honoured:
build_awsreads~/.aws/configand~/.aws/credentialsfor a region and keys when the options do not supply them, and issues oneHEADper bucket to discover an unset region. Neither happened before, so a machine with an unrelated AWS profile may sign a request that previously went unsigned — passanonymous=Truefor a public bucket. - Store-build and read errors now carry polars' context.
polarsgains theaws/gcp/azurefeatures, which pull in polars-io'scloudstack (and transitivelyfile_cache,serde_json,rand).
0.20.0 — 2026-08-13¶
Added¶
- Every
.contouraccessor now takes a set of contours, not just one. A column may be aCONTOUR_SCHEMAstruct per row or theCONTOUR_SET_SCHEMAlist of them thatextract_contours()produces; the result is wrapped to match, soarea()returnsFloat64for the first andList(Float64)for the second, one entry per contour. Fifteen of the eighteen accessors calledparse_contourdirectly and rejected a set outright, which meant the namespace could not read the column its own pipeline produced.
Two-operand accessors (iou, dice, hausdorff_distance) broadcast: a
set on one side and a single contour on the other yields one result per
contour, whichever side the set is on. A set on both sides raises rather
than guessing — it could mean the N×M matrix (pairwise_iou) or an
index-wise pairing (.explode() one side), and those are different answers.
The set-level accessors (pairwise_iou, match_detections, label_reduce)
run the rule backwards through the same parse_contour_set repack, so a lone
contour is read as a set of one.
The arity is one value, read from the column dtype, and it drives both
halves of every accessor: src/geom_arity.rs wraps the declared element type
for output_type_func and the produced results with the same Arity::wrap,
and contour_accessor! emits both from a single -> <elem> declaration. That
matters because the two halves are otherwise unconnected — declaring
Float64 while building List(Float64) is exactly the plan/exec divergence
test_schema_parity_namespaces exists to catch, and per-accessor list
handling would have been fifteen chances to introduce it. The case table there
is now swept in both arities, so an accessor that skips the macro fails
rather than shipping a schema its data contradicts.
-
Pipeline.assert_shape(dims=[...])— the positional spelling, and the one that pins the output rank alongside the per-dimension sizes. This is what makes an assertion reach the published schema for alist/arraysource, whose rank is not knowable until execution: theheight=/width=/channels=form set the hints, butexpected_shapeonly publishes at rank 3, so the assertion changed nothing and thearraysink went on refusing the query. -
.contour.ensure_winding(direction=)and.contour.scale(origin=)accept a Polars expression, resolved per row like every other non-structural parameter. Per-row eligibility is decided by whether a value affects the output's shape, rank or dtype — not by its type — and neither of these does: the column staysList(Struct(CONTOUR_SCHEMA))whichever winding or origin a row picks. They were nonetheless literal-only whilelabel_reduce'sreductionandregion_mode, two enums of the same kind in the same namespace, were already per-row.
Worse, the rejection asserted the opposite of the truth — "'direction' is
structural (it fixes the output shape/rank/dtype at planning time)" — for a
parameter that fixes none of them. Both now ride _ArgBinder, like scale's
own sx/sy: the expression is appended as a plugin input, named in
input_slots, and read per row against the same NAMED table a literal is
validated against, so moving the check to execution does not lose it. Routing
them through contour_row also brings both under the shared null-parameter
policy, so on_null("null") now applies to them.
test_non_structural_geometry_enums_accept_an_expression reads the geometry
namespaces' live signatures, so a new non-structural enum has to be plumbed
for per-row use or exempted with the structural reason.
- The remote source has a benchmark. Every scenario in the suite was handed
bytes that were already in memory, so the
fetch.rs/cloud.rsstage — the one everys3://,gs://,az://andhttp://path goes through — had no coverage at all.benchmarks/scenarios/remote_source.pymeasures it against a loopback HTTP server, selectable from the regression suite as--scenarios remote.
Loopback rather than a real bucket because the transports differ in signing,
not in structure: each lands in cloud::read_file, which builds a client and
issues one GET per file, and each is batched by the same fetch::prefetch. A
wide-area measurement is dominated by the link and hides what the plugin
costs; a --latency-ms knob injects a synthetic delay for when modelling the
link is the point. Real S3/GCS numbers need credentials and a bucket, so
they cannot be a committed benchmark.
It reports three timings over one corpus whose differences isolate a stage,
and one thing that is not a timing: requests per connection, counted by
the server. On this code that ratio is 1.00 — a fresh connection for every
file, because read_http constructs a reqwest::Client inside the per-file
read and the three cloud backends each build a store per file. On loopback
that costs about 0.5 ms/file; against a real endpoint it is a TCP and TLS
handshake per file. Measured and reported rather than fixed: caching a
credentialed store is a correctness question before it is a performance one.
Fixed¶
assert_shaperejects a contradiction where it is written, instead of blaming the plugin atcollect().resize(224, 224).assert_shape(height=999)was accepted, published as the output schema, and reported much later byvalidate_output_schemaas "the planner's shape contract disagrees with the Rust implementation" — the plugin taking the blame for a value the caller had typed three lines earlier. An assertion that contradicts a dimension the planner already knows now raises at the call, naming both values and the op that established the known one. When a divergence does survive to execution, the message distinguishes the two cases: an output whose shape was asserted says so and points at the assertion, while an inferred one keeps the contract-bug wording, which is correct there. The two are told apart by ashape_assertedflag on the output spec, tracked per dimension and cleared whenever the schema fold recomputes it — therasterize(shape=<node>)canvas is deliberately not counted, since it comes from another node's inferred hints.
The height/width/channels keywords are also rejected once the rank is
known to be anything but 3. The hints are positional — they are dimensions 0,
1 and 2 — so after a transpose([2, 0, 1]) dimension 0 is the channel axis
and calling it height is a lie. dims= is the honest spelling there.
-
The
arraysink's error names remedies that exist. It advised.resize()/.assert_shape(), and the source that reaches it most often is alist/arraycolumn — whose shape genuinely is not knowable until execution, and which therefore got the same advice back after following it..resize()supplies height and width but never the channel count. The message now names the missing dimensions and what each of the three remedies actually supplies, and is written once per side rather than at four sites. The unreachable copy inencode.rsbecame an internal-invariant error rather than a fourth restatement of user advice, sincedtype_for_outputreads the same two fields and refuses first. -
.contour.ensure_winding()and.contour.scale(origin=)reject a value they do not recognise instead of guessing one. These were the only two user-facing string parameters in the plugin that did not read aNAMEDtable. Both parsed by hand and ended in_ => <default>, so a spelling the parser did not know was answered with a plausible one:ensure_winding("CW")returned counter-clockwise — the opposite of the request — andscale(origin="top_left")scaled about the centroid. Silently, in both cases; a capitalisation that does not match is not an exotic input.
ScaleOrigin and Winding now go through named_variants! + registry!
like every other enum, which is also what makes them checkable: they are
surfaced over enum_variants, covered by every_named_enum_is_registered,
and demanded by test_every_rust_enum_is_parity_checked — hence
_types.ScaleOrigin and _types.Winding, which replace the Literal[...]
annotations that used to shadow them. The long "clockwise" /
"counterclockwise" spellings the parser has always accepted are kept, as
aliases inside NAMED rather than a second table.
One behaviour change beyond the rejection: a contour_scale graph that omits
origin entirely now defaults to Origin rather than Centroid, matching
what the Python signature has always declared. Every pipeline built through
.contour.scale() already sent an explicit value, so this is reachable only
by hand-writing the plugin kwargs.
numpy_from_struct()no longer reads"u8"asuint64. Its dtype allowlist was hand-typed and admitted numpy's character codes alongside its spelled names. numpy reads"u8"as uint64 and"i8"as int64 — the opposite of what those strings mean everywhere else in this project, where they aredtype_table!'s short names for uint8 and int8. So a caller hand-building a struct withdtype="u8"— the spelling.cast("u8")uses — got a uint64 reinterpretation of the bytes: wrong values, and either a wrong shape or areshapeerror far from the cause. The function is exported from the package root and takes a plain dict, so this needed no compiled plugin to reach. The list also admittedbool/b1/f2, which the engine cannot produce.
The accepted names are now generated into python/polars_cv/_dtype_names.py
from dtype_table! by scripts/gen_dtype_names.py, guarded by
regenerate-and-diff. Character codes are rejected; the numpy/torch sink has
never emitted them.
- The contour struct layout is declared once, and checked against what the
engine emits. Three places spelled it out:
contour_struct_dtype()insrc/graph/encode.rs(the only one execution reads), the publicCONTOUR_SCHEMA, and a private copy inmetrics/_matching/_contour.pythat did not import the public one. A fourth site,tests/test_contour_source.py, shadowedCONTOUR_SET_SCHEMAwith a local alias of the constant it had just imported. The private copy and the shadow are gone;CONTOUR_SCHEMAremains, as the public mirror.
Nothing had been checking any of them. The two Python copies agreed with each other, which is not the property that matters — a field renamed or reordered in Rust would have left both still agreeing and both wrong, and the metrics matcher casts a real sink output to its copy, so the divergence would have surfaced as a cast error inside a detection pipeline rather than at the schema.
test_contour_schema_is_what_rust_emits runs an extraction through the plugin
and compares CONTOUR_SET_SCHEMA to the dtype Polars actually returned, so it
is behaviour-observed rather than two Python constants compared to each other.
It was watched failing against a deliberately reordered contour_struct_dtype().
- A
DetectionTable's dtypes are declared, not invented at the empty path.DETECTION_COLUMNSandIMAGE_META_REQUIREDlisted column names with no dtypes, so the only place the dtypes existed was a hand-written literal inside_empty_detection_table()— twelvepl.types that nothing connected to what a populated match returns. The two frames meet: an all-background image yields the empty table and a normal one does not, and a caller concatenating them would have discovered a disagreement at the concat rather than at the schema.
They are now DETECTION_SCHEMA / IMAGE_META_SCHEMA, name→dtype maps that
_empty_detection_table() builds from directly.
test_matcher_schemas_match_the_declaration runs ContourMatcher,
BBoxMatcher and PreMatchedAdapter on real inputs and compares each
result's schema to the declaration, so the declaration records observed
behaviour rather than an intention; a new matcher is added to _PRODUCERS,
which is what gets its output checked at all. Watched failing by widening one
declared dtype — the three matchers rejected it while the empty path, which
builds from the declaration, could not.
from_matched still validates names only. It accepts caller-assembled
frames, and rejecting a UInt64 det_idx that Polars compares happily
against a UInt32 would break working pipelines for no benefit; what the
matchers produce and what a caller may supply are different claims.
statistics()andstatistics_lazy()read one table instead of four. Each carried avalid_statsset and a five-armif/elifdispatch — the same list of statistics written out four times — and both dispatches ended inelse: continue. So the two halves could disagree in the direction that says nothing: a name the validator accepted but the chain did not know was dropped from the output silently, producing a struct with a missing field rather than an error.
Both now build their nodes through _stat_nodes(), which reads
_STAT_REDUCERS for the accepted names and the dispatch, so the two cannot
come apart. The methods differ only in the alias prefix and in whether they
sink, which is what they always differed in.
test_stat_reducers_are_all_pipeline_methods rejects an entry naming a
method that does not exist or one that is not a reduction — watched failing
for both.
_require_concrete_sink_dtype also stops spelling its own ("list",
"array"). That vocabulary is now SINKS_WITH_TYPED_ELEMENTS in _types,
beside the SOURCES_RESOLVED_FROM_COLUMN the same function already read from
there.
- "Channels from rank" is derived in one place. The mapping
3 => Some(shape[2]), 2 => Some(1), _ => Noneappeared character-for-character at three sites, each feeding the sameImageCodeccheck:execute.rs(the encoder),graph/decode.rs(the planner), andto_dynamic_imagein view-buffer'sinterop/image.rs— whose own comment said "a second copy here is how the two would come to disagree" while being exactly that.
Both now call ImageCodec::check_shape, which takes the shape and derives
the rank and channel count itself, so neither caller can derive them
differently. image_metadata.rs keeps its own, different mapping: it answers
.cv.channels() for buffers of any rank and its callers need a number rather
than a maybe, so it interprets ranks 0 and 1 that the codec check must reject.
A comment there now says so, rather than leaving it looking like a third copy.
- Three error/null policies stop being spelled by hand in Python, and
enum_variantsloses its last hand-written arm.("raise", "null")and friends appeared at five Python sites covering three different policies —RowErrorPolicy(3 values),NullParamPolicy(2) and thefile_path/read_bytesfetch policy (2). Over-listing a value errors loudly at the boundary; under-listing one rejects a value the plugin would have accepted, silently, forever.
The fetch policy was not even an enum: parse_on_error matched two string
literals and hand-wrote (expected 'raise' or 'null'). It is now
FetchErrorPolicy, and all three carry named_variants! tables mirrored by
_types.RowErrorPolicy / NullParamPolicy / FetchErrorPolicy, which the
five sites read.
Reaching them required lending the plugin crate the mechanism rather than
cloning its exception: named_variants! and registry! are exported, and
the plugin declares PLUGIN_REGISTRY, which enum_variants/enum_names
chain onto the engine's. Registering an enum is therefore the same act as
getting it checked — test_every_rust_enum_is_parity_checked reads
enum_names() and demands a Python mirror for each.
BinaryOp came along: its name table lived in the plugin crate purely by
habit, which is what made it the one enum needing a bespoke FFI arm and an
exemption from that test. BINARY_OPS is now BinaryOp::NAMED beside the
enum in view-buffer, registered like everything else — so a new BinaryOp
variant no longer compiles until it is named. The BINARY_OP_ENUM constant
and the enum_variants special case are deleted.
Two new Rust tests cover the one thing the parity test cannot see: these
policies have two mechanisms describing one vocabulary, serde's
rename_all reading the wire and NAMED publishing it.
row_error_policy_names_match_serde and its NullParamPolicy twin compare
them, so a rename on one side alone can no longer leave Python confidently
sending a value the graph cannot parse.
-
Four guards that only asserted an absence now also assert a presence. A test whose every assertion is "this string is not there" passes just as happily when the thing it guards has been renamed, moved or deleted — it stops distinguishing "the pattern is gone" from "the code is gone". Each was watched failing after the positive half was made to fail:
-
test_enum_validation_uniformnow builds each parameter with a real variant as well as a bogus one, and asserts that variant is oneenum_variantsactually publishes. Writing it caught a hard-coded"dhash"thatHashAlgorithmdoes not have. TestParamPolicyRatchetassertsexecute.rsstill mentionsresolve_usizeandparams::getbefore ratcheting against two idioms that use them. Renaming the resolver used to leave the scan green.test_domain_vocabulary_declared_oncenow checks that a wrong-domain op still raises and that the reported domain is a_types.Domainmember. Deleting the domain check entirely used to pass every assertion.-
test_anti_alias_is_gone_from_the_type_stubrequires the stub to declarerasterizefirst — an emptylazy.pyiused to satisfy it. -
The documentation is built by CI.
docs.ymlwasworkflow_dispatch:-only and ranmkdocs build --clean, so nothing checked the docs on a push or a PR and a warning did not fail the build. A broken link or a nav entry pointing at a deleted page was found by whoever next deployed by hand, if at all.
A docs job in ci.yml now runs mkdocs build --strict on every push and
PR, the deploy workflow uses --strict too (so it cannot ship a site CI would
reject), and scripts/verify.sh runs the same command — which
test_verify_script_covers_every_ci_check now requires, so the three cannot
come apart. The job is in ci.yml deliberately: that test reads only that
file, so a check placed anywhere else is one it cannot see. Watched failing
against a nav entry naming a missing page.
docs/panic-audit.md joins the nav under a Development section. It was the
one page in docs/ absent from nav, i.e. built but unreachable from the
site. (Not a --strict blocker, as it happens — mkdocs reports an
out-of-nav page at INFO, not WARNING — but a page nothing links to is not
published in any useful sense.)
- The examples are executed, and two of them were broken.
polars-cv/examples/held fourteen files that nothing ran: no test, no workflow, andruff'sextend-excludeskips the directory. They read as documentation while being unverified for the life of the directory.test_examples_runexecutes each as a subprocess and requires exit 0, markedslowand discovered throughtests/_discovery.pyso an empty sweep fails rather than passes.
Its first run found two independent bugs in 08_ml_integration.py:
.assert_shape(channels=3, height=96, width=96)was applied after.transpose([2, 0, 1]). Those keywords describe an[H, W, C]buffer, so asserting them on an already-transposed one made the planner publish[96, 96, 3]for something that executes as[3, 96, 96]— the query died atcollect()with a plan/exec mismatch. The assertion moves before the transpose, where it means what it says.- The
arraysink was given noshape=, on the assumption that.assert_shape()supplies one. It does not, and neither does.resize()— only.sink("array", shape=[...])does, which the example now passes.
matplotlib joins the dev group: ten of the thirteen runnable examples plot,
and without it those ten would not skip but fail, since the runner asserts
exit 0.
- The documentation's vocabularies are checked against the code's. Three
places in
docs/restate something the code owns, all hand-maintained and none compared to anything:domains.md's table (now required to equal_types.Domain),sources.md'sauto-resolution table (its formats must exist inSourceFormat), and the methods called in the operations pages' Python examples (each must resolve to a realPipeline/LazyPipelineExpr/ namespace method).
The last one is not the table-diff it looks like it should be: those pages
name operations in prose headings, not tables, so a "rows ⊆ OP_NAMES" guard
would have matched nothing and read as green forever. It reads the code
blocks instead — what a reader actually copies. Its _FOREIGN_METHODS
exemption list is itself guarded, and immediately caught three of our own
methods wrongly exempted while it was being written.
The Markdown readers live in tests/_doc_tables.py with committed fixtures
in tests/test_doc_table_fixtures.py, which caught a lookbehind that
excluded df.method( — the ordinary shape of a call, i.e. most of every page.
- The structural guards run before a commit, not just in CI. They check the shape of the codebase — registry parity, single-authority ratchets, removed-surface pins, the doc vocabularies — and read the source tree rather than exercising a pipeline, so the whole lane is ~650 tests in about five seconds. That is affordable in a pre-commit hook; the full suite is not.
-m structural is declared in pyproject.toml and applied per module. A
structural-checks hook joins .pre-commit-config.yaml (which previously
stopped after cargo clippy), the lint job runs the lane without waiting
on the build matrix, and scripts/verify.sh runs it as its own check so a
failure says "the hook would have caught this" rather than arriving buried in
the full suite's output. The hook passes --no-sync: without it uv run
re-syncs the project and rebuilds the Rust extension on every commit.
Two guards keep the lane from becoming decorative.
test_every_source_scanning_module_declares_its_lane derives its set —
anything importing _discovery, the mandated way to find files to scan, must
declare structural or slow — so a new source-scanning guard cannot join
without choosing. test_the_structural_lane_is_not_empty catches a renamed
marker or a dropped declaration, which would otherwise make the hook select
nothing and pass in milliseconds.
- A check added to CI can no longer be missing from
verify.shunnoticed.test_verify_script_covers_every_ci_checkpinned a hand-written list in both directions, but a check added to CI and never added to that list was simply invisible: the local run kept reporting PASS while CI failed.
test_no_ci_check_is_missing_from_the_verify_script reads every command in
ci.yml's run: blocks and requires each to be classified — either a
check verify.sh must also run, or setup. An unclassified command fails, so
a new checker cannot pass unnoticed. It proved itself immediately: adding the
structural lane to CI made it fail until the command was classified.
-
scripts/verify.shno longer rebuilds the extension in release mode. Themkdocscheck added above was written without--no-sync, souv runsynced the project on every verification — and syncing builds the plugin through the PEP 517 backend, which maturin runs at--profile release. That is a multi-minute re-optimisation of the whole polars stack, on every run, for a documentation build that never loads the extension. With--no-sync(what every other lane in the script already passes) the check takes about six seconds. -
Review pass: five guards that could pass while checking nothing, and six false statements. An independent review of the entries above found that the weakest part of the work was the part meant to be strongest.
Guards that could be evaded, each now watched failing against the exact evasion:
test_no_ci_check_is_missing_from_the_verify_scriptread only therun: |block form, socargo test -p view-buffer— a checkci.ymlhas run all along — was invisible to it. It also classified a whole line by its first token, socd polars-cv && deno lintfiled undercd, and exempted everyuv run python scripts/check_*.pyas setup. It now reads bothrun:forms, splits on&&/;/||, and has no blanket script exemption.- The same guard compared
verify.shas one string, so a comment naming a deleted check satisfied it. It now reads onlyrun_checklines. _module_markswalked the whole AST, so apytestmarknested in a class counted as a module declaration: moving one cut the lane from 651 tests to 547 while both lane guards reported it healthy.- The lane's "derived" set was a substring match on
"_discovery import", whichfrom tests import _discovery as _ddefeated. It is parsed now. test_the_structural_lane_is_not_emptycompared a count against a threshold with a margin of one. It names the load-bearing modules instead.- Nothing read
.pre-commit-config.yamlat all, though the hook is the premise for the marker on ten modules.test_the_precommit_hook_runs_the_structural_lanedoes.
And the exemption lists, which are how a guard gets quietly narrowed:
_FOREIGN_METHODS is now empty — once the resolution learned about the
metrics API and module-level pl.col, all 34 entries turned out to be
resolvable or absent from the docs — and _NON_SCRIPT_EXAMPLES is guarded
against exempting anything with a __main__ block.
-
test_benchmark_list_is_currentnow exists.benchmarks/AGENTS.mdcited it as pinning the benchmark list. It had never been written — a guard's name in prose, which is the failure this whole effort is against. It exists, it readsget_single_op_benchmarks(), and it was watched failing on all three ways that sentence has been wrong: a wrong count, a drifted name, and a benchmark added to the code. -
A plugin-crate
named_variants!table left out ofPLUGIN_REGISTRYnow fails. Exporting the macro gave the plugin the ability to declare vocabularies without giving itevery_named_enum_is_registered, which scans view-buffer only. An unregistered enum was invisible toenum_variants, toenum_names()and so to the parity test — 128 tests passed with one sitting there.every_plugin_named_enum_is_registeredis the missing half. -
_STAT_REDUCERSis pinned by value, not just by shape. The existing guard checked that each entry resolves to a reduction; it could not tell one reduction from another, so"mean": "reduce_max"— the plausible copy/paste error in the refactor that removed the explicit dispatch — passed the whole suite, because the fixtures paint constant images where mean equals max.test_each_statistic_computes_its_own_reductionuses a gradient on which all five differ. -
The documented-method sweep covers every page. It read five hand-picked pages of twenty-one, because
pl.colandpl.DataFrameare module-level and the resolution only knewpl.Expr/pl.DataFrame, makingcolunresolvable almost everywhere. The fence reader also missed indented fences (admonitions, tabs, list items) and annotated ones (title=,hl_lines=) — both formsmkdocs.ymlenables — so examples inside them were never checked. Fixtured. -
statistics(include=<generator>)raisedIndexErrorinstead of a message._stat_nodesiteratedincludetwice, so a one-shot iterable was exhausted by the validation pass and produced no nodes. Off the declaredlist[str] | None, but a rawIndexErrorfrom a private helper is not an answer. It is materialised once now. -
The
zero_copybenchmark scenario could never run. It is offered inALL_SCENARIOSand named inbenchmark.yml's input description, but selecting it died in_aggregate_bestwith'BenchmarkResult' object has no attribute 'framework'. The scenario returned its ownBenchmarkResult— a different record that happened to shadowbenchmarks.frameworks.BenchmarkResult— and_run_onceappended those to a list of these without converting. Nothing ever exercised it.
The local record is IngestionResult now, so two records meaning different
things read differently, and to_suite_results is the one conversion between
them. _run_once rejects any result that is not the suite's own record, so a
scenario that skips the conversion fails at the boundary naming the scenario
rather than several frames later naming a missing field.
-
The benchmark workflow measured a binary nobody installs.
ci.ymlandpublish.ymlboth clearRUSTFLAGSso.cargo/config.toml'starget-cpu=x86-64-v3is not baked in;benchmark.ymldid not. It was therefore timing an AVX2/BMI2/FMA build while published wheels are built without those, so its numbers described a configuration no user runs. Cleared, with the same comment the other two carry. -
A release tag is checked against the version being published. maturin reads the version from
polars-cv/pyproject.toml; a tag is only a label on a commit, and nothing compared the two. A release taggedv0.20.0cut before the bump landed would have uploadedpolars_cv-0.19.0under it, or built every wheel and then failed at the upload with PyPI's "file already exists". Acheck-versionjob runs first, and both build jobs need it.
It reads one manifest rather than re-comparing all four: the other three are
pinned to pyproject.toml by test_version_consistency.py, which ci.yml
runs on every push to main, so restating that comparison in the workflow would
be a second authority for one fact. A workflow_dispatch run has no tag to
check against, so it warns rather than skipping silently.
- Documentation that described code as it is not.
geometry.mdstill calledscale'soriginandensure_winding'sdirectionliteral-only "structural parameters" after both became per-row expressions, and omittedlabel_reduce's two enums from the list of expression-capable parameters. It said nothing about every.contouraccessor now reading a contour set.image-ops.mddocumented only theheight/width/channelsspelling ofassert_shape, notdims=— the one that pins the rank, and so the only one that gets alist/arraysource to a fixed-shapearraysink.CONTRIBUTING.md's release steps claimed GitHub Actions publishes to TestPyPI and tests the install from there;publish.ymlhas never done either.
Four remaining recommendations of maturin develop --release for the develop
loop — in build_info()'s docstring and three test_version_consistency.py
messages — now say maturin develop, finishing what the earlier docs pass
started. The benchmark docs keep --release, which they need.
0.19.0 — 2026-08-09¶
Added¶
- Operation parameters that no code path reads are now rejected.
OpSpecis the documented exception to this crate'sdeny_unknown_fieldsrule — its params ride on#[serde(flatten)], which serde cannot combine with unknown-field rejection — so the wire format could not refuse a parameter no operation understood, and nothing else did either. That is howscaleandclampcarried anout_dtypefor releases.
Every parameter read now goes through OpParams, a tracking wrapper, and
resolve_op rejects any name the arm did not touch. resolve_op_inner takes
the op name and the wrapper, never the OpSpec, so an arm cannot reach the
underlying map and read without recording — the compiler found two of crop's
reads doing exactly that when the signature was narrowed. Reads are recorded
in a bitmask over the map's key order, so tracking allocates nothing on the
per-row path. It fires at both ends of the boundary with no extra wiring:
resolve_op_from_json backs op_schema, so a stray parameter fails in Python
while the Pipeline is being built, and CompiledGraph::compile resolves
every spec before any row executes. OpParams::acknowledge is the explicit
escape for a parameter a layer above the arm consumes (rasterize's
shape_ref) or that a branch legitimately cannot use (rotate's
interpolation and border_value on the lattice rotations, which resample
nothing).
- A schema-parity matrix: plan-time dtype == execution-time dtype, swept.
Polars publishes an expression's dtype at planning time and again when the
data arrives, and in this plugin the two are computed by different code —
dtype_for_outputfrom the hint bundle Python folds into the graph JSON, andbuild_series_from_specfrom the buffers execution actually produced. The runtime half reads the data to decide: the typed list builder takes both the leaf dtype and the nesting depth from the first non-null row, andresolved_output_specsresolves every output's"auto"dtype frominputs.first(). Whether that agrees with the plan depended on which row happened to be first.
Six new files under tests/test_schema_parity_*.py, on a shared harness
(tests/_schema_parity.py) that asserts the planned dtype against the
produced one under both engines, streaming first, and checks the engines
agree. Rejection at build or plan time is an acceptable outcome for any cell;
planning succeeding and execution then failing or producing something else is
not. Every sweep carries an assert_not_vacuous check, because a sweep whose
cells all reject is green and worthless.
The axes:
- Row layout. Nine patterns — all-null, null-first, null-last, a null run
longer than a morsel, 64 alternating rows across a streaming batch
boundary, and rows whose shapes differ from one another, chunked so the
boundaries do not line up with the nulls. This is the axis that
distinguishes "the plan" from "row 0".
- Operations. All 71 chainable ops, driven from tests/_op_cases.py —
the table test_op_case_table_is_complete already pins to
_chainable_pipeline_ops(), moved out of test_append_contract.py so both
read one copy. An op cannot join the library without getting a
plan-vs-exec cell. Roughly forty had none before, including every
reduction, all four morphology ops, every colour conversion, and all five
histogram output modes.
- Sources and dtypes. Every SourceFormat (including auto, the
default, which had no plan-vs-exec test at all) and all ten DTypes; only
four were previously compared against data. Channel counts 1/2/3/4,
including the 2-channel GrayA that StripProcessRestore produces and
that nothing had ever sunk.
- Chains. Composed effects asserted at every prefix, the steps
op_infer_shape refuses, all eleven binary ops (read from Rust's
BINARY_OPS registry), CSE-shared prefixes, two-root graphs, and affine
fusion.
- sink("array"), asserted in both directions: a knowable shape must be
accepted and exact, an unknowable one refused while planning. Its
dimensions are cross-checked against the shape the numpy sink reports for
the same pipeline — two independent producers of one fact.
- The namespace accessors. .cv, .contour, .point and .bbox are
separate #[polars_expr] functions declaring their own output types, a
second plan/runtime pair with no parity test — a gap
geometry/AGENTS.md flagged. The case tables are completeness-asserted
against the namespaces' real public methods.
Three files carried their own copy of the plan-vs-exec assertion
(_assert_plan_matches_data, _planned_and_realized,
_assert_plan_equals_exec); they now call the harness, and pick up the
streaming engine in the process.
Four divergences found, and fixed (see Fixed below).
Also recorded, without a bug marker: expected_shape is gated on rank 3, so
a rank-1 or rank-2 pipeline can never auto-shape an array sink even when the
planner has the dimensions. That is a deliberate conservative choice —
publishing [H, W, C] for a rank-2 output is how channel_select once
declared a schema execution could not produce — and the test asserts both the
refusal and that the withheld dimensions were correct.
allowed_roots: a sandbox for path columns.source("file_path", allowed_roots=[...])and.cv.read_bytes(allowed_roots=[...])restrict which locations a path column may read from. Until now neither surface sanitized paths — they read whatever the column named, any local file and any reachable URL, which is right for your own paths and wrong for paths that arrived as data.
Opt-in. The default is unrestricted, so every existing pipeline is
unaffected and an unrestricted source serializes byte-identically to before
(graph_json is the compiled-graph cache key, so an always-emitted field
would split the cache for everyone and change nothing else). Once you ask
for a sandbox it denies by default: a path matching no entry is refused.
One list covers local and remote — an entry that parses as a remote URI is
matched as a URI prefix, anything else as a local directory. Splitting them
is how a sandbox comes to cover the filesystem while leaving s3:// open.
Local paths are canonicalized before comparison, so .. segments and
symlinks out of the tree are resolved rather than compared as text, and
matching is component-wise, so /srv/images does not admit
/srv/images-private. A refusal is an ordinary read failure as far as
on_error is concerned, so on_error="null" nulls those rows.
The check lives in fetch.rs, the one stage the two surfaces share, and
both prefetch and row_bytes take the policy as a required argument —
a caller that forgets it fails to compile rather than silently reading
unrestricted. Denied remote paths are dropped before the fetch, so a refusal
never becomes a network request.
Fixed¶
scale(out_dtype=...)andclamp(out_dtype=...)were accepted and then discarded. The parameter was validated, serialized into the op's params — so it entered the op's identity, and therefore the CSE and compiled-graph cache keys — and read by nobody: noresolve_oparm looked at it, andoutput_dtype_forhonours anout_dtypeoverride only for theConfigurabledtype rule, which isnormalize's, not theirs (PromoteToFloat). A plan-vs-exec test could not see it, because plan and execution agreed; they agreed on the promoted float rather than on what was asked for.
Both now honour it, by lowering the request to a trailing cast — the
mechanism preserve_dtype=True already used, and the one kernel fusion
already handles exactly once (try_fuse pins FusedKernel::out_dtype to what
the unfused chain would have produced). Giving the ops their own output dtype
instead would have meant teaching extract_ops and that pinning about it, and
would have turned PromoteToFloat — which preserves f64 input — into a
Configurable(F32) that silently downgrades it. out_dtype and
preserve_dtype are now one mechanism with one target, in
_out_dtype_target / _apply_out_dtype.
-
warp_affinesilently substituted the identity for a singular matrix. Warping is inverse mapping — for each output pixel, ask where it came from — so a matrix collapsing the plane onto a line or a point has no answer. The runner returned the input unchanged and reported success, so a caller who passed a degenerate transform got their image back with no signal. It is now rejected where the user supplies it, naming the determinant.AffineParams::is_invertibleis the single authority, read by both the rejection and the runner'sdebug_assert. The check is skipped under a plan-time probe, where every expression parameter is bound to the same placeholder and a per-row matrix is singular by construction; per-row values are checked per row, where they exist. -
The channel rule had two readings. Python's
_update_channels_from_rulere-derived the output channel count by parsing the stringified rule, and disagreed with view-buffer'sOutputChannelRule::applyonNotApplicable:applyreturns "no channel count", Python left the hint unchanged. It stayed invisible because everyNotApplicableop also drops below rank 3, where_drop_hints_below_rankclears channels anyway — excepthistogram(output="quantized"), which preserves rank and was mislabelledNotApplicablewhile actually preserving channels. Two errors cancelled.
Quantized now declares PreserveChannels (its own comment already said so),
and Python reads apply through a new op_output_channels FFI instead of
holding a copy of the arithmetic. _drop_hints_below_rank stays: it is a
separate, correct invariant, not the mitigation it was doubling as.
- The contour round trip did not close:
source("contour")could not read a contour set.extract_contours().sink("native")emitsList(CONTOUR_SCHEMA)— one set per row — but the source parsed a singleStructper row and read a list as one contour's ring of points. Feeding the sink's own output back in therefore failed withPoint struct missing 'x' field, the contour struct'sexterior/holes/is_closedbeing read as a point'sx/y. The asymmetry was known and worked around intest_schema_parity_sources.py, which fed back a single element of the set rather than the set.
The source now reads both shapes, through one parser (parse_contour_set).
The two list forms are told apart by element dtype — a List of point structs
is a ring, anything else in a List is a set — rather than by trying one and
falling back, because the fallback is what turned a legitimate contour set
into a complaint about points. Both shapes are now swept by
test_schema_parity_sources.py::test_contour_source.
- Rasterizing more than one contour with
fill_value < backgroundreturned an all-background canvas. The graph executor rendered each contour to its own mask and folded them withmax, which is only union-like when the fill value is the larger of the two:rasterize(fill_value=0, background=255)silently erased every region as soon as the contour set held two of them (one contour was fine, which is why it survived). Sequential painting into one canvas would have swapped that for a different fault — a later contour's hole eating an earlier contour's fill, making the mask depend on the set's order.
rasterize now takes the whole set, resolves coverage as a union first and
colours it once, so neither fault is reachable and the single-contour case is
the one-element case rather than a second implementation. The
contour source and the rasterize op share it, so a mask no longer depends on
which of the two routes produced it — asserted by
TestMaskContourRoundTrip::test_the_two_routes_to_a_mask_agree.
.sink()accepted any keyword at all, and dropped the ones its format does not read. The sink takes**kwargsand spreads them into the graph'sSinkSpec, sosink("jpeg", qualtiy=50)built a graph carryingqualtiy, serde dropped it as an unknown field, and the query encoded at the default quality with nothing said. Of the three real keywords onlydtypepoliced where it applied:qualityon a png sink andshapeon a png sink were accepted and ignored.
quality turned out to be jpeg-only. SinkSpec documented it as "JPEG and
WebP" and the sink docstring said "jpeg/webp", but the WebP arm of
encode_image calls an encoder that takes no quality argument — so a webp
quality has always been silently discarded. It is now rejected rather than
accepted-and-dropped; supporting it is an encoder change, not a parameter
change.
Both ends of the pipeline now answer this question from one place:
SOURCE_PARAM_APPLIES and SINK_PARAM_APPLIES in _types.py, next to the
specs they describe, read by one reject_inapplicable_params. A name that is
not in the table is rejected too, which is what closes the open **kwargs.
Rust's SinkSpec gained #[serde(deny_unknown_fields)], the same mechanism
GraphNode already uses, so a hand-built graph cannot carry a sink field
nothing reads either.
The quality declaration is checked against the encoders rather than merely
asserted: for each image sink, whether the output changes between quality 10
and 95 must equal what the table claims.
-
Three Python spec classes nothing referenced.
_types.SinkSpec,OutputSpecandMultiSinkSpecwere unreachable — the sink is built from raw kwargs in_graph.py, and nothing in the package or tests names them. They also carried a fourth, stale copy of the applicability fact (if format == JPEG or WEBP: result["quality"]), wrong about WebP in exactly the way the docstrings were. Deleted, with a guard intest_removed_surfaces.py. -
Source parameters that do not apply to the chosen format were silently ignored, and each one policed itself differently.
Pipeline.source()takes eleven keywords, and most apply to a subset of the eight formats. Of the seven with a limited scope, one raised (decode_max_size), one warned (cloud_options), and five were dropped without a word:width,height,shape,fill_valueandbackgroundoutside"contour",require_contiguousoutside the nested-column decode,allowed_rootsoutside a path read. Sosource("image_bytes", width=224)built a pipeline that ignored the size, andsource("contour", allowed_roots=[...])accepted a sandbox it never applied.
Applicability now has one authority — _SOURCE_PARAM_APPLIES, each parameter
against exactly the formats whose decode reads it — and one check, which
raises. _is_supplied compares against the signature's own defaults, and
source() hands the check locals() rather than a hand-written list, so
the parameters it validates are the parameters it has.
Guarded three ways: the table's keys must equal source()'s keywords, so a
new parameter cannot inherit "applies everywhere" by omission; the check must
read locals(); and the full parameter x format grid is swept, every
applicable pair accepted and every inapplicable pair rejected. thumbnail()
reads the table too — it writes decode_max_size, and had drifted into
refusing an "auto" source that source(decode_max_size=...) accepted.
Two behaviour changes: cloud_options on a non-path source now raises
instead of warning (and its warn-and-drop branch, unreachable once the check
ran first, is gone rather than left behind) (a warning is filtered out of a query's output and the
credentials do nothing either way), and dtype on a contour source is
rejected as described below. test_cloud_options_on_non_file_path_source_warns
was rewritten to pin the rejection, and the decode_max_size and
thumbnail() message assertions now match the shared wording.
source("contour")published a rank and nothing else, then dropped an asserted dtype. The source decodes by rasterizing, so what it hands the first op is what therasterizeop hands its successor — an[H, W, 1]u8 mask. It stated a hand-written_expected_ndim = 3and left the dtype"auto", the channel count and the canvas unstated, whileGeometryOp::Rasterizehad declared all four all along. Both typed sinks need a concrete element dtype, sosource("contour").sink("list")was unplannable and a no-op.cast("u8")was the only way through;sink("array")additionally demanded an explicitshape=for a canvas fixed by the source's ownwidth/height.
The source now folds GeometryOp::Rasterize's contract through the same
op_contract / op_schema / op_infer_shape FFI an appended op goes
through, so the two routes to a mask publish one contract
(TestContourSourcePlanTimeContract). source("contour", dtype=...) is
rejected rather than dropped: rasterizing fixes u8, and the parameter reached
SourceSpec.dtype and stopped there — an asserted "f32" bought a u8 column.
rasterize(width=, height=)published no shape, though its docstring saidinfer_shapesupplied one. The planner declines to callop_infer_shapewhen the input rank is unknown, and the contour domain has no rank — so the one op whose canvas is fixed entirely by its own parameters was the one op never asked._input_dims_fornow recognises a step that builds a buffer out of a non-buffer domain (from the contract:input_domainsexcludes buffer,output_domainis buffer) and asks with no input dims, which is what a contour input actually is. Ops that consume a buffer are unaffected, and the contour measures — whoseinfer_shapedescribes one value per contour, not the vector's length — stay out of it by producing a vector rather than a buffer.
op_infer_shape on a rasterize(shape=<node>) also reported a 1x1 canvas as
fact: introspection has to supply width/height for the op to resolve at all,
and the placeholders were literals, so they read as fixed across probes. They
are expression placeholders now, and the dimension reads as unknown
(TestOpInferShapeAuthority::test_rasterize_by_shape_reference_is_unknown).
- The runtime series builders read the data to decide the schema.
build_typed_list_series_from_rows_with_dtypetook both the element dtype and the nesting depth from the first non-null row, using the planner'sexpected_dtype/expected_shape/expected_ndimonly as a fallback when every row was null; the array builder likewise fell back to the first row's shape. That inverts the contract — the spec is whatcollect_schema()already promised — and it made the outcome depend on where the nulls fell: an all-null column honoured the plan while the same pipeline with one value in it did not.
The spec is now the authority in both builders. A row whose data contradicts it is an error naming both dtypes rather than a silent reinterpretation, and the array builder has no data fallback at all: a fixed-shape column whose dtype depends on which row arrived first is the thing that sink exists to rule out.
- An
"auto"output dtype was taken from the input column, not folded through the ops.resolved_output_specsassigned the input column's leaf type straight to any output still marked"auto", which is correct only for a preserve-input lineage:source("list")over aList(UInt8)column followed byscale()plannedList(UInt8)for data that arrives f32. The per-rowvalidate_output_schemaguard caught it, so it surfaced as a mid-collect()failure rather than a wrong column — planned wrong either way.
The dtype is now folded through the ops' OutputDTypeRules from the resolved
source type by fold_output_dtype, the exact twin of the fold_output_rank
that sits beside it and had been doing this for the rank all along.
- A partially-resolved dtype is no longer discarded. Folding a rule over an
unknown input used to collapse back to
"auto", throwing away real information: whatever a PNG or TIFF turns out to decode to,PromoteToFloatof it is a float.PlannedDType(view-buffer,core/dtype.rs) is the three-state lattice —Known(DType),SomeFloat,Unknown— andOutputDTypeRule::resolve_plannedis the symbolic twin ofresolve.
SomeFloat is deliberately not a dtype: PromoteToFloat yields f32 for
integers and preserves f64, so there is no single answer, and inventing one
is a lie the runtime guard would catch. It is enough to decide a codec
though — ImageCodec::check_planned refuses only when every candidate
fails — which closes the last plan-then-fail case:
source("image_bytes").scale(...).sink("jpeg") is now a planning error
instead of an encoder error, while the same source without a promoting op
still plans, because it could still decode to u8.
Every site that compared a dtype string to "auto" now asks
PlannedDType::parse(...).is_concrete() instead — a hand-written comparison
would have let the second sentinel through to dtype_str_to_polars's u8
arm.
- A multi-root graph resolved every output's dtype from the first column.
merge_pipeand the binary ops join twopl.col()lineages into onevb_graphcall, andresolved_output_specsfilled in each output's still-"auto"element dtype frominputs.first()regardless of which branch the output belonged to. Two list columns of different leaf types plannedStruct({x: List(UInt8), y: List(UInt8)})and executed withyas Float32 — caught by the per-rowvalidate_output_schemaguard, so it surfaced as a mid-collect()failure rather than a wrong answer, but planned wrong either way. Two separate expressions never showed it: each is its own plugin call with its own single input.
Each output is now resolved against the column its own lineage reads, found
by walking upstream to a root and reading column_bindings.
source("auto")over a Binary column planned a rank it could not know.autoleaves_expected_ndimunset, and thelistsink waived the unknown-rank error for it on the theory that Rust resolves the rank from the column type. That holds for a List/Array column, whereresolved_output_specsfolds the nesting depth through the op rank rules, and not for image bytes, whose rank is only settled by the decode.dtype_for_outputfell through to a depth-1List(UInt8)while execution producedList(List(List(UInt8))).
It now refuses instead, naming the two ways out (name the source, or use a
sink that does not encode the rank in its dtype). auto over List/Array
columns is unaffected.
This changes an accepted query into a planning error.
source("auto", dtype=...) into a list sink over a binary image column was
accepted before; it was never correct, and eagerly it happened to return a
column, which is why test_auto_image_list_sink_with_explicit_dtype passed
while asserting only that the column came back. An explicit dtype settles the
element type, not the rank.
- The image-encoder sinks checked their preconditions in the encoder.
png,jpeg,webpandtiffeach restrict the buffer's dtype, and all four need an image-shaped buffer — facts that follow from the buffer's description, which is exactly whatOutputSpeccarries. Checking them at encode time meant a pipeline that promotes to f32, or flattens to rank 1, planned asBinaryand then died part-way throughcollect().
ImageCodec::check_support (view-buffer, interop/image.rs) is now the one
table: read by dtype_for_output when the query is planned, by encode_sink,
and by the encoders themselves — encode_tiff's fallback arm asks it for the
message rather than writing a second one. An unknown is never a rejection, so
a source that could still decode to u8 is not refused — but "unknown" is now
a lattice rather than a single state, so a partially resolved dtype is not
wasted (see below).
Guarded in Rust by the_table_never_promises_what_an_encoder_cannot_deliver,
which runs every (codec, dtype, channels) cell through both the table and the
real encoder and requires that the table never admits what the encoder
refuses. Two cells where it is deliberately stricter — 16-bit JPEG and WebP,
which image accepts by silently halving the bit depth — are listed with
their reason and separately pinned.
-
Three vector-domain sinks planned one thing and executed another. The two halves of the sink contract keyed on different facts:
dtype_for_outputdecides the Polars dtype from(expected_domain, format), whileencode_node_outputdecided the value from the runtimeNodeOutputvariant. Those are not the same thing — a domain can arrive in more than one representation. A perceptual hash is avector-domain output that rides as aBuffer(apply_perceptual_hashreturns a 1-Du8buffer);extract_shapeproduces a realVector. Wherever the two diverged, so did the contract: -
perceptual_hash().sink("native")plannedList(UInt8)and failed atcollect()with "Buffer outputs require explicit format" —nativeon a vector output is the documented spelling. extract_shape().sink("array", shape=[3])plannedArray(Float64, 3)and failed with "Unsupported sink format: array". The schema arm for("vector", "array")had been added to fix an earlier divergence without the encode arm that makes it real.perceptual_hash().sink("array")reported a shape mismatch against the buffer rather than encoding the hash.
The pairs that did work did so because the two dispatches happened to agree,
not by construction. encode_node_output now keys on the planned domain —
the same key its counterpart uses — and its arms mirror that function's one
for one; the NodeOutput variant is used only to reach the data, which is
what it actually tells you.
Guarded by tests/test_sink_contract.py, which sweeps every
(domain representation × SinkFormat) pair and asserts the relationship
rather than a blessed list: a pair rejected at plan time is fine, but one
that survives planning must execute to exactly the dtype planning promised.
Both axes are completeness-asserted, and the two vector representations are
pinned to encode identically.
Changed¶
-
Declaring a
named_variants!table and registering it are one act.every_named_enum_is_registeredfails if an enum declares aNAMEDtable without appearing innaming::REGISTRY. Registration is what surfaces an enum overenum_variants, what gets its names checked for duplicates, and what puts it inenum_names()— which the Python suite iterates to decide what to parity-check. An unregistered enum was therefore not "private", it was unchecked, with a Python mirror free to disagree with it. The check runs in both directions, so a scan that stops matching fails rather than passing vacuously. -
SourceFormatis pinned to Rust'sKNOWN_SOURCE_FORMATS. They are two hand-written spellings of one vocabulary, and a note in the test suite claimed there was nothing to pin them to — true of sink formats (which are matched on and rejected by fall-through, never enumerated), not of sources, which the graph validator checks against an explicit list. A Python-only format built a graph the validator rejected at execution; a Rust-only one was a decode path nothing could reach. -
op_contractpublishes each fact once. It carried bothinput_domain(a singleDomain) andinput_domains(the accepted set) across the FFI; only the set was ever read, and the two were free to disagree the moment a step accepted more than one domain — which binary ops and reductions do. The singular key is gone andtest_contract_publishes_no_second_spellingpins the key set in both directions. -
The graph wire format is closed at the node.
GraphNodenow carries#[serde(deny_unknown_fields)], so a stale or misspelled key fails loudly instead of being silently dropped — which is howshape_hintswent on being emitted long after its last reader. Fields only Python consumes (domain,output_dtype) are declared on the Rust struct to keep it closed.OpSpecis deliberately not closed: its parameters ride on#[serde(flatten)], which serde documents as incompatible withdeny_unknown_fields.
This is a behaviour change under version skew. A newer Python emitting a
node field an older compiled .so does not declare now fails the query
rather than dropping the field silently. The install is editable, so a stale
extension is the normal state after pulling Rust changes — re-run
maturin develop, or check polars_cv.build_info(), which reports the three
versions that must agree.
-
Two
unreachable!()panics removed frombuild_plan. They were reachable by an op author declaringMemoryEffect::Viewon a compute or image op — a runtime abort for a contract mistake. The materialisation decision is now a direct comparison that treats aViewdeclaration as "no materialisation needed", which is what it means. -
Binary ops and reductions declare that they accept a vector. Their
GraphStepcontract saidbuffer, which read as "images only" and was wrong: a perceptual hash is a 1-Du8buffer encoded as avector, and the library's ownhamming_distanceishash_a ^ hash_b → reduce_popcountwith both operands invector. Nothing enforced input domains from that contract until the planner started to, so the mistake was invisible.input_domains()is now a set (["buffer", "vector"]for those two), which keepsextract_contours().reduce_sum()rejected at build time — widening them toDomain::Anywould not have. The rejection message names the accepted set, so it now reads "expects buffer or vector input". -
Shape hints are invalidated, not carried forward, when a step has no inferable shape. Axis reductions, histograms, channel merge and the binary ops are graph-level steps
op_infer_shaperejects; keeping the pre-op H/W across them let a pipeline publish[100, 200, 2]for data that executes as[200, 3, 2]. Unknown is always safe — a typed sink asks for an explicit shape instead of planning a wrong one. -
transposeandflipvalidate their axes against the tracked rank.transpose([1,0,2])after achannel_select(rank 2) reached view-buffer'sinfer_shape, which indexes the input shape unchecked, and surfaced as aPanicExceptionwith a Rust backtrace from an ordinary builder call. The builders now raiseValueError, andop_infer_shapecatches unwinds so no future op can leak a panic into the planner. -
Each operation's input domain now comes from its Rust contract. Builders passed their own expected domain to
_validate_domain(self.DOMAIN_BUFFER, ...)at 53 call sites, restating a factop_contract(...)["input_domain"]already published and nothing read. The check moved into the append path, andPipeline'sDOMAIN_*string constants — a third copy of the domain vocabulary behind Rust'sDomain::NAMEDand the PythonDomainenum — are gone. -
op_infer_shapenow covers geometry steps.rasterize's output canvas is fixed by its own width/height, but the FFI accepted only buffer ops, so the Python builder assigned those hints itself as a side effect of building the op's parameters — which the continuation replay skipped.GeometryOp'sinfer_shapeis now read like any other, and a step whose shape is genuinely data-dependent (extract_contours) reports "not knowable" rather than rank 0. -
assert_shape()records the position it was written at. A user assertion outranks inference, but only from that point on, so it can survive a continuation replay without overriding later ops that legitimately change the shape (assert_shape(channels=3).grayscale()still reports 1 channel). -
average_precision(interpolation="all_points")omitted the first recall segment._all_points_apbuilt the monotone precision envelope correctly then integrated only over the recall values present in the curve, never anchoring at recall = 0. The leftmost block —recall[0] × envelope[0]— was dropped every time, so a three-detection curve that should score 2/3 reported 1/3, and a perfect single-TP result reported 0.0. Both_all_points_apand the matching per-replicate trapezoid inbootstrap_pr_aucnow prepend a recall-0 anchor (COCO / scikit-learnΣ (Rₙ − Rₙ₋₁) · PₙwithR₀ = 0). This changes reported AP numbers forinterpolation="all_points"(the default);11_pointis unaffected. -
PreMatchedAdapterderived the image population from detections only. Images with no detections got no metadata row, silently deleting the negative population and inflating recall / FP-per-image.match()now accepts an optionalimage_metaframe that defines the full evaluation population; omitting it keeps the previous behaviour but emits aUserWarning.image_metais the sole source ofimage_metadata, so combining it withn_gts_col/weight_col/gt_label_col/group_col— which only ever described how to derive metadata from the detection frame — raises rather than accepting the arguments and discarding them. -
FROC double-counted detections when
image_idrepeated in metadata._curve_from_detectionsjoined detections toimage_metadataweights without deduping byimage_id, so a shared rendered image owned by two cases (or a bootstrap-with-replacement draw) fan-out-multiplied every detection before TP/FP aggregation. The weight lookup is now unique by(image_id, class_id). The same join bug madeFROCResult.bootstrap_ciproduce intervals that excluded the point estimate and replica sensitivities above 1.0; both are fixed. -
A FROC metadata row is one (image, class), but the FP-per-image denominator counted rows.
froc_curvetookn_imagesfromimage_metadata.height, so a two-class table divided the false-positive rate by two. The image count is now the number of distinctimage_ids, read once through_count_imagesby bothfroc_curveandFROCResult._reconstruct— the two had drifted apart, the replicate counting draws while the point estimate counted rows, and the bootstrap distribution was therefore not on the same scale as the estimate it bracketed._curve_from_detectionsno longer takesn_imagesat all; it reads the metadata it was already given. -
Bootstrap draws are distinct evaluation units, not repeated images.
FROCResult._reconstructresampled with replacement and carried the duplicates as repeatedimage_ids, leaving every downstream count to guess whether a repeat was a redraw or one image legitimately owned by two cases. Each draw now gets a syntheticimage_id(<id>#draw<n>), so a redraw contributes its own detections, its ownn_gtsand its own slot in the FP-per-image denominator, and a repeatedimage_idagain means only one thing. -
FROC / LROC curves were returned in ascending-threshold order, so
fp_per_image/fpfran downwards through the frame andplt.step(..., where="post")drew backwards. Curves are now sorted by descendingthreshold, which is ascendingfp_per_image/fpf. Sorting on the x-column directly looks equivalent and is not: thresholds are unique so the order is total, whilefp_per_imageties constantly — every threshold bucket adding only true positives leaves it unchanged — and Polars'sortdefaults tomaintain_order=False, so the y at each tie boundary, and therefore the trapezoid leaving it, was unspecified and could differ between runs and platforms. (Commit7829291fixed the same class of nondeterminism inbootstrap_pr_aucafter it surfaced as a macOS CI failure.) This changes reported FROC/LROC AUC, deterministically. -
MetricResult.aucand.interpolateeach re-sorted the curve by x alone, inheriting that same tie ambiguity no matter how the curve was stored, and then read the first row of a tie group — sosensitivity_at_fp(0.0)reported the origin's0.0rather than the sensitivity a detector actually reaches while making no false positives. Both now go through oneMetricResult._curve_xy, which collapses tied x to the maximum y (the standard ROC upper envelope) and yields strictly increasing x.summary_table's y column is Float64 even when every operating point is out of range, instead of collapsing toNulldtype. -
The unweighted branch of
_curve_from_detectionswas unreachable.weightis inIMAGE_META_REQUIREDandDetectionTable.from_matchedvalidates it, sohas_weightswas always true and the second curve implementation behind it could never run. Deleted; an all-1.0 weight column reduces the weighted formulas to the plain counts exactly. -
FROC weighted sensitivity was order-dependent when a duplicated
image_idcarried conflicting weights. The weight lookup deduped byimage_id(first row wins) while denominators summed every metadata row, so[1, 5]vs[5, 1]flipped sensitivity (0.1667vs0.8333). Equal weights still dedupe cleanly; conflicting weights now raiseValueErrorpointing at a compositeimage_idor a single weight per unit. The lookup key is(image_id, class_id)when class is present, and the conflict check is onimage_idalone — which subsumes it. Aweightis a property of an image, and the FP-per-image denominator dedupes onimage_id, so two classes of one image disagreeing about its weight left that denominator depending on which rowuniquehappened to keep. -
MetricResult.interpolate(andsensitivity_at_fp/sensitivity_at_fpf/summary_table) clamped past the observed x-range instead of returning null. Queries beyond the curve's max now returnNone/ null so an unreachable operating point is visible rather than silently repeating the last y value.examples/06_detection_metrics.pyformats through afmt()helper accordingly —round(None, 4)is aTypeError. -
partial_aucrejected integer bounds. The boundary points it appends atlo/hiwere built with an inferred dtype, so the natural spelling of a range —froc.auc(fp_range=(0, 8)), as the docs andexamples/06_detection_metrics.pyboth write it — produced anInt64Series that would not concatenate onto theFloat64curve and raisedSchemaError: type Int64 is incompatible with expected type Float64whenever the curve did not already span the requested range. Bounds are coerced to float and the boundary Series are constructed asFloat64explicitly. -
.contour.label_reduce()was a second, divergent implementation. The accessor scored contours with a plugin-local routine whilePipeline.label_reduce()used the engine'sscore_contours_on_buffer— the same math, moved intoview-bufferearlier, whose plugin-side copy was never deleted. The two had drifted: -
region_mode="boundary"was rejected by the accessor and accepted by the pipeline. - A contour catching no pixel centre (a sub-pixel detection) scored 0.0 through the accessor and its centroid value through the pipeline. The accessor's zero then read downstream as a detection with no evidence.
- The accessor copied the whole image into a
Vec<Vec<f64>>before scoring.
Both entry points now call score_contours_on_buffer and parse reduction /
region_mode against the engine's NAMED tables, so accepted names cannot
drift either. This changes reported numbers for .contour.label_reduce()
on sub-pixel contours, which now score at their centroid instead of 0.0.
Pipeline.label_reduce() and ContourMatcher are unaffected — they were
already on the engine path.
The existing parity test covered only region_mode="bbox", the one mode the
two agreed on; it now runs across every reduction and region mode.
measures::signed_areais nowgeo's signed area rather than a hand-written shoelace loop, andtransforms::ensure_windingreadsmeasures::windinginstead of re-deriving it — leavinggeothe single authority for polygon area. (geotreats aLineStringas one-dimensional with zero area, so the ring is measured as the hole-free polygon it bounds.)
Removed¶
OutputDType, a partial second dtype table whose one distinct value meant "the default". It listedf32/f64/u8alongside the spellingsdtype_table!already owns, plus"preserve", documented in the enum and inclamp's docstring as "keep input dtype (floats preserved, integers -> f32)" — character for character whatOutputDTypeRule::PromoteToFloatdoes, which is what passing nothing already did. It was a synonym for the default, not an unimplemented feature:normalizerejected it by hand for that reason, andscale/clampaccepted it into the op's identity and dropped it.
The behaviour the word suggests (u8 in, u8 out) is preserve_dtype=True,
which is wired and tested. out_dtype now validates against DType, so every
dtype is requestable and there is one table of dtype names.
-
Ten unreferenced
view-bufferpublic items. Zero-argument wrappers over a parameterised primitive (from_slice_simd_aligned,is_simd_aligned— thefrom_slice_aligned/is_alignedpair answers strictly more, andSIMD_ALIGNMENTis now the named argument in their documented example), asize_byteswhose docstring claimed a tiling caller that did not exist, thelayout_report/LayoutReportinspection pair with no consumer,ScalarOp:: op_typesuperseded by thedescriptionbeside it, and fourNodeOutputaccessors and constructors nothing called.ImageView::get_pixelwas kept and documented instead: it is that public view type's only accessor, so deleting it would leave it unable to read what it borrows. -
Three surviving second copies.
polars_dtype_to_str(graph/compiled.rs) was a second Polars-type→dtype map besidedecode::dtype_from_polars_datatype— the two were free to disagree about which Polars types are buffer elements at all, and did.output::dtype_to_stringwas a second name forDType::numpy_namewith one caller.cloud::is_cloud_pathwas a second path classifier beside the onefetch.rsactually uses, reachable only from its own test; an unused classifier sitting next to a used one is an invitation to pick the wrong one. -
scripts/verify.sh's pinned Rust toolchain. It setRUSTUP_TOOLCHAIN=1.96— a third declaration behindrust-toolchain.toml(stable) and CI (dtolnay/rust-toolchain@stable), disagreeing with both. A localPASSwas checking a different compiler than CI, and downloading a toolchain to do it.rust-toolchain.tomlis now the only declaration. -
rasterize(anti_alias=). It was threaded from the builder through the op spec, the JSON graph,resolve_rasterize_styleandGeometryOp::Rasterizeintogeometry::rasterize, whose signature named it_anti_aliasand ignored it. Documented as "not yet implemented", but not free: it entered the op's identity, so two pipelines that behave identically hashed differently for CSE and compiled to separate graph-cache entries. Passing it now raisesTypeErrorinstead of being silently discarded. -
view-buffer's unreachable pipeline-composition layer.
ops/io.rs(SourceFormat,SinkFormat,PlaceholderMeta) and theExprNodevariants only it fed —LazySource,Placeholder,Sink— plus their constructors. Nothing in the workspace called them; the plugin builds its own source/sink vocabulary. EverymatchoverExprNodecarried arms for them, two of which werepanic!("must be resolved before building plan"). Also removes the never-enablednumpy_interop/torch_interopfeatures, and retires the "three-way format representation split" that two comments described as a divergence to live with. -
The cost-reporting subsystem.
ops/cost.rs,OpCost,OpCostReport,PipelineCostReport,ViewExpr::cost_report(),explain_costs()andOp::intrinsic_cost(). Exercised only by view-buffer's own tests — no Python surface reached it, so every op author maintained a declaration for nobody.MemoryEffectis retained and its documentation corrected: it called itself legacy and pointed atintrinsic_cost(), when it is in fact the materialisation authority (build_planinsertsMaterializeContiguousfrom it) and cost could never have replaced it, because theMemoryEffect -> OpCostconversion collapsedStridePreservingandRequiresContiguousinto a single value. -
Node-level
shape_hintsfrom the graph JSON. No Rust code ever read the key. Becausegraph_jsonis the compiled-graph cache key, two pipelines that execute identically but carry different hints occupied separate cache entries. Plan-time shape still crosses the boundary asexpected_shapeon the output spec, which Rust does read. -
Unreachable geometry vocabulary.
GeometryOpcarried ten variants (Winding,Flip,EnsureWinding,Normalize,ToAbsolute,IsConvex,ContainsPoint,IoU,Dice,HausdorffDistance) that noresolve_oparm could construct, together with an arm in the plugin's graph executor whose only job was to reject them. Those operations are served by the.contournamespace, which calls theview-buffergeometry functions directly.GeometryOpnow lists only what the pipeline graph routes, and the executor's match is exhaustive by construction rather than by a runtime error string. -
Dead
view-buffer::geometryAPI:pairwise::bbox_to_contour,predicates::point_in_polygon,measures::distance_to_segment,Point::distance_squared_to,Contour::{from_int_tuples, add_hole, iter, points}andBoundingBox::unionhad no callers.rasterize_simpleis now#[cfg(test)]: it is the per-pixel oracle the scanline filler is checked against, not a second rasterization path. -
polars_cv.geometry.validation, along with the re-exportedGeometryValidationError,OpenContourError,CoordinateRangeErrorandInvalidContourError. The three validator functions were never called and the exception classes were never raised, so nothing could catch them. TheRaises: OpenContourErrorline on.contour.area()documented an error that could not occur.
Fixed¶
-
Plan-time shape tracking was opt-in, and most builders opted out. Appending an operation required a sequence of calls each builder made by hand. All 60 ran the schema fold (
_update_output_dtype); only 19 also updated the shape hints. Ops that change shape but skipped it published a planned schema execution could not produce, which surfaced atcollect()as "planned shape [...] but execution produced [...]. The planner's shape contract disagrees with the Rust implementation." -
transpose(axes)kept the pre-transpose H/W (a 100×200 image still reported 100×200 after[1,0,2]) and kept a channel count itsunknownchannel rule says is not knowable. channel_select(index)dropped rank 3 → 2 whileexpected_shapekept publishing a three-dimensional shape.channel_mergekept the first operand's channel count.
Pipeline._push_op() is now the only code that may mutate _ops, and it
applies the domain check, the schema fold and the hint update together.
Guarded structurally by test_op_append_is_structurally_exclusive, which
fails if anything else touches _ops — replacing a ratchet that enumerated
one of the two required calls.
- The lazy continuation's shape replay never ran.
LazyPipelineExpr.pipe()re-applied each op's shape effect over the upstream hints, but assigned_expected_ndimafter that loop, so every replayed op sawndim = Noneand the H/W half of the replay returned at its opening guard. A post-loop overlay masked it for absolute targets likeresize(h, w)while producing wrong values elsewhere —.pipe(p).resize_to_height(50)reported a width of 50,resize_max(120)reported a square, andpad/pad_to_size/rotatesilently kept the upstream shape.
The continuation now folds state and hints together one op at a time through
the same append path the eager builders use, so the two spellings of an
operation — .pipe(p.op()) and .pipe(p).op() — agree by construction.
Pinned by test_eager_and_lazy_agree_on_shape_state across every chainable
op, with the op table completeness-asserted so a new operation cannot join
without a case.
0.18.0 — 2026-07-31¶
Fixed¶
- Contour IoU and Dice were wrong for most real contours. Overlap went through
a hand-rolled Sutherland–Hodgman clipper, which is only correct when the clip
polygon is convex, and whose half-plane test hard-coded a counter-clockwise
winding. Neither condition is required by
CONTOUR_SCHEMA, and neither was enforced. Concretely, against an identical copy of itself: a clockwise-wound square scored 0.0, an L-shape 0.2, a U-shape 0.0.iou(a, b)was not even symmetric — it returned 1.0 or 0.0 for the same pair depending on argument order. Holes compounded it: they were subtracted from each contour's area but ignored when intersecting, so a holed contour against itself produced an intersection larger than its union, saved only by a final clamp; with a large enough hole the union went negative and the result collapsed to 0.0.
Segmentation contours are essentially never convex, so this affected
.contour.iou(), .contour.dice(), .contour.pairwise_iou(),
.contour.match_detections(), and everything built on them —
ContourMatcher, DetectionTable, and the FROC/LROC/precision-recall curves.
Overlap is now computed exactly, by geo's boolean operations. Results are
winding-independent, hole-aware and symmetric. This changes reported numbers:
IoU and Dice will generally go up, so thresholds calibrated against the old
behaviour should be re-checked.
-
Bounding-box IoU no longer round-trips through polygon clipping; it is computed analytically from the rectangle overlap.
-
Overlapping and nested hole rings.
Contour::areasubtracted each hole's area in turn, double-counting wherever two hole rings overlapped, and reporting a different region thancontains_pointandrasterize(which have always treated the region as the exterior minus the union of the hole rings). Area now uses that same region. A square with a hole containing a further ring reports 3600 rather than 3200, andiouagainst the solid square reports 0.36 rather than 0.4348. -
hausdorff_distanceon an empty contour returned-1.797e308— a negative distance — becausegeofolds withBounded::min_value()where the previous implementation usedf64::INFINITY. It returnsINFINITYagain. -
extract_contoursdid not trace boundaries at all. The Moore-neighbour walk resumed its neighbourhood sweep five positions past the direction it had just moved, rather than at the background cell it had arrived from. From the top-left of a filled square its first step went inward along the diagonal; it then bounced between four cells and returned to the start. A filled 400×400 region came back as 399 degenerate 2×2 contours, one per row — never its outline. Everything downstream inherited it:.extract_contours(), contour metrics, andContourMatcher, wheremetricscarried a filter for detections whose "rasterized interior is empty and are provably artifacts of the boundary tracer". A region now traces to exactly one border, following its rim.
Two related faults went with it. The sweep's starting side is now chosen the
way Suzuki–Abe does — west where a foreground run begins, east where one ends —
so a hole's rim traces correctly instead of wandering the interior until a
length guard stopped it; mode="all" reports the exterior plus one border per
enclosed background region. And a trace no longer starts at a cell touching
background only diagonally (the inside of a reflex corner, or a cell
catty-corner to a hole), which was producing spurious extra contours on L, U
and plus shapes.
The traced outline runs through the centres of the boundary pixels, so it
is inset by half a pixel: a region filling w × h pixels returns bounding
(w-1) × (h-1). That is inherent to describing a pixel set by a polygon, and
is now documented and asserted rather than incidental.
-
Rasterized masks were one pixel too wide at every right-hand edge. The scanline filler behind
source("contour", ...)andPipeline.rasterize()rounded each span outward —ceil(left)tofloor(right)— instead of asking which pixel centres fall inside it. Since the scanline itself samples aty + 0.5, the vertical extent was already right, so masks came out asymmetric: a 400×400 box rasterized 401×400 pixels, and every interior hole was likewise a column too wide. A mask's pixel count therefore did not agree with the contour'sarea(), andapply_maskwith a contour mask included a column of pixels outside the contour. Masks are now exactly the set of pixels whose centre lies in the shape — the rulecontains_pointand the area measures already followed — so an axis-aligned box on integer coordinates rasterizes to precisely its area. Zero-width or zero-height output no longer panics. -
centroidmeasured a different region thanarea. It was the one measure left on the holes-as-interior-rings representation, so it subtracted each hole's moment in turn — double-counting wherever two hole rings overlap, and subtracting a nested ring that lies in an already-removed part. It now measures the same region asarea,contains_point,iouand rasterization. For a 100×100 square with holes[10,50]²and[30,70]², the centroid moves from(54.71, 54.71)— a shape that does not exist — to(53.89, 53.89). Contours with no holes, and contours whose holes are disjoint, are unaffected.
Changed¶
-
view-buffernow depends ongeo(0.33,default-features = false) for its polygon maths. Alongside the clipper, the hand-rolled Douglas–Peucker simplification, Graham-scan convex hull, ray-casting point-in-polygon, convexity test, shoelace centroid and point-to-segment projection were deleted in favour ofgeo's implementations. Public signatures inview-buffer::geometryare unchanged, but several semantics are not: -
The boolean ops, convexity test and point-in-polygon now use exact orientation predicates instead of epsilon comparisons. A point within
1e-10of an edge previously read as on the boundary and now reads as inside or outside. (Area, centroid, simplification and distance remain ordinary floating point.) hausdorff_distancewalks hole vertices as well as the exterior. A holed contour against the same solid shape used to report 0.0 and now reports a positive distance.centroidof a zero-area ring returnsgeo's length-weighted centroid rather than the mean of the vertices. Non-degenerate contours are unaffected.simplify(tolerance=0.0)is now a no-op rather than dropping collinear points, and simplification runs on the closed ring, so a ring is never collapsed below 4 points —simplifywith a huge tolerance returns the shape instead of a degenerate 2-point line.-
nearest_point_on_contourreturns null for a ring whose points are all identical, where it used to return that point. -
Hole-ness is documented as structural, and only structural. The
holesfield ofCONTOUR_SCHEMAis the sole carrier; ring winding is never interpreted as a hole signal. The schema docstrings previously also stated a "CCW = exterior, CW = hole" convention that no code read or enforced — following it (via.contour.ensure_winding("cw")) is precisely what produced an IoU of 0..contour.winding()reports point order and.contour.ensure_winding()sets it; nothing else consults it.is_closedis documented as reserved: it is written unconditionally astrueand never read back.
Added¶
-
polars_cv.build_info()reports the three versions that must agree:__version__(the imported Python source), the compiled extension's version (now exposed aspolars_cv._lib.__version__, baked in fromCargo.toml), and the installed distribution's. The install is editable, so Python edits are live while the compiled extension stays at its lastmaturin develop— after agit pullthat touches Rust, an unrebuilt environment silently runs new Python against old Rust. This makes that visible. -
tests/test_version_consistency.pyfails when the version manifests drift apart or when the installed package is stale. The release checklist inCONTRIBUTING.mdpreviously noted that nothing checked them.
0.17.0 — 2026-07-31¶
Added¶
- Null handling for per-row expression parameters. A parameter given as a
pl.Expris read from an ordinary column, which may contain nulls; until now that was unconditionally fatal.Pipeline.on_null_param("raise" | "null")chooses between failing the query (the default, unchanged) and yielding null for the affected rows — the same null-in/null-out behaviour a null input image has always had. The.contour/.point/.bboxnamespaces bypass the graph engine, so they carry the policy on the accessor instead:pl.col("c").contour.on_null("null").normalize(pl.col("w"), 100).
This is one shared mechanism rather than per-operation handling: the policy
rides on ParamCtx, and every null — numeric, enum, flag or list element, for
any of the ~70 operations — passes through ParamCol::on_null. No resolve_op
arm changed.
Two properties distinguish it from the existing on_error("null"):
- Node-scoped. A null parameter leaves that node without an output for the row, which propagates through the path a null input already takes, so only the outputs that actually depend on it go null — not every output of the row.
- Independent of error reporting. A null parameter is not treated as an
error, so it records no
_errormessage undernull_with_message, and decode, encode and genuine operation failures still raise.
There is deliberately no "fallback default" mode: pl.col("h").fill_null(224)
already expresses it in Polars, and adding a per-parameter policy would have
had to enter the ParamValue wire format and its equality/hash (which CSE
depends on).
Fixed¶
- A null operand no longer raises "references unknown node". A node that
produced nothing for a row — null input bytes,
source(on_error="null"), or now a null parameter — was indistinguishable from a node missing from the graph, so a null image in amerge_pipe/apply_mask/channel_mergeoperand column failed the query instead of nulling that row. All five cross-node operand reads now go throughCompiledGraph::operand, which separates the two cases. source("contour", shape=...)no longer fails a row whose shape reference is null. A null image in the shape branch raisedShape reference '<id>' not found. Ensure the shape source is defined before this contour pipeline.— a message pointing at a graph-wiring problem that did not exist.source("contour", shape=expr)now works whenexpris referenced only as the shape, with no nulls involved. The source recorded the reference by node id but never registered it as a dependency, so the node was left out of the graph entirely and execution failed on a dangling reference. It happened to work whenever the shape node was also consumed some other way — masking with it, which is what every example does — which is why it went unnoticed.- A null in a non-primitive parameter column reported a cast failure from
try_extractrather than the null-value error, bypassing the null path.
Documentation¶
- The API reference gained a
BBoxNamespacesection —.bboxwas the one namespace with no reference page — plus theon_nullpolicy the three geometry accessors share, which is inherited from a mixin and so was not rendered from any of their own class bodies. LazyPipelineExpr's reference page now says that thePipelinemethods mirrored onto it are generated at import time, which is why they are absent from the members list below it.- The composition rule for
on_null_paramis stated correctly: unlikeon_error, an explicit"raise"composed with a"null"gives the graph"null"rather than being rejected as a conflict, because only non-default policies are collected and there are only two values. shape=on the contour source is documented as registering an upstream dependency, so referencing a pipeline only as a shape is a supported shape of graph, and a null in that branch nulls the mask.
0.16.0 — 2026-07-30¶
Added¶
- Per-row expression parameters in the geometry namespaces.
.contour,.pointand.bboxpreviously accepted only literals — and four.contourmethods annotatedint | pl.Exprwhile unconditionally raisingTypeErroron exactly that type, an impossible signature that the API reference published. Their numeric parameters now resolve per row:
# Normalize each contour against its own image's dimensions
df.with_columns(
norm=pl.col("contour").contour.normalize(pl.col("img_w"), pl.col("img_h"))
)
Covers normalize, to_absolute, translate, scale, simplify,
area(signed=) and match_detections(threshold=) on .contour; normalize,
to_absolute, translate, scale, rotate(angle=) and interpolate(t=) on
.point; and match_detections(threshold=) on .bbox. Aggregations
broadcast and null parameters are errors, matching the image operations.
-
Per-row elements in list-valued parameters. The list length stays structural — it fixes a kernel size or channel count at planning time — while each element may now be an expression, the encoding
warp_affine's matrix already used. This coversconvolve2d(kernel=),normalize(mean=, std=)andchannel_swap(order=), and in turn makessharpen(strength=)per-row, whose docstring previously described the limitation as permanent. -
Per-row non-structural enums and flags.
filter(every resize variant andletterbox),interpolation(rotate,warp_affine),pad(mode=),pad_to_size(position=),convolve2d(border=),extract_contours(mode=, method=),label_reduce(reduction=, region_mode=)— on bothPipelineand the.contournamespace — plus the flagsapply_mask(invert=),convolve2d(normalize=)andarea(signed=). A parameter is eligible only when it has no effect on output shape, rank or dtype; that invariant is what lets plan-time shape probing substitute the default. (rasterize(anti_alias=)is plumbed identically, but view-buffer's rasterizer still ignores the flag, so it has no observable effect yet.) -
rotate_and_scale(angle=, scale=, center=, output_size=)accept expressions, building the affine matrix from expression arithmetic. letterbox(filter=)is now exposed; Rust already read it. The historicallanczos3default is unchanged.- Contour-source
fill_value/backgroundaccept expressions, matching the identical parameters on therasterizeop. FilterTypeexposes all five view-buffer filters.catmullromandgaussianwere deliberately withheld from Python while Rust accepted them. Makingfilterper-row broke that restriction asymmetrically — a literal was checked against the smaller Python enum, a column value went straight to Rust's larger table — so rather than validate the same subset twice, the subset was dropped andFilterTypeis now full parity.
Changed¶
- A Boolean column no longer satisfies a numeric per-row parameter. It used
to coerce to
0/1, which reads as a mis-routed expression far more often than as a value someone chose; passing one now raises the same cast error aStringcolumn does.
Fixed¶
- Plan/execution rank desync for ops with list-valued parameters on
listandarraysources.fold_output_rankre-serialized ops that graph compilation had already bound, and the compiled-onlySlot/Listforms are#[serde(skip)]— so the op silently failed to resolve, the output rank was recorded as unknown, and a plannedList(List(List(f32)))collapsed toList(f32). Affectedwarp_affineon list/array sources. - Single-channel affine warps dropped the channel axis.
warp_affine(and the arbitrary-anglerotate/shear/rotate_and_scalepaths that share its kernel) collapsed a[H, W, 1]input to[H, W], contradicting the op's owninfer_shape, which preserves the input rank. A[H, W]input still produces[H, W]; only the explicit single-channel axis is now retained. This was masked by the rank-folding bug above. convolve2dskipped kernel/ksizevalidation entirely whenksizewas an expression, letting a mismatched kernel reach Rust unchecked. The kernel length is structural, so it is now always validated as an odd perfect square.- Structural parameters given an expression now report why.
cast(dtype=),normalize(method=),histogram(closed=, output=),perceptual_hash(algorithm=)and thetranspose/flipaxis lists failed opaquely insidebool()("the truth value of an Expr is ambiguous"), on.value, or at JSON encoding; they now raise the same clear "is structural"TypeErroras the other literal-only parameters. docs/user-guide/operations/geometry.mddocumentedpoint.normalizeandpoint.to_absolutewithref_width=/ref_height=— the Rust kwarg names. Both examples raisedTypeError; the Python parameters arewidth/height.- The
Pipelinedocstring claimed all operations accept expressions, and the image-ops guide's "all resize variants" claim sat directly abovethumbnail, whosemax_sizeis literal-only. Both now state the actual rule. - The quickstart repeated the same "any parameter" claim; it now states the
eligibility rule and links the full table. The pipelines concept page states
the rule once where chaining is introduced, and points at the geometry
namespaces, which now follow it too. The geometry guide called
.contourand.pointthe geometry namespaces while documenting.bboxalongside them.
0.15.0 — 2026-07-26¶
Added¶
.cv.read_bytes()— read a path column's bytes without decoding. Thefile_pathsource has always been two stages (fetch the bytes a path names, then decode them as an image); the fetch stage is now available on its own and returnsBinary. Bytes come back verbatim, so an encoded file survives the round trip byte-for-byte and can be written back unchanged — the only lossless path through the plugin, since re-encoding a decoded JPEG never reproduces the original and no image sink carries EXIF/ICC metadata. It also lets the header-only metadata methods (.cv.width()and friends, which take binary columns) reach local and remote files for the first time:
df = df.with_columns(raw=pl.col("path").cv.read_bytes(cloud_options=options))
df = df.filter(pl.col("raw").cv.width() > 512) # header-only, no decode
df = df.with_columns(thumb=pl.col("raw").cv.pipe(pipe).sink("png"))
Takes the same cloud_options as source("file_path") and the same
on_error values ("raise" / "null"). Fetching is per plugin call — one
morsel under the streaming engine — so distinct remote paths are deduped and
fetched concurrently exactly as the source already does, and a bytes column
stays morsel-bounded under engine="streaming".
Changed¶
- The
file_pathsource andread_bytesshare one fetch implementation (src/fetch.rs). The path-to-bytes stage — remote dedup and concurrent prefetch, the local-read path that avoids misparsing colon-bearing filenames as cloud URLs, and theon_errorvocabulary — now lives in one module used by both, so their credentials, concurrency, and error messages cannot drift. Behaviour of existing pipelines is unchanged.
0.14.0 — 2026-07-25¶
Added¶
source("auto")— decode path inferred from the column dtype. A source no longer has to be told how to read its column when the Polars dtype already says:String→file_path,List→list,Array→array, andBinary→blobwhen the bytes carry the VIEW protocol magic,image_bytesotherwise. The resolution happens once per batch (the column dtype is constant across rows), so the now-default path costs no per-row work. Everything that applied to the concrete formats still applies:cloud_optionsand concurrent remote prefetch (an auto source over a URL column still prefetches),decode_max_size, andrequire_contiguous. A dtype that cannot be routed (e.g. a plain numeric column) raises and names the explicit formats to choose from.- 16-bit PNG output. A
u16buffer now encodes to a 16-bit PNG (Luma16/LumaA16/Rgb16/Rgba16) instead of failing with "Image export requires U8 dtype", so a 16-bit PNG or TIFF read through a pipeline and back out tosink("png")keeps its bit depth end to end.
Changed¶
Pipeline.source()defaults to"auto"(was"image_bytes"). ABinaryimage column behaves exactly as before; aStringcolumn, which previously had to be spelledsource("file_path"), now reads as a path; and a VIEW-taggedBinarycolumn resolves toblobrather than failing an image decode. Passing an explicit format still overrides the inference in every case.
Fixed¶
- Actionable errors for bit-depth mismatches at the image sinks. JPEG and
WebP are 8-bit formats: sinking a non-
u8buffer to them now fails before encoding with an error naming the dtype and pointing at.cast("u8")or the PNG/TIFF sinks, instead of a generic encode failure. Float buffers into any image sink likewise suggest casting to an integer dtype or sinking to TIFF, which supports floating point.
0.13.0 — 2026-07-24¶
Added¶
- Federated GCS authentication (Workload/Workforce Identity Federation) —
Application Default Credentials of type
external_accountandexternal_account_authorized_user(e.g. an OIDC identity exchanged into Google through an identity pool) now work without minting a bearer token by hand. When polars-cv detects a federated ADC (fromGOOGLE_APPLICATION_CREDENTIALS, an explicitgoogle_application_credentialsoption, or the well-known gcloud path), it delegates togcloud auth application-default print-access-token— which understands the entire federation matrix — and uses the resulting token, cached until just before it expires. Requires thegcloudCLI onPATH; setPOLARS_CV_DISABLE_GCS_FEDERATION=1to disable. CloudOptions.token_command— a provider-agnostic hook: any shell command whose stdout is an OAuth access token. Use it to source tokens from a custom broker, wrapper script, or CLI. It applies to both OAuth-bearer backends — GCS and Azure (e.g.az account get-access-token) — takes precedence over the automaticgclouddelegation, and its output is cached like any other token. It does not apply to S3 (SigV4, not bearer-based); passing it with ans3://source raises rather than silently ignoring the credential.
Fixed¶
- GCS
gcs_bearer_tokenno longer defeated by unparseable ambient credentials — bumpedobject_storeto 0.13. In 0.12 the GCS builder read and parsed Application Default Credentials (GOOGLE_APPLICATION_CREDENTIALSor the well-known~/.config/gcloud/application_default_credentials.json) insidebuild()unconditionally, before honoring an explicitly supplied credential provider. So when the ambient ADC was a federatedexternal_account_authorized_usercredential — exactly the casegcs_bearer_tokenexists to work around —build()failed on the ADC parse and the bearer token was never consulted. object_store 0.13 skips the ADC read whenever an explicit credential (or service account) is configured, restoring the intended escape-hatch behavior. The bump also dedupesobject_storeto a single version, since polars 0.54 already pulls 0.13.
0.12.0 — 2026-07-23¶
Added¶
- f16 tensor output —
.sink("numpy"|"torch", dtype="f16")downcasts the output to half precision at the encode boundary, halving the output-tensor bytes and host→device transfer. (The engine has no native f16 dtype; only the sink container is f16.) - Per-row affine/shear parameters —
warp_affine(matrix=...)accepts a per-element mix of literals and Polars expressions, andshear(sx=..., sy=...)accepts expressions, so a batch can apply a different (e.g. random) affine or shear per row in one call. Pipeline.thumbnail(max_size)— explicit, chainable form of the existing JPEG IDCT-scaled decode (source(..., decode_max_size=...)), for cheap decode-aware curation passes.- Single-threaded execution warning — a one-time notice when a large batch
runs single-threaded under the in-memory engine, pointing to
engine="streaming". Silence withPOLARS_CV_SILENCE_ENGINE_WARNING=1; tune withPOLARS_CV_ENGINE_WARN_ROWS. CloudOptions.storage_optionspass-through — arbitrary cloud options are now forwarded verbatim to the underlyingobject_storebackend using its native config keys (e.g.google_service_account_key,google_application_credentials,aws_endpoint), so any option the backend understands is available without new plumbing. Keys instorage_optionswin over the namedCloudOptionsfields on collision. This replaces the previous fixed allow-list, which silently dropped unrecognized keys.CloudOptions.gcs_bearer_token— supply a pre-obtained GCS OAuth access token directly. This unblocks federated/brokered Google credentials (ADC of typeexternal_account_authorized_user) thatobject_storecannot parse natively: mint a token out of band (e.g.gcloud auth application-default print-access-token) and pass it in.
Changed¶
warp_affinematrixwire format — the serializedmatrixparameter is now an array of per-elementParamValueentries (each may be a literal or an expression) rather than a raw float array. This is an internal graph-JSON change; pipelines built through the Python API are unaffected. A persisted or hand-written graph using the old raw-floatmatrixarray must be regenerated.
Fixed¶
cloud_optionson non-file_pathsources — passingcloud_optionsto a source format that ignores it (e.g.image_bytes) now emits aUserWarninginstead of silently dropping the credentials.normalize(out_dtype=...)planned vs. produced dtype — the planner honoredout_dtypebut execution always produced f32, so anyout_dtypeother than f32 tripped the dtype-contract guard. Normalization now casts its f32 result to the requested dtype, sonormalize(out_dtype="u8"|"f64")works and the planned dtype always matches the produced dtype. (out_dtype="preserve"is rejected with a clear message — it is not meaningful for normalization.)
0.11.0 — 2026-07-10¶
Added¶
file://URLs in the local read path —source("file_path")now decodesfile://URIs, not just bare local paths, routing local reads through the samecloud::read_fileentry point as remote schemes.
Changed¶
- Cloud access is signed by default, anonymous is opt-in — GCS now honors
CloudOptions(anonymous=True)(viawith_skip_signature) like S3 and Azure. The previously documented "anonymous-first" credential chain — which no provider actually implemented — is gone; requests are signed unless you explicitly opt into anonymous access.
Fixed¶
- Blob header validation — untrusted
blobVIEW-protocol headers are now validated with checked arithmetic and stride-span checks, rejecting malformed inputs instead of risking overflow. - Arrow C-Data import — the array offset is respected and null-bearing arrays are rejected on import, fixing silently wrong reads of sliced/offset Arrow buffers.
- Buffer deallocation — owned buffers are freed with their original alignment, fixing a mismatched-layout deallocation.
- Materializing kernels now declare contiguous output, and
gamma's integer range is corrected. - Metrics — the VOC 11-point AP denominator and the bootstrap PR-AUC estimator mismatch are fixed.
- Panics eliminated — axis reductions over 1-D buffers, and the
u8RGBA→Lab alpha color path (which also now preserves dtype) no longer panic. - Rotation fusion uses position-correct shapes, and batch re-folds seed from the post-source state.
- Colon-bearing local paths are read literally, and a dead
Caststride arm was removed. - Builds on current stable Rust — the transitive
ethnumdependency is pinned to 1.5.3, which drops theTryFromIntErrortransmute that failed to compile (E0512) on newerrustc. The stack builds cleanly against polars 0.54.4 / pyo3-polars 0.27 / pyo3 0.28.
0.10.0 — 2026-06-13¶
Added¶
preserve_dtypeon intensity ops —scale,clamp, andadjust_brightnessacceptpreserve_dtype=Trueto cast thef32result back to the input dtype (e.g.u8in →u8out) instead of promoting tof32. Requires a known input dtype and is mutually exclusive without_dtype.- Graph-level error policy —
Pipeline.on_error("raise" | "null" | "null_with_message")(mirrored onLazyPipelineExpr). Failing rows null all outputs, optionally attaching a reserved_errormessage field, instead of failing the whole batch. - Full
LazyPipelineExprmethod parity — every chainablePipelineoperation can now be called directly on the lazy expression returned by.cv.pipe(...), without wrapping each step in its ownPipeline. Guarded bytest_lazy_pipeline_method_parity. - Shape references for rasterization —
rasterize(shape=<expr>)andsource("contour", shape=<expr>)size the raster canvas from a referenced pipeline's output[H, W], implemented end-to-end. - Rotation parameters —
rotate()acceptsinterpolationandborder_value.
Changed¶
- Single schema authority in view-buffer — the Python planner no longer
keeps a parallel contract table. Per-op output domain, dtype, rank, and
channel count are read from view-buffer's declarative rules
(
OutputChannelRule/OutputRankRule). TheOPERATION_CONTRACTS,AlphaMode, andNdimEffectstructures were removed. - Plan == exec — planned dtype/shape now matches execution for promoting
binary ops (e.g. true division →
f32) and for explicitly-typed 16-bit image sources. - Rotation unified through the affine path — arbitrary-angle
rotate()routes throughComputeOp::RotateAffine→apply_affine_warp(); zero-copy 90/180/270 fast paths are preserved. - Portable CI / wheel builds — abi3 wheels on Linux and a fixed macOS build.
Fixed¶
cloud_optionsnow round-trips for remotefile_pathsources.- Assorted
ruff/clippylint findings; CI excludes flaky network-marked tests.
Performance¶
- Kernel fusion extended: casts fold into the
FusedKernel(u8 → cast(f32) → scale → clamp → reluruns as one pass);rotate→warp_affinechains fuse. - Vectorized
erode/dilate,convolve2d(interior/border split), and separable Gaussian blur; Canny rewritten withoutatan2(bit-exact). - Remote
file_pathsources prefetch concurrently per batch. - Graphs compile once into a process-wide cache (
graph/compiled.rs), so repeat invocations (e.g. per streaming morsel) pay only a hash lookup.
Releases earlier than 0.10.0 predate this changelog; see the git history for details.