polars_cloud.LazyFrameRemote.sink_batches

LazyFrameRemote.sink_batches(function: Callable[[DataFrame], bool | None], *, chunk_size: int | None = None, maintain_order: bool = False, optimizations: QueryOptFlags = <polars.lazyframe.opt_flags.QueryOptFlags object>) → DirectQuery | ProxyQuery

Evaluate the query and call a user-defined function for every ready batch.

This allows streaming results that are larger than RAM in certain cases.

Warning

This method is much slower than native sinks. Only use it if you cannot implement your logic otherwise.

Parameters:

function

Function to run with a batch that is ready. If the function returns True, this signals that no more results are needed, allowing for early stopping. Note: this function should be idempotent, as it might be called multiple times from different workers.

chunk_size

The number of rows that are buffered before the callback is called.

maintain_order

Maintain the order in which data is processed. When True, function will be called serially for all batches, instead of in parallel across the workers in the cluster.

optimizations

The optimization passes done during query optimization.

Examples

>>> lf = pl.scan_csv("/path/to/my_larger_than_ram_file.csv")  
>>> lf.sink_batches(lambda df: print(df))