Autoscaling

Polars can scale its worker pool while the cluster is running, instead of holding a fixed number of workers. A query states how many workers it needs, and the cluster grows to meet that demand, instead of the query being limited to the workers that happen to be running when it is submitted.

Cluster configuration

The cluster must be configured to scale before queries can ask it to. See the dedicated cluster type pages for more information.

• Bare metal
• Kubernetes Helm Chart

Sizing a query

The number of workers a query uses is set through min_workers and max_workers.

lf.remote(context=ctx).distributed(min_workers=2, max_workers=8).execute()

min_workers is the number of workers that must be available before execution starts. The cluster scales up to meet it, and the query waits until those workers have joined. If they do not arrive in time, the query fails once it has waited longer than the configured scale-up timeout.

max_workers is the most workers the query will use. It also tells the query planner how many data partitions to create, so setting it is useful on an autoscaled cluster even if you do not need to limit capacity, because without it the planner only has the current worker count to work with.

A query is also capped by two limits the cluster sets: its per-query maximum (max_workers_per_query) and, on an autoscaled cluster, the maximum it is allowed to scale to. A query asking for more than the lower of the two is capped to it and runs at that size, rather than being rejected.

These settings are not specific to autoscaling. On a fixed-size cluster, min_workers waits for workers that are already part of the cluster instead of causing new ones to start, and max_workers caps the query and sets its partition count just the same.

Both settings are optional, and can be given on their own. If you do not set max_workers, the ceiling comes from the cluster: its default_workers_per_query when one is configured, and otherwise every worker the cluster has, including any that join while the query runs.

Setting	Behavior
Neither	Starts with the workers that are available, and grows to the cluster's ceiling.
min_workers only	Waits for that many workers, then grows to the cluster's ceiling.
max_workers only	Starts as soon as one worker is available, and grows to at most that many.
Both	Waits for min_workers, then uses up to max_workers.

Setting max_workers = 1 runs the query on a single node, without distributing it.