TL;DR
Get monitors, keyboards and dev gear delivered free — and shop member deals
- Fast, free delivery on millions of items
- Access to Prime Big Deal Days deals on October 6–7
- Prime Video, Amazon Music and more included
Polars says version 2.0 makes its streaming engine the default for lazy queries and enables initial out-of-core processing that can spill supported workloads to disk. The release also expands SQL support and adds a Map data type; benchmark claims favor Polars, but come from tests run and reported by the Polars team.
Polars has released version 2.0, making its streaming engine the default for lazy queries and enabling initial out-of-core processing that can spill some workloads to disk. The changes are intended to improve memory use and query performance, but they also alter the default row-order guarantees for certain operations, so users who rely on ordering may need to update their queries.
Under the new default, calling collect on a LazyFrame uses the streaming engine. Polars says that engine can reduce memory use and improve performance on many queries. However, operations including joins, group-bys and unpivots do not guarantee observable row order by default. Users who need to preserve order can set maintain_order=True for supported operations.
Version 2.0 also enables spill-to-disk processing by default. The release report says spilling begins at about 80% of available RAM, with a default disk budget of 64GB. Current support includes sorts, window functions and many expressions. Joins and group-bys are not yet listed as supported out-of-core operations; Polars says it plans to add them later.
The release introduces a native Map data type for Arrow MapType data, which Polars previously represented as a list of key-value structs. It also promotes SQL as a first-class interface and cites optimizer and engine work, including join reordering, common-subplan elimination and dynamic predicates or bloom filters.
Changes to Query Defaults and Memory Use
The default switch to streaming is the release’s most immediate practical change for users with existing lazy-query workflows: they may see different memory and performance behavior without changing how they call collect. The trade-off is that row order is no longer guaranteed for some operations unless users explicitly request it. That makes testing results and checking assumptions about ordering important when upgrading.
Disk spilling may also let supported queries finish when their working sets exceed available memory, reducing the chance that a workload fails solely because it cannot fit in RAM. Its reach remains limited for now: the release does not claim out-of-core support for joins or group-bys, and the 64GB disk budget is a default that users may need to review for their systems. The SQL and optimizer updates matter to teams using Polars for relational workloads, but performance will depend on query shape and hardware.
high performance SSD for data spillover
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
How Polars Reports Its Benchmarks
Polars’ release report presents SQL tests against data derived from TPC-H and TPC-DS, comparing Polars with DuckDB 1.5.6, a DuckDB 2.0 development build and DataFusion 54.0.0. The tests used two AWS machine configurations, one with 16 vCPUs and 32GB of RAM and another with 192 vCPUs and 384GB. Each query ran five times in a hot setting, and the report compared the best run for each query using total and geometric-mean query times.
These are benchmarks reported by Polars, not an independent assessment. The company says Polars and both DuckDB versions completed all queries; DataFusion timed out on some TPC-DS queries and ran out of memory on one TPC-H query on the smaller machine. Those queries were excluded from the comparison for all engines. Polars also says its default configuration was fastest in all but one benchmark, while acknowledging overhead at 192 threads that hurt smaller queries. It says a 32-core Polars configuration was competitive or leading in all benchmarks and that it hopes to address the scaling issue in a later release.
The major-version change is tied to the new streaming default and its ordering behavior, rather than being described by the developers as a broad feature release. The release report says the team had explained the rationale for the version bump in an earlier announcement, but that post’s details are not included in the supplied material.
“Calling collect on a LazyFrame will now default to the streaming engine.”
— Polars, in its release report
large capacity external hard drive 64GB
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Limits Still Affect Upgrade Decisions
The supplied release material does not give an independent replication of the benchmark results, and its reported lead should not be treated as a guarantee for other hardware, datasets or query mixes. The team invites readers to reproduce the tests and provides a benchmark repository, but the source material does not include external results.
It is also unclear from the supplied report how much users should expect performance or memory use to change across their own workloads. Spill thresholds may need tuning, and some operations—especially joins and group-bys—do not yet have the stated out-of-core support. The report does not specify a release date, migration guide details, or the exact compatibility changes beyond the row-order behavior described.
As an affiliate, we earn on qualifying purchases.
More Spill Support Is Planned
Polars says it plans to extend out-of-core processing to joins and group-bys, which would broaden the range of workloads able to spill to disk. It also says it hopes to address the high-thread-count overhead identified in its benchmark testing in a subsequent release; no delivery date is provided.
For teams upgrading now, the immediate next step is to test lazy queries that depend on row order and workloads that approach memory limits. Readers seeking to evaluate the performance claims can consult the benchmark repository named in the release report, while treating the results as specific to the published test setup until independent comparisons are available.
As an affiliate, we earn on qualifying purchases.
Key Questions
What is the main change in Polars 2.0?
LazyFrame.collect now uses the streaming engine by default. Polars also enables initial out-of-core spilling for supported operations and adds SQL and data-type changes.
Will Polars 2.0 preserve row order?
Not by default for some operations, including joins, group-bys and unpivots, according to Polars. Users who need observable order can set maintain_order=True where supported.
Which operations can spill to disk?
The release report lists sorts, window functions and many expressions. It says joins and group-bys are planned for future out-of-core support, rather than supported in this initial release.
Did Polars independently verify its benchmark lead?
The benchmark results in the supplied material were run and reported by Polars. The team shared a repository for replication, but the source does not provide independent confirmation.
Source: hn
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
