pola-rs/polars: Python Polars 1.36.0

🏆 Highlights

• Add Extension types (#25322)

🚀 Performance improvements

• New single file IO sink pipeline enabled for sink_parquet (#25670)
• New partitioned IO sink pipeline enabled for sink_parquet (#25629)
• Correct overly eager local predicate insertion for unpivot (#25644)
• Reduce HuggingFace API calls (#25521)
• Use strong hash instead of traversal for CSPE equality (#25537)
• Fix panic in is_between support in streaming Parquet predicate push down (#25476)
• Faster kernels for rle_lengths (#25448)
• Allow detecting plan sortedness in more cases (#25408)
• Enable predicate expressions on unsigned integers (#25416)
• Mark output of more non-order-maintaining ops as unordered (#25419)
• Fast find start window in group_by_dynamic with large offset (#25376)
• Add streaming native LazyFrame.group_by_dynamic (#25342)
• Add streaming sorted Group-By (#25013)
• Add parquet prefiltering for string regexes (#25381)
• Use fast path for agg_min/agg_max when nulls present (#25374)
• Fuse positive slice into streaming LazyFrame.rolling (#25338)
• Mark Expr.reshape((-1,)) as row separable (#25326)
• Use bitmap instead of Vec<bool> in first/last w. skip_nulls (#25318)
• Return references from aexpr_to_leaf_names_iter (#25319)

✨ Enhancements

• Add bin.slice(), bin.head(), and bin.tail() methods (#25647)
• Add SQL support for the QUALIFY clause (#25652)
• New partitioned IO sink pipeline enabled for sink_parquet (#25629)
• Add SQL syntax support for CROSS JOIN UNNEST(col) (#25623)
• Add show methods for DataFrame and LazyFrame (#19634)
• Allow pl.Object in pivot value (#25533)
• Extend SQL UNNEST support to handle multiple array expressions (#25418)
• Minor improvement for as_struct repr (#25529)
• Temporal quantile in rolling context (#25479)
• Add support for Float16 dtype (#25185)
• Add strict parameter to pl.concat(how='horizontal') (#25452)
• Add leftmost option to str.replace_many / str.find_many / str.extract_many (#25398)
• Add quantile for missing temporals (#25464)
• Expose and document pl.Categories (#25443)
• Support decimals in search_sorted (#25450)
• Add SQL support for named WINDOW references (#25400)
• Add Extension types (#25322)
• Add having to group_by context (#23550)
• Allow elementwise Expr.over in aggregation context (#25402)
• Add SQL support for ROW_NUMBER, RANK, and DENSE_RANK functions (#25409)
• Automatically Parquet dictionary encode floats (#25387)
• Add empty_as_null and keep_nulls to {Lazy,Data}Frame.explode (#25369)
• Allow hash for all List dtypes (#25372)
• Support unique_counts for all datatypes (#25379)
• Add maintain_order to Expr.mode (#25377)
• Display function of streaming physical plan map node (#25368)
• Allow slice on scalar in aggregation context (#25358)
• Allow implode and aggregation in aggregation context (#25357)
• Add empty_as_null and keep_nulls flags to Expr.explode (#25289)
• Add ignore_nulls to first / last (#25105)
• Allow Expr.unique on List/Array with non-numeric types (#25285)
• Allow Expr.rolling in aggregation contexts (#25258)
• Allow bare .row() on a single-row DataFrame, equivalent to .item() on a single-element DataFrame (#25229)
• Raise suitable error on non-integer "n" value for clear (#25266)
• Support additional forms of SQL CREATE TABLE statements (#25191)
• Add LazyFrame.pivot (#25016)

🐞 Bug fixes

• Rechunk on nested dtypes in take_unchecked_impl parallel path (#25662)
• New single file IO sink pipeline enabled for sink_parquet (#25670)
• Fix streaming SchemaMismatch panic on list.drop_nulls (#25661)
• Correct overly eager local predicate insertion for unpivot (#25644)
• Address potential "batch_size" parameter collision in scan_pyarrow_dataset (#25654)
• Fix "dtype is unknown" panic in cross joins with literals (#25658)
• Fix "unreachable code" panic in UDF dtype inference (#25655)
• Fix panic on Boolean rolling_sum calculation for list or array eval (#25660)
• Fix panic edge-case when scanning hive partitioned data (#25656)
• Fix lifetime for AmortSeries lazy group iterator (#25620)
• Improve SQL GROUP BY and ORDER BY expression resolution, handling aliasing edge-cases (#25637)
• Fix empty format handling (#25638)
• Prevent false positives in is_in for large integers (#25608)
• Differentiate between empty list an no list for unpivot (#25597)
• Properly resolve HAVING clause during SQL GROUP BY operations (#25615)
• Fix spearman panicking on nulls (#25619)
• Make sum on strings error in group_by context (#25456)
• Hang in multi-chunk DataFrame .rows() (#25582)
• Bug in boolean unique_counts (#25587)
• Set Float16 parquet schema type to Float16 (#25578)
• Correct arr_to_any_value for object arrays (#25581)
• Have PySeries::new_f16 receive pf16s instead of f32s (#25579)
• Fix occurence of exact matches of .join_asof(strategy="nearest", allow_exact_matches=False, ...) (#25506)
• Fix incorrect .list.eval after slicing operations (#25540)
• Reduce HuggingFace API calls (#25521)
• Strict conversion AnyValue to Struct (#25536)
• Fix panic in is_between support in streaming Parquet predicate push down (#25476)
• Always respect return_dtype in map_elements and map_rows (#25504)
• Rolling mean/median for temporals (#25512)
• Add .rolling_rank() support for temporal types and pl.Boolean (#25509)
• Fix dictionary replacement error in write_ipc() (#25497)
• Fix group lengths check in sort_by with AggregatedScalar (#25503)
• Fix expr slice pushdown causing shape error on literals (#25485)
• Allow empty list in sort_by in list.eval context (#25481)
• Prevent panic when joining sorted LazyFrame with itself (#25453)
• Validate list.slice parameters are not lists (#25458)
• Apply CSV dict overrides by name only (#25436)
• Incorrect result in aggregated first/last with ignore_nulls (#25414)
• Fix arr.{eval,agg} in aggregation context (#25390)
• Support AggregatedList in list.{eval,agg} context (#25385)
• Improve SQL UNNEST behaviour (#22546)
• Remove ClosableFile (#25330)
• Resolve edge-case with SQL aggregates that have the same name as one of the GROUP BY keys (#25362)
• Revert pl.format behavior with nulls (#25370)
• Remove Expr casts in pl.lit invocations (#25373)
• Nested dtypes in streaming first_non_null/last_non_null (#25375)
• Correct eq_missing for struct with nulls (#25363)
• Unique on literal in aggregation context (#25359)
• Allow implode and aggregation in aggregation context (#25357)
• Aggregation with drop_nulls on literal (#25356)
• Address multiple issues with SQL OVER clause behaviour for window functions (#25249)
• Bugs in pl.from_repr with signed exponential floats and line wrapping (#25331)
• Schema mismatch with list.agg, unique and scalar (#25348)
• Correct drop_items for scalar input (#25351)
• SQL NATURAL joins should coalesce the key columns (#25353)
• Mark {forward,backward}_fill as length_preserving (#25352)
• Nested dtypes in streaming first/last (#25298)
• Fix link errors reported by markdown-link-check (#25314)
• Parquet is_in for mixed validity pages (#25313)
• Fix length preserving check for eval expressions in streaming engine (#25294)
• Don't quietly allow unsupported SQL SELECT clauses (#25282)
• Reverse on chunked struct (#25281)
• Panic exception when calling Expr.rolling in .over (#25283)
• Run async DB queries with regular asyncio if not inside a running loop (#25268)
• Correct {first,last}_non_null if there are empty chunks (#25279)
• Incorrect results for aggregated {n_,}unique on bools (#25275)
• Ensure out-of-range integers and other edge case values don't give wrong results for index_of() (#24369)
• Correctly prune projected columns in hints (#25250)
• Allow Null dtype values in scatter (#25245)
• Correct handle requested stops in streaming shift (#25239)
• Make str.json_decode output deterministic with lists (#25240)

📖 Documentation

• Document schema parameter in meta methods (#25543)
• Correct link to datetime_range instead of date_range in resampling page (#25532)
• Explain aggregation & sorting of lists (#25260)
• Update LazyFrame.collect_schema() docstring (#25508)
• Remove lzo from parquet write options (#25522)
• Deprecate Categorical functions for lexical ordering and local checks (#25514)
• Update on-premise documentation (#25489)
• Add LazyFrame.pivot to reference guide (#25482)
• Add docstring example showing str.slice taking Expression params (#25461)
• Fix incorrect 'bitwise' in any_horizontal/all_horizontal docstring (#25469)
• Add Extension and BaseExtension to doc index (#25444)
• Add polars-on-premise documentation (#25431)
• Add having API references (#25428)
• Fix link errors reported by markdown-link-check (#25314)
• Update user guide for QueryProgress rename to QueryProfile (#25195)

🛠️ Other improvements

• Avoid rechunk requirement for Series.iter() (#25603)
• Use dtype for group_aware evaluation on ApplyExpr (#25639)
• Add "panic" and "streaming" tagging to issue-labeler workflow (#25657)
• Fix rolling kernel dispatch with monotonic group attribute (#25494)
• Ensure we hash all attributes and visit all children in traverse_and_hash_aexpr (#25627)
• Ensure literal-only SELECT broadcast conforms to SQL semantics (#25633)
• Rename polars-on-premise to polars-on-premises (#25617)
• Constrain new issue-labeler workflow to the Issue title (#25614)
• Help categorise Issues by automatically applying labels (using the same patterns used for labelling PRs) (#25599)
• Add disk-cleaning step for Ubuntu runners (#25593)
• Show on streaming engine (#25589)
• Ignore a couple of unexplained typing errors (#25580)
• Skip existing files in pypi upload (#25576)
• Fix template path in release-python workflow (#25565)
• Skip rust integration tests for coverage in CI (#25558)
• Add Final type-qualifier to module-level constants (#25556)
• Add asserts and tests for list.eval on multiple chunks with slicing (#25559)
• Add assert_sql_matches coverage for SQL DISTINCT and DISTINCT ON syntax (#25440)
• Use strong hash instead of traversal for CSPE equality (#25537)
• Update partitioned sink IR (#25524)
• Print expected DSL schema hashes if mismatched (#25526)
• Fix --uv argument for benchmark-remote (#25513)
• Add proptest DataFrame strategy (#25446)
• Run maturin with --uv option (#25490)
• Fix feature gating TZ_AWARE_RE again (#25493)
• Add some cleanup (#25445)
• Test for group_by(...).having(...) (#25430)
• Remove aggregation context Context (#25424)
• Refactor sink IR (#25308)
• Remove ClosableFile (#25330)
• Remove debug file write from test suite (#25393)
• Add ElementExpr for _eval expressions (#25199)
• Dispatch Series.set to zip_with_same_dtype (#25327)
• Better coverage for group_by aggregations (#25290)
• Enable more streaming tests (#25364)
• Remove Column::Partitioned (#25324)
• Add toolchain file to runtimes for sdist (#25311)
• Remove PyPartitioning (#25303)
• Directly take CloudScheme in parse_cloud_options() (#25304)
• Refactor dt_range functions (#25225)
• Fix typo in CI release workflow (#25309)
• Use dedicated runtime packages from template (#25284)
• Simplify sink parameter passing from Python (#25302)
• Add test for unique with column subset (#25241)
• Fix Decimal precision annotation (#25227)
• Add LazyFrame.pivot (#25016)

Thank you to all our contributors for making this release possible! @AndreaBozzo, @DannyStoll1, @EndPositive, @JakubValtar, @Jesse-Bakker, @Kevin-Patyk, @MarcoGorelli, @TNieuwdorp, @Voultapher, @alexander-beedie, @borchero, @c-peters, @cBournhonesque, @camriddell, @carnarez, @cmdlineluser, @coastalwhite, @cr7pt0gr4ph7, @davanstrien, @davidia, @dsprenkels, @etiennebacher, @feliblo, @guilhem-dvr, @itamarst, @jannickj, @jetuk, @kdn36, @lun3x, @marinegor, @mcrumiller, @nameexhaustion, @orlp, @pomo-mondreganto, @ritchie46, @vyasr and @wtn