pola-rs/polars: Rust Polars 0.40.0

💥 Breaking changes

• Remove incremental read based batched CSV reader (#16259)
• separate rolling_*_by from rolling_*(..., by=...) in Rust (#16102)
• Move CSV read options from CsvReader to CsvReadOptions (#16126)
• Rename all 'Chunk's to RecordBatch (#16063)
• prepare for join coalescing argument (#15418)
• Rename to CsvParserOptions to CsvReaderOptions, use in CsvReader (#15919)
• Add context trace to LazyFrame conversion errors (#15761)
• Move schema resolving of file scan to IR phase (#15739)
• Move schema resolving to IR phase. (#15714)
• Rename LogicalPlan and builders to reflect their uses better (#15712)

🚀 Performance improvements

• Use branchless uleb128 decoding for parquet (#16352)
• Reduce error bubbling in parquet hybrid_rle (#16348)
• use is_sorted in ewm_mean_by, deprecate check_sorted (#16335)
• Optimize is_sorted for numeric data (#16333)
• do not use pyo3-built (#16309)
• Faster bitpacking for Parquet writer (#16278)
• Avoid importing ctypes.util in CPU check script if possible (#16307)
• Don't rechunk when converting DataFrame to numpy/ndarray (#16288)
• use zeroed vec in ewm_mean_by for sorted fastpath (#16265)
• use zeroable_vec in ewm_mean_by (#16166)
• Improve cost of chunk_idx compute (#16154)
• Don't rechunk by default in concat (#16128)
• Ensure rechunk is parallel (#16127)
• Don't traverse deep datasets that we repr as union in CSE (#16096)
• Ensure better chunk sizes (#16071)
• Don't rechunk in parallel collection (#15907)
• Improve non-trivial list aggregations (#15888)
• Ensure we hit specialized gather for binary/strings (#15886)
• Limit the cache size for to_datetime (#15826)
• skip initial null items and don't recompute slope in interpolate (#15819)
• Fix quadratic in binview growable same source (#15734)

✨ Enhancements

• Raise when joining on the same keys twice (#16329)
• Don't require data to be sorted by by column in rolling_*_by operations (#16249)
• Add struct.field expansion (regex, wildcard, columns) (#16320)
• Faster bitpacking for Parquet writer (#16278)
• Add struct.with_fields (#16305)
• Handle implicit SQL string → temporal conversion in the BETWEEN clause (#16279)
• Add new index/range based selector cs.by_index, allow multiple indices for nth (#16217)
• Show warning if expressions are very deep (#16233)
• Native CSV file list reading (#16180)
• Register memory mapped files and raise when written to (#16208)
• Raise when encountering invalid supertype in functions during conversion (#16182)
• Add SQL support for GROUP BY ALL syntax and fix several issues with aliased group keys (#16179)
• Allow implicit string → temporal conversion in SQL comparisons (#15958)
• separate rolling_*_by from rolling_*(..., by=...) in Rust (#16102)
• Add run-length encoding to Parquet writer (#16125)
• add date pattern dd.mm.YYYY (#16045)
• Add RLE to RLE_DICTIONARY encoder (#15959)
• Support non-coalescing joins in default engine (#16036)
• Move diagonal & horizontal concat schema resolving to IR phase (#16034)
• raise more informative error messages in rolling_* aggregations instead of panicking (#15979)
• Convert concat during IR conversion (#16016)
• Improve dynamic supertypes (#16009)
• Additional uint datatype support for the SQL interface (#15993)
• Support Decimal read from IPC (#15965)
• Add typed collection from par iterators (#15961)
• Add by argument for Expr.top_k and Expr.bottom_k (#15468)
• Add option to disable globbing in csv (#15930)
• Add option to disable globbing in parquet (#15928)
• Rename to CsvParserOptions to CsvReaderOptions, use in CsvReader (#15919)
• Expressify dt.round (#15861)
• Improve error messages in context stack (#15881)
• Add dynamic literals to ensure schema correctness (#15832)
• dt.truncate supports broadcasting lhs (#15768)
• Expressify str.json_path_match (#15764)
• Support decimal float parsing in CSV (#15774)
• Add context trace to LazyFrame conversion errors (#15761)

🐞 Bug fixes

• correct AExpr.to_field for bitwise and logical and/or (#16360)
• cargo clippy for uleb128 safety comment (#16368)
• Infer CSV schema as supertype of all files (#16349)
• Address overflow combining u64 hashes in Debug builds (#16323)
• Don't exclude explicitly named columns in group-by context' expr expansion (#16318)
• Harden Series.reshape against invalid parameters (#16281)
• Fix list.sum dtype for boolean (#16290)
• Don't stackoverflow on all/any horizontal (#16287)
• compilation error when both lazy and ipc features are enabled (#16284)
• `rolling_*_by was throwing incorrect error when dataframe was sorted by contained multiple chunks (#16247)
• Clippy Error for CPUID (#16241)
• Reading CSV with low_memory gave no data (#16231)
• Empty unique (#16214)
• Fix empty drop nulls (#16213)
• Fix get expression group-by state (#16189)
• Fix rolling empty group OOB (#16186)
• offset=-0i was being treated differently to offset=0i in rolling (#16184)
• Fix panic on empty frame joins (#16181)
• Fix streaming glob slice (#16174)
• Fix CSV skip_rows_after_header for streaming (#16176)
• Flush parquet at end of batches tick (#16073)
• Check CSE name aliases for collisions. (#16149)
• Don't override CSV reader encoding with lossy UTF-8 (#16151)
• Add missing allow macros for windows (#16130)
• Ensure hex and bitstring literals work inside SQL IN clauses (#16101)
• Revert "Add RLE to RLE_DICTIONARY encoder" (#16113)
• Respect user passed 'reader_schema' in 'scan_csv' (#16080)
• Lazy csv + projection; respect null values arg (#16077)
• Materialize dtypes when converting to arrow (#16074)
• Fix casting decimal to decimal for high precision (#16049)
• Fix printing max scale decimals (#16048)
• Decimal supertype for dyn int (#16046)
• Do not set sorted flag on lexical sorting (#16032)
• properly handle nulls in DictionaryArray::iter_typed (#16013)
• Fix CSE case where upper plan has no projection (#16011)
• Crash/incorrect group_by/n_unique on categoricals created by (q)cut (#16006)
• Ternary supertype dynamics (#15995)
• Treat splitting by empty string as iterating over chars (#15922)
• Fix PartialEq for DataType::Unknown (#15992)
• Do not reverse null indices in descending arg_sort (#15974)
• Finish adding typed_lit to help schema determination in SQL "extract" func (#15955)
• do not panic when comparing against categorical with incompatible dtype (#15857)
• Join validation for multiple keys (#15947)
• Set default limit for String column display to 30 and fix edge cases (#15934)
• typo in add_half_life takes ln(negative) (#15932)
• Remove ffspec from parquet reader (#15927)
• avoid WRITE+EXEC for CPUID check (#15912)
• fix inconsistent decimal formatting (#15457)
• Preserve NULLs for is_not_nan (#15889)
• double projection check should only take the upstream projections into account (#15901)
• Ensure we don't create invalid frames when combining unit lit + … (#15903)
• Clear cached rename schema (#15902)
• Fix OOB in struct lit/agg aggregation (#15891)
• create (q)cut labels in fixed order (#15843)
• Tag shrink_dtype as non-streaming (#15828)
• drop-nulls edge case; remove drop-nulls special case (#15815)
• ewm_mean_by was skipping initial nulls when it was already sorted by "by" column (#15812)
• Consult cgroups to determine free memory (#15798)
• raise if index count like 2i is used when performing rolling, group_by_dynamic, upsample, or other temporal operatios (#15751)
• Don't deduplicate sort that has slice pushdown (#15784)
• Fix incorrect is_between pushdown to scan_pyarrow_dataset (#15769)
• Handle null index correctly for list take (#15737)
• Preserve lexical ordering on concat (#15753)
• Remove incorrect unsafe pointer cast for int -> enum (#15740)
• pass series name to apply for cut/qcut (#15715)
• count of null column shouldn't panic in agg context (#15710)

📖 Documentation

• Clarify arrow usage (#16152)
• Solve inconsistency between code and comment (#16135)
• add filter docstring examples to date and datetime (#15996)
• update the link to R API docs (#15973)
• Fix a typo in categorical section of the user guide (#15777)
• Fix incorrect column name in LazyFrame.sort doc example (#15658)

📦 Build system

• Update Rust nightly toolchain version (#16222)
• Don't import jemalloc (#15942)
• Use default allocator for lts-cpu (#15941)
• replace all macos-latest referrals with macos-13 (#15926)
• pin mimalloc and macos-13 (#15925)
• use jemalloc in lts-cpu (#15913)

🛠️ Other improvements

• simplify interpolate code, add test for rolling_*_by with nulls (#16334)
• Move expression expansion to conversion module (#16331)
• Add polars-expr README (#16316)
• Move physical expressions to new crate (#16306)
• Use cls (not self) in classmethods (#16303)
• conditionally print the CSEs (#16292)
• Rename ChunkedArray.chunk_id to chunk_lengths (#16273)
• Use Scalar instead of Series some aggregations (#16277)
• Use CsvReadOptions in LazyCsvReader (#16283)
• Do not hardcode bash path in Makefile (#16263)
• Add IR::Reduce (not yet implemented) (#16216)
• Remove incremental read based batched CSV reader (#16259)
• move all describe, describe_tree and dot-viz code to IR instead of DslPlan (#16237)
• move describe to IR instead of DSL (#16191)
• Use Duration.is_zero instead of comparing Duration.duration_ns to 0 (#16195)
• Remove unused code (#16175)
• Don't override CSV reader encoding with lossy UTF-8 (#16151)
• Move CSV read options from CsvReader to CsvReadOptions (#16126)
• Bump sccache action (#16088)
• Fix failures in test coverage workflow (#16083)
• Rename all 'Chunk's to RecordBatch (#16063)
• Use UnionArgs for DSL side (#16017)
• Add some comments (#16008)
• prepare for join coalescing argument (#15418)
• Pin coverage job to MacOS 13 for now (#15918)
• Reorganize from_iter and dispatch to collect_ca when possible (#15904)
• More polars-io cleanup (#15885)
• Improve type-coercion (#15879)
• Move type coercion to IR conversion phase (#15868)
• Reorganize polars_io::parquet module (#15860)
• Reorganize polars_io::csv module (#15831)
• Always expand horizontal_any/all (#15816)
• Rename decimal_float to decimal_comma (#15817)
• Move IO-related options structs to polars-io (#15806)
• Split coverage calculation (#15780)
• Update readme (#15787)
• Move schema resolving of file scan to IR phase (#15739)
• Factor out ensure_is_constant_duration (#15733)
• Move schema resolving to IR phase. (#15714)
• Rename LogicalPlan and builders to reflect their uses better (#15712)

Thank you to all our contributors for making this release possible! @CanglongCl, @JulianCologne, @KDruzhkin, @MarcoGorelli, @NedJWestern, @NexVeridian, @NickCondron, @Robinsane, @ShivMunagala, @TobiasDummschat, @YichiZhang0613, @alexander-beedie, @avimallu, @bertiewooster, @brandon-b-miller, @c-peters, @coastalwhite, @dangotbanned, @datenzauberai, @deanm0000, @dependabot, @dependabot[bot], @eitsupi, @gasmith, @haocheng6, @ion-elgreco, @itamarst, @janpipek, @jr200, @jrycw, @jsarbach, @luke396, @marenwestermann, @max-muoto, @mbuhidar, @nameexhaustion, @orlp, @pydanny, @r-brink, @reswqa, @ritchie46, @stinodego, @thalassemia, @tharunsuresh-code, @twoertwein, @wence- and @wsyxbcl