<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements.  See the NOTICE file
distributed with this work for additional information
regarding copyright ownership.  The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License.  You may obtain a copy of the License at

  http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied.  See the License for the
specific language governing permissions and limitations
under the License.
-->

# Apache DataFusion 52.0.0 Changelog

This release consists of 549 commits from 121 contributors. See credits at the end of this changelog for more information.

See the [upgrade guide](https://datafusion.apache.org/library-user-guide/upgrading.html) for information on how to upgrade from previous versions.

**Breaking changes:**

- Force `FileSource` to be constructed with a `Schema` [#18386](https://github.com/apache/datafusion/pull/18386) (adriangb)
- Support Arrow IPC Stream Files [#18457](https://github.com/apache/datafusion/pull/18457) (corasaurus-hex)
- Change default of `AggregateUDFImpl::supports_null_handling_clause` to `false` [#18441](https://github.com/apache/datafusion/pull/18441) (Jefffrey)
- [Minor] Remove RawTableAllocExt [#18748](https://github.com/apache/datafusion/pull/18748) (Dandandan)
- Change `CacheAccessor::remove` to take `&self` rather than `&mut self` [#18726](https://github.com/apache/datafusion/pull/18726) (alchemist51)
- Move statistics handling into FileScanConfig [#18721](https://github.com/apache/datafusion/pull/18721) (adriangb)
- chore: remove `pyarrow` feature [#18528](https://github.com/apache/datafusion/pull/18528) (timsaucer)
- Limit visibility of internal impl functions in function crates [#18877](https://github.com/apache/datafusion/pull/18877) (Jefffrey)
- FFI: return underlying trait type when converting from FFI structs [#18672](https://github.com/apache/datafusion/pull/18672) (timsaucer)
- Refactor crypto functions code [#18664](https://github.com/apache/datafusion/pull/18664) (Jefffrey)
- move projection handling into FileSource [#18627](https://github.com/apache/datafusion/pull/18627) (adriangb)
- Add PhysicalOptimizerRule::optimize_plan to allow passing more context into optimizer rules [#18739](https://github.com/apache/datafusion/pull/18739) (adriangb)
- Optimize planning / stop cloning Strings / Fields so much (2-3% faster planning time) [#18415](https://github.com/apache/datafusion/pull/18415) (alamb)
- Adds memory-bound DefaultListFilesCache [#18855](https://github.com/apache/datafusion/pull/18855) (BlakeOrth)
- Allow Logical expression ScalarVariable to represent an extension type or metadata [#18243](https://github.com/apache/datafusion/pull/18243) (batmnnn)
- feat: Implement the `statistics_cache` function [#19054](https://github.com/apache/datafusion/pull/19054) (nuno-faria)
- Move `newlines_in_values` from `FileScanConfig` to `CsvSource` [#19313](https://github.com/apache/datafusion/pull/19313) (adriangb)
- Remove SchemaAdapter [#19345](https://github.com/apache/datafusion/pull/19345) (adriangb)
- feat: hash partitioning satisfies subset [#19304](https://github.com/apache/datafusion/pull/19304) (gene-bordegaray)
- feat: update FFI TableProvider and ExecutionPlan to use FFI Session and TaskContext [#19281](https://github.com/apache/datafusion/pull/19281) (timsaucer)
- Allow logical optimizer to be run without evaluating now() & refactor SimplifyInfo [#19505](https://github.com/apache/datafusion/pull/19505) (adriangb)
- Make default ListingFilesCache table scoped [#19616](https://github.com/apache/datafusion/pull/19616) (jizezhang)

**Performance related:**

- Normalize partitioned and flat object listing [#18146](https://github.com/apache/datafusion/pull/18146) (BlakeOrth)
- perf: Improve NLJ for very small right side case [#17562](https://github.com/apache/datafusion/pull/17562) (2010YOUY01)
- Consolidate `EliminateNestedUnion` and `EliminateOneUnion` optimizer rules' [#18678](https://github.com/apache/datafusion/pull/18678) (alamb)
- perf: improve performance of `vectorized_equal_to` for `PrimitiveGroupValueBuilder` in multi group by aggregation [#17977](https://github.com/apache/datafusion/pull/17977) (rluvaton)
- optimizer: Support dynamic filter in `MIN/MAX` aggregates [#18644](https://github.com/apache/datafusion/pull/18644) (2010YOUY01)
- perf: use `new_repeated` when converting scalar to an array [#19018](https://github.com/apache/datafusion/pull/19018) (rluvaton)
- perf: optimize CASE WHEN lookup table (2.5-22.5 times faster) [#18183](https://github.com/apache/datafusion/pull/18183) (rluvaton)
- add specialized InList implementations for common scalar types [#18832](https://github.com/apache/datafusion/pull/18832) (adriangb)
- Add hashing microbenchmark `with_hashes` [#19373](https://github.com/apache/datafusion/pull/19373) (alamb)
- Optimize muti-column grouping with StringView/ByteView (option 2) - 25% faster [#19413](https://github.com/apache/datafusion/pull/19413) (alamb)
- Optimize hashing for StringView and ByteView (15-70% faster) [#19374](https://github.com/apache/datafusion/pull/19374) (alamb)
- perf: Improve performance of `to_hex` (> 2x) [#19503](https://github.com/apache/datafusion/pull/19503) (andygrove)
- perf: improve performance of string repeat [#19502](https://github.com/apache/datafusion/pull/19502) (andygrove)
- perf: Optimize `starts_with` and `ends_with` for scalar arguments [#19516](https://github.com/apache/datafusion/pull/19516) (andygrove)
- perf: improve performance of string replace [#19530](https://github.com/apache/datafusion/pull/19530) (viirya)
- perf: improve performance of levenshtein by reusing cache buffer [#19532](https://github.com/apache/datafusion/pull/19532) (viirya)
- perf: improve performance of translate by reusing buffers [#19533](https://github.com/apache/datafusion/pull/19533) (viirya)
- perf: Optimize `contains` for scalar search arg [#19529](https://github.com/apache/datafusion/pull/19529) (andygrove)
- perf: improve performance of lpad/rpad by reusing buffers [#19558](https://github.com/apache/datafusion/pull/19558) (viirya)
- perf: optimize regexp_count to avoid String allocation when start position is provided [#19553](https://github.com/apache/datafusion/pull/19553) (viirya)
- perf: Improve performance of `md5` [#19568](https://github.com/apache/datafusion/pull/19568) (andygrove)
- perf: optimize strpos by eliminating double iteration for UTF-8 [#19572](https://github.com/apache/datafusion/pull/19572) (viirya)
- perf: optimize factorial function performance [#19575](https://github.com/apache/datafusion/pull/19575) (getChan)
- perf: Improve performance of ltrim, rtrim, btrim [#19551](https://github.com/apache/datafusion/pull/19551) (andygrove)
- perf: optimize `HashTableLookupExpr::evaluate` [#19602](https://github.com/apache/datafusion/pull/19602) (UBarney)
- perf: Improve performance of `split_part` [#19570](https://github.com/apache/datafusion/pull/19570) (andygrove)
- Optimize `Nullstate` / accumulators [#19625](https://github.com/apache/datafusion/pull/19625) (Dandandan)

**Implemented enhancements:**

- feat: Enhance `array_slice` functionality to support `ListView` and `LargeListView` types [#18432](https://github.com/apache/datafusion/pull/18432) (Weijun-H)
- feat: support complex expr for prepared statement argument [#18383](https://github.com/apache/datafusion/pull/18383) (chenkovsky)
- feat: Implement `SessionState::create_logical_expr_from_sql_expr` [#18423](https://github.com/apache/datafusion/pull/18423) (petern48)
- feat: added clippy::needless_pass_by_value lint rule to datafusion/expr [#18532](https://github.com/apache/datafusion/pull/18532) (Gohlub)
- feat: support nested key for get_field [#18394](https://github.com/apache/datafusion/pull/18394) (chenkovsky)
- feat: Add `ansi` enable parameter for execution config [#18635](https://github.com/apache/datafusion/pull/18635) (comphead)
- feat: Add evaluate_to_arrays function [#18446](https://github.com/apache/datafusion/pull/18446) (EmilyMatt)
- feat: support named variables & defaults for `CREATE FUNCTION` [#18450](https://github.com/apache/datafusion/pull/18450) (r1b)
- feat: Add new() constructor for CachedParquetFileReader [#18575](https://github.com/apache/datafusion/pull/18575) (petern48)
- feat: support decimal for math functions: power [#18032](https://github.com/apache/datafusion/pull/18032) (theirix)
- feat: selectivity metrics (for Explain Analyze) in Hash Join [#18488](https://github.com/apache/datafusion/pull/18488) (feniljain)
- feat: Handle edge case with `corr` with single row and `NaN` [#18677](https://github.com/apache/datafusion/pull/18677) (comphead)
- feat: support spark csc [#18642](https://github.com/apache/datafusion/pull/18642) (psvri)
- feat: support spark sec [#18728](https://github.com/apache/datafusion/pull/18728) (psvri)
- feat(parquet): Implement `scan_efficiency_ratio` metric for parquet reading [#18577](https://github.com/apache/datafusion/pull/18577) (petern48)
- feat: Enhance map handling to support NULL map values [#18531](https://github.com/apache/datafusion/pull/18531) (Weijun-H)
- feat: add RESET statement for configuration variabless [#18408](https://github.com/apache/datafusion/pull/18408) (Weijun-H)
- feat: add human-readable formatting to EXPLAIN ANALYZE metrics #18689 [#18734](https://github.com/apache/datafusion/pull/18734) (T2MIX)
- feat: support Spark-compatible `abs` math function part 1 - non-ANSI mode [#18205](https://github.com/apache/datafusion/pull/18205) (hsiang-c)
- feat: Support Show runtime settings [#18564](https://github.com/apache/datafusion/pull/18564) (Weijun-H)
- feat(small): Support `<slt:ignore>` marker in `sqllogictest` for non-deterministic expected parts [#18857](https://github.com/apache/datafusion/pull/18857) (2010YOUY01)
- feat: allow custom caching via logical node [#18688](https://github.com/apache/datafusion/pull/18688) (jizezhang)
- feat: add `array_slice` benchmark [#18879](https://github.com/apache/datafusion/pull/18879) (dqkqd)
- feat: Support recursive queries with a distinct 'UNION' [#18254](https://github.com/apache/datafusion/pull/18254) (Tpt)
- feat: Makes error macros hygienic [#18995](https://github.com/apache/datafusion/pull/18995) (Tpt)
- feat: Add builder API for CreateExternalTable to reduce verbosity [#19066](https://github.com/apache/datafusion/pull/19066) (AryanBagade)
- feat(spark): Implement Spark functions `url_encode`, `url_decode` and `try_url_decode` [#17399](https://github.com/apache/datafusion/pull/17399) (anhvdq)
- feat: Move DefaultMetadataCache into its own module [#19125](https://github.com/apache/datafusion/pull/19125) (AryanBagade)
- feat: Add `remove_optimizer_rule` to `SessionContext` [#19209](https://github.com/apache/datafusion/pull/19209) (nuno-faria)
- feat: integrate batch coalescer with repartition exec [#19002](https://github.com/apache/datafusion/pull/19002) (jizezhang)
- feat: Preserve File Partitioning From File Scans [#19124](https://github.com/apache/datafusion/pull/19124) (gene-bordegaray)
- feat: Add constant column extraction and rewriting for projections in ParquetOpener [#19136](https://github.com/apache/datafusion/pull/19136) (Weijun-H)
- feat: Support sliding window queries for MedianAccumulator by implementing `retract_batch` [#19278](https://github.com/apache/datafusion/pull/19278) (petern48)
- feat: add compression level configuration for JSON/CSV writers [#18954](https://github.com/apache/datafusion/pull/18954) (Smotrov)
- feat(spark): implement Spark `try_sum` function [#18569](https://github.com/apache/datafusion/pull/18569) (davidlghellin)
- feat: Support log for Decimal32 and Decimal64 [#18999](https://github.com/apache/datafusion/pull/18999) (Mark1626)
- feat(proto): Add protobuf serialization for HashExpr [#19379](https://github.com/apache/datafusion/pull/19379) (adriangb)
- feat: Add decimal support for round [#19384](https://github.com/apache/datafusion/pull/19384) (kumarUjjawal)
- Support nested field access in `get_field` with multiple path arguments [#19389](https://github.com/apache/datafusion/pull/19389) (adriangb)
- feat: fix matching for named parameters with non-lowercase signatures [#19378](https://github.com/apache/datafusion/pull/19378) (bubulalabu)
- feat: Add per-expression evaluation timing metrics to ProjectionExec [#19447](https://github.com/apache/datafusion/pull/19447) (2010YOUY01)
- feat: Improve sort memory resilience [#19494](https://github.com/apache/datafusion/pull/19494) (EmilyMatt)
- feat: Add DELETE/UPDATE hooks to TableProvider trait and to MemTable implementation [#19142](https://github.com/apache/datafusion/pull/19142) (ethan-tyler)
- feat: implement partition_statistics for WindowAggExec [#18534](https://github.com/apache/datafusion/pull/18534) (0xPoe)
- feat: integrate batch coalescer with async fn exec [#19342](https://github.com/apache/datafusion/pull/19342) (feniljain)
- feat: output statistics for constant columns in projections [#19419](https://github.com/apache/datafusion/pull/19419) (shashidhar-bm)
- feat: `to_time` function [#19540](https://github.com/apache/datafusion/pull/19540) (kumarUjjawal)
- feat: Implement Spark functions hour, minute, second [#19512](https://github.com/apache/datafusion/pull/19512) (andygrove)
- feat: plan-time SQL expression simplifying [#19311](https://github.com/apache/datafusion/pull/19311) (theirix)
- feat: Implement Spark function `space` [#19610](https://github.com/apache/datafusion/pull/19610) (kazantsev-maksim)
- feat: Implement `partition_statistics` API for `SortMergeJoinExec` [#19567](https://github.com/apache/datafusion/pull/19567) (kumarUjjawal)
- feat: add list_files_cache table function for `datafusion-cli` [#19388](https://github.com/apache/datafusion/pull/19388) (jizezhang)
- feat: implement metrics for AsyncFuncExec [#19626](https://github.com/apache/datafusion/pull/19626) (feniljain)
- feat: split BatchPartitioner::try_new into hash and round-robin constructors [#19668](https://github.com/apache/datafusion/pull/19668) (mohit7705)
- feat: add Time type support to date_trunc function [#19640](https://github.com/apache/datafusion/pull/19640) (kumarUjjawal)
- feat: Allow log with non-integer base on decimals [#19372](https://github.com/apache/datafusion/pull/19372) (Yuvraj-cyborg)

**Fixed bugs:**

- fix: Eliminate consecutive repartitions [#18521](https://github.com/apache/datafusion/pull/18521) (gene-bordegaray)
- fix: `with_param_values` on `LogicalPlan::EmptyRelation` returns incorrect schema [#18286](https://github.com/apache/datafusion/pull/18286) (dqkqd)
- fix: Nested arrays should not get a field in lookup [#18745](https://github.com/apache/datafusion/pull/18745) (EmilyMatt)
- fix: update schema's data type for `LogicalPlan::Values` after placeholder substitution [#18740](https://github.com/apache/datafusion/pull/18740) (dqkqd)
- fix: Pick correct columns in Sort Merge Equijoin [#18772](https://github.com/apache/datafusion/pull/18772) (tglanz)
- fix: remove `WorkTableExec` special case in `reset_plan_states` [#18803](https://github.com/apache/datafusion/pull/18803) (geoffreyclaude)
- fix: display the failed sqllogictest file and query that failed in case of a panic [#18785](https://github.com/apache/datafusion/pull/18785) (rluvaton)
- fix: preserve byte-size statistics in AggregateExec [#18885](https://github.com/apache/datafusion/pull/18885) (Tamar-Posen)
- fix: Track elapsed_compute metric for CSV scans [#18901](https://github.com/apache/datafusion/pull/18901) (Nithurshen)
- fix: Implement Substrait consumer support for like_match, like_imatch, and negated variants [#18929](https://github.com/apache/datafusion/pull/18929) (Nithurshen)
- fix: Initialize CsvOptions::double_quote from proto_opts.double_quote [#18967](https://github.com/apache/datafusion/pull/18967) (martin-g)
- fix: `rstest` is a DEV dependency [#19014](https://github.com/apache/datafusion/pull/19014) (crepererum)
- fix: partition pruning stats pruning when multiple values are present [#18923](https://github.com/apache/datafusion/pull/18923) (Mark1626)
- fix: deprecate data_type_and_nullable and simplify API usage [#18869](https://github.com/apache/datafusion/pull/18869) (BipulLamsal)
- fix: pre-warm listing file statistics cache during listing table creation [#18971](https://github.com/apache/datafusion/pull/18971) (bharath-techie)
- fix: log metadata differences when comparing physical and logical schema [#19070](https://github.com/apache/datafusion/pull/19070) (erratic-pattern)
- fix: fix panic when lo is greater than hi [#19099](https://github.com/apache/datafusion/pull/19099) (tshauck)
- fix: escape underscores when simplifying `starts_with` [#19077](https://github.com/apache/datafusion/pull/19077) (willemv)
- fix: custom nullability for length (#19175) [#19182](https://github.com/apache/datafusion/pull/19182) (skushagra)
- fix: inverted null_percent logic in in_list benchmark [#19204](https://github.com/apache/datafusion/pull/19204) (geoffreyclaude)
- fix: Ensure column names do not change with `expand_views_at_output` [#19019](https://github.com/apache/datafusion/pull/19019) (nuno-faria)
- fix: bitmap_count should report nullability correctly [#19195](https://github.com/apache/datafusion/pull/19195) (harshitsaini17)
- fix: bit_count function to report nullability correctly [#19197](https://github.com/apache/datafusion/pull/19197) (harshitsaini17)
- fix: derive custom nullability for spark `bit_shift` [#19222](https://github.com/apache/datafusion/pull/19222) (kumarUjjawal)
- fix: spark elt custom nullability [#19207](https://github.com/apache/datafusion/pull/19207) (EeshanBembi)
- fix: `array_remove`/`array_remove_n`/`array_remove_all` not using the same nullability as the input [#19259](https://github.com/apache/datafusion/pull/19259) (rluvaton)
- fix: typo in sql/ddl [#19276](https://github.com/apache/datafusion/pull/19276) (mag1c1an1)
- fix: flaky cache test [#19140](https://github.com/apache/datafusion/pull/19140) (xonx4l)
- fix: Add custom nullability for Spark ILIKE function [#19206](https://github.com/apache/datafusion/pull/19206) (Eshaan-byte)
- fix: derive custom nullability for spark `map_from_arrays` [#19275](https://github.com/apache/datafusion/pull/19275) (kumarUjjawal)
- fix: derive custom nullability for spark map_from_entries [#19274](https://github.com/apache/datafusion/pull/19274) (kumarUjjawal)
- fix: derive custom nullable for spark `make_dt_interval` [#19236](https://github.com/apache/datafusion/pull/19236) (kumarUjjawal)
- fix: derive custome nullable for the spark last_day [#19232](https://github.com/apache/datafusion/pull/19232) (kumarUjjawal)
- fix: derive custom nullable for spark `date_sub` [#19225](https://github.com/apache/datafusion/pull/19225) (kumarUjjawal)
- fix: Fix a few minor issues with join metrics [#19283](https://github.com/apache/datafusion/pull/19283) (linhr)
- fix: derive nullability for spark `bit_get` [#19220](https://github.com/apache/datafusion/pull/19220) (kumarUjjawal)
- fix: pow() with integer base and negative float exponent returns error [#19303](https://github.com/apache/datafusion/pull/19303) (adriangb)
- fix(concat): correct nullability inference (nullable only if all arguments nullable) [#19189](https://github.com/apache/datafusion/pull/19189) (ujjwaltwri)
- fix: Added nullable return from date_add(#19151) [#19229](https://github.com/apache/datafusion/pull/19229) (manishkr)
- fix: spark sha1 nullability reporting [#19242](https://github.com/apache/datafusion/pull/19242) (shashidhar-bm)
- fix: derive custom nullability for the spark `next_day` [#19253](https://github.com/apache/datafusion/pull/19253) (kumarUjjawal)
- fix: preserve ListFilesCache TTL when not set in config [#19401](https://github.com/apache/datafusion/pull/19401) (shashidhar-bm)
- fix: projection for `CooperativeExec` and `CoalesceBatchesExec` [#19400](https://github.com/apache/datafusion/pull/19400) (haohuaijin)
- fix: spark crc32 custom nullability [#19271](https://github.com/apache/datafusion/pull/19271) (watanaberin)
- fix: Fix skip aggregate test to cover regression [#19461](https://github.com/apache/datafusion/pull/19461) (kumarUjjawal)
- fix: [19450]Added flush for tokio file(substrait) write [#19456](https://github.com/apache/datafusion/pull/19456) (manishkr)
- fix: csv schema_infer_max_records set to 0 return null datatype [#19432](https://github.com/apache/datafusion/pull/19432) (haohuaijin)
- fix: Add custom nullability for Spark LIKE function [#19218](https://github.com/apache/datafusion/pull/19218) (KaranPradhan266)
- fix: implement custom nullability for spark abs function [#19395](https://github.com/apache/datafusion/pull/19395) (batmnnn)
- fix: custom nullability for format_string (#19173) [#19190](https://github.com/apache/datafusion/pull/19190) (skushagra)
- fix: Implement `reset_state` for `LazyMemoryExec` [#19362](https://github.com/apache/datafusion/pull/19362) (nuno-faria)
- fix: CteWorkTable: properly apply TableProvider::scan projection argument [#18993](https://github.com/apache/datafusion/pull/18993) (Tpt)
- fix: Median() integer overflow [#19509](https://github.com/apache/datafusion/pull/19509) (kumarUjjawal)
- fix: Reverse row selection should respect the row group index [#19557](https://github.com/apache/datafusion/pull/19557) (zhuqi-lucas)
- fix: emit empty RecordBatch for empty file writes [#19370](https://github.com/apache/datafusion/pull/19370) (nlimpid)
- fix: handle invalid byte ranges in calculate_range for single-line files [#19607](https://github.com/apache/datafusion/pull/19607) (vigimite)
- fix: NULL handling in arrow_intersect and arrow_union [#19415](https://github.com/apache/datafusion/pull/19415) (feniljain)
- fix(doc): close #19393, make upgrading guide match v51 api [#19648](https://github.com/apache/datafusion/pull/19648) (mag1c1an1)
- fix(spark): Use wrapping addition/subtraction in `SparkDateAdd` and `SparkDateSub` [#19377](https://github.com/apache/datafusion/pull/19377) (mzabaluev)
- fix(functions): Make translate function postgres compatible [#19630](https://github.com/apache/datafusion/pull/19630) (devanshu0987)
- fix: Return Int for Date - Date instead of duration [#19563](https://github.com/apache/datafusion/pull/19563) (kumarUjjawal)
- fix: DynamicFilterPhysicalExpr violates Hash/Eq contract [#19659](https://github.com/apache/datafusion/pull/19659) (kumarUjjawal)

**Documentation updates:**

- [main] Update version to 51.0.0, add Changelog (#18551) [#18565](https://github.com/apache/datafusion/pull/18565) (alamb)
- refactor: include metric output_batches into BaselineMetrics [#18491](https://github.com/apache/datafusion/pull/18491) (nmbr7)
- chore(deps): bump maturin from 1.9.6 to 1.10.0 in /docs [#18590](https://github.com/apache/datafusion/pull/18590) (dependabot[bot])
- Update release download links on download page [#18550](https://github.com/apache/datafusion/pull/18550) (alamb)
- docs: fix rustup cmd for adding rust-analyzer [#18605](https://github.com/apache/datafusion/pull/18605) (Jefffrey)
- Enforce explicit opt-in for `WITHIN GROUP` syntax in aggregate UDAFs [#18607](https://github.com/apache/datafusion/pull/18607) (kosiew)
- docs: fix broken catalog example links [#18765](https://github.com/apache/datafusion/pull/18765) (nlimpid)
- doc: Add documentation for error handling [#18762](https://github.com/apache/datafusion/pull/18762) (2010YOUY01)
- docs: Fix the examples for char_length() and character_length() [#18808](https://github.com/apache/datafusion/pull/18808) (martin-g)
- chore: Support 'untake' for unassigning github issues [#18637](https://github.com/apache/datafusion/pull/18637) (petern48)
- chore: Add filtered pending PRs link to main page [#18854](https://github.com/apache/datafusion/pull/18854) (comphead)
- Docs: Enhance contributor guide with testing section [#18852](https://github.com/apache/datafusion/pull/18852) (alamb)
- Docs: Enhance testing documentation with examples and links [#18851](https://github.com/apache/datafusion/pull/18851) (alamb)
- chore(deps): bump maturin from 1.10.0 to 1.10.2 in /docs [#18905](https://github.com/apache/datafusion/pull/18905) (dependabot[bot])
- Update links in documentation to point at new example locations [#18931](https://github.com/apache/datafusion/pull/18931) (alamb)
- Add Kubeflow Trainer to known users [#18935](https://github.com/apache/datafusion/pull/18935) (andreyvelich)
- Add PGO documentation section to crate configuration [#18959](https://github.com/apache/datafusion/pull/18959) (jatinkumarsingh)
- Add upgrade guide for PhysicalOptimizerRule::optimize_plan [#19030](https://github.com/apache/datafusion/pull/19030) (adriangb)
- doc: add `FilterExec` metrics to `user-guide/metrics.md` [#19043](https://github.com/apache/datafusion/pull/19043) (2010YOUY01)
- Add `force_filter_selections` to restore `pushdown_filters` behavior prior to parquet 57.1.0 upgrade [#19003](https://github.com/apache/datafusion/pull/19003) (alamb)
- Implement FFI task context and task context provider [#18918](https://github.com/apache/datafusion/pull/18918) (timsaucer)
- Minor: fix link errors in docs [#19088](https://github.com/apache/datafusion/pull/19088) (alamb)
- Cut `Parquet` over to PhysicalExprAdapter, remove `SchemaAdapter` [#18998](https://github.com/apache/datafusion/pull/18998) (adriangb)
- Update Committer / PMC list [#19105](https://github.com/apache/datafusion/pull/19105) (alamb)
- Revert adding PhysicalOptimizerRule::optimize_plan [#19186](https://github.com/apache/datafusion/pull/19186) (adriangb)
- Push down InList or hash table references from HashJoinExec depending on the size of the build side [#18393](https://github.com/apache/datafusion/pull/18393) (adriangb)
- Move partition handling out of PhysicalExprAdapter [#19128](https://github.com/apache/datafusion/pull/19128) (adriangb)
- Push down projection expressions into ParquetOpener [#19111](https://github.com/apache/datafusion/pull/19111) (adriangb)
- Track column sizes in Statistics; propagate through projections [#19113](https://github.com/apache/datafusion/pull/19113) (adriangb)
- Improve ProjectionExpr documentation and comments [#19263](https://github.com/apache/datafusion/pull/19263) (alamb)
- Update README occording to the new examples (#18529) [#19257](https://github.com/apache/datafusion/pull/19257) (cj-zhukov)
- Add make_time function [#19183](https://github.com/apache/datafusion/pull/19183) (Omega359)
- Update to_date udf function to support a consistent set of argument types [#19134](https://github.com/apache/datafusion/pull/19134) (Omega359)
- Add library user guide for extending SQL syntax [#19265](https://github.com/apache/datafusion/pull/19265) (geoffreyclaude)
- Add runtime config options for `list_files_cache_limit` and `list_files_cache_ttl` [#19108](https://github.com/apache/datafusion/pull/19108) (delamarch3)
- Minor: clean up titles and links n extending operators and optimizer pages [#19317](https://github.com/apache/datafusion/pull/19317) (alamb)
- Establish the high level API for sort pushdown and the optimizer rule and support reverse files and row groups [#19064](https://github.com/apache/datafusion/pull/19064) (zhuqi-lucas)
- Add Decimal support to Ceil and Floor [#18979](https://github.com/apache/datafusion/pull/18979) (kumarUjjawal)
- doc: add example for cache factory [#19139](https://github.com/apache/datafusion/pull/19139) (jizezhang)
- chore(deps): bump sphinx-reredirects from 1.0.0 to 1.1.0 in /docs [#19455](https://github.com/apache/datafusion/pull/19455) (dependabot[bot])
- Add:arrow_metadata() UDF [#19435](https://github.com/apache/datafusion/pull/19435) (xonx4l)
- Update date_bin to support Time32 and Time64 data types [#19341](https://github.com/apache/datafusion/pull/19341) (Omega359)
- Update `to_unixtime` udf function to support a consistent set of argument types [#19442](https://github.com/apache/datafusion/pull/19442) (kumarUjjawal)
- docs: Improve config tables' readability [#19522](https://github.com/apache/datafusion/pull/19522) (nuno-faria)
- Introduce `TypeSignatureClass::Any` [#19485](https://github.com/apache/datafusion/pull/19485) (Jefffrey)
- Enables DefaultListFilesCache by default [#19366](https://github.com/apache/datafusion/pull/19366) (BlakeOrth)
- Fix typo in contributor guide architecture section [#19613](https://github.com/apache/datafusion/pull/19613) (cdegroc)
- docs: fix typos in PartitionEvaluator trait documentation [#19631](https://github.com/apache/datafusion/pull/19631) (SolariSystems)
- Respect execution timezone in to_timestamp and related functions [#19078](https://github.com/apache/datafusion/pull/19078) (Omega359)
- perfect hash join [#19411](https://github.com/apache/datafusion/pull/19411) (UBarney)

**Other:**

- chore(deps): bump taiki-e/install-action from 2.62.46 to 2.62.47 [#18508](https://github.com/apache/datafusion/pull/18508) (dependabot[bot])
- Consolidate builtin functions examples (#18142) [#18523](https://github.com/apache/datafusion/pull/18523) (cj-zhukov)
- refactor: update cmp and nested data in binary operator [#18256](https://github.com/apache/datafusion/pull/18256) (sunng87)
- Fix: topk_aggregate benchmark failing [#18502](https://github.com/apache/datafusion/pull/18502) (randyli)
- refactor: Add `assert_or_internal_err!` macro for more ergonomic internal invariant checks [#18511](https://github.com/apache/datafusion/pull/18511) (2010YOUY01)
- chore: enforce clippy lint needless_pass_by_value to datafusion-physical-optimizer [#18555](https://github.com/apache/datafusion/pull/18555) (foskey51)
- chore: enforce clippy lint needless_pass_by_value for datafusion-sql [#18554](https://github.com/apache/datafusion/pull/18554) (foskey51)
- chore: enforce clippy lint needless_pass_by_value to physical-expr-common [#18556](https://github.com/apache/datafusion/pull/18556) (foskey51)
- chore: Enforce lint rule `clippy::needless_pass_by_value` to `datafusion-physical-expr` [#18557](https://github.com/apache/datafusion/pull/18557) (corasaurus-hex)
- Fix out-of-bounds access in SLT runner [#18562](https://github.com/apache/datafusion/pull/18562) (theirix)
- Make array_reverse faster for List and FixedSizeList [#18500](https://github.com/apache/datafusion/pull/18500) (vegarsti)
- Consolidate custom data source examples (#18142) [#18553](https://github.com/apache/datafusion/pull/18553) (cj-zhukov)
- chore(deps): bump taiki-e/install-action from 2.62.47 to 2.62.49 [#18581](https://github.com/apache/datafusion/pull/18581) (dependabot[bot])
- chore: Remove unused `tokio` dependency and clippy [#18598](https://github.com/apache/datafusion/pull/18598) (comphead)
- minor: enforce `clippy::needless_pass_by_value` for crates that don't require code changes. [#18586](https://github.com/apache/datafusion/pull/18586) (2010YOUY01)
- refactor: merge CoalesceAsyncExecInput into CoalesceBatches [#18540](https://github.com/apache/datafusion/pull/18540) (Tim-53)
- Enhance the help message for invalid command in datafusion-cli [#18603](https://github.com/apache/datafusion/pull/18603) (klion26)
- Update Release README.md with latest process [#18549](https://github.com/apache/datafusion/pull/18549) (alamb)
- Add timezone to date_trunc fast path [#18596](https://github.com/apache/datafusion/pull/18596) (hareshkh)
- Coalesce batches inside FilterExec [#18604](https://github.com/apache/datafusion/pull/18604) (Dandandan)
- Fix misleading boolean 'null' interval tests [#18620](https://github.com/apache/datafusion/pull/18620) (pepijnve)
- Clarify tests for `Interval::and`, `Interval::not`, and add `Interval::or` tests [#18621](https://github.com/apache/datafusion/pull/18621) (pepijnve)
- bugfix: correct regression on TableType for into_view [#18617](https://github.com/apache/datafusion/pull/18617) (timsaucer)
- Separating Benchmarks for physical sorted union over large columns in SQL planner based on Datatype [#18599](https://github.com/apache/datafusion/pull/18599) (logan-keede)
- Add RunEndEncoded type coercion [#18561](https://github.com/apache/datafusion/pull/18561) (vegarsti)
- minor: refactor with `assert_or_internal_err!()` in `datafusion/sql` [#18614](https://github.com/apache/datafusion/pull/18614) (2010YOUY01)
- chore: ASF tracking process on `.asf.yaml` [#18636](https://github.com/apache/datafusion/pull/18636) (comphead)
- Refactor bit aggregate functions signature [#18593](https://github.com/apache/datafusion/pull/18593) (Jefffrey)
- chore(deps): bump taiki-e/install-action from 2.62.49 to 2.62.50 [#18645](https://github.com/apache/datafusion/pull/18645) (dependabot[bot])
- bugfix: select_columns should validate column names [#18623](https://github.com/apache/datafusion/pull/18623) (timsaucer)
- Consolidate data io examples (#18142) [#18591](https://github.com/apache/datafusion/pull/18591) (cj-zhukov)
- Correct implementations of `NullableInterval::and` and `NullableInterval::or`. [#18625](https://github.com/apache/datafusion/pull/18625) (pepijnve)
- chore: ASF tracking process on `.asf.yaml` [#18652](https://github.com/apache/datafusion/pull/18652) (comphead)
- Refactor Spark bitshift signature [#18649](https://github.com/apache/datafusion/pull/18649) (Jefffrey)
- chore(deps): bump crate-ci/typos from 1.39.0 to 1.39.1 [#18667](https://github.com/apache/datafusion/pull/18667) (dependabot[bot])
- Update docs for aggregate repartition test [#18650](https://github.com/apache/datafusion/pull/18650) (xanderbailey)
- chore: Enforce lint rule `clippy::needless_pass_by_value` to `datafusion-catalog` [#18638](https://github.com/apache/datafusion/pull/18638) (Standing-Man)
- [main] Update Changelog (#18592) [#18616](https://github.com/apache/datafusion/pull/18616) (alamb)
- Refactor distinct aggregate implementations to use common buffer [#18348](https://github.com/apache/datafusion/pull/18348) (Jefffrey)
- chore: enforce lint rule `clippy::needless_pass_by_value` to `datafusion-datasource-avro` [#18641](https://github.com/apache/datafusion/pull/18641) (Standing-Man)
- Refactor Spark expm1 signature [#18655](https://github.com/apache/datafusion/pull/18655) (Jefffrey)
- chore(core): Enforce lint rule `clippy::needless_pass_by_value` to `datafusion-core` [#18640](https://github.com/apache/datafusion/pull/18640) (Standing-Man)
- Refactor substr signature [#18653](https://github.com/apache/datafusion/pull/18653) (Jefffrey)
- minor: Use allow->expect to explicitly suppress Clippy lint checks [#18686](https://github.com/apache/datafusion/pull/18686) (2010YOUY01)
- chore(deps): bump taiki-e/install-action from 2.62.50 to 2.62.51 [#18693](https://github.com/apache/datafusion/pull/18693) (dependabot[bot])
- chore(deps): bump crate-ci/typos from 1.39.1 to 1.39.2 [#18694](https://github.com/apache/datafusion/pull/18694) (dependabot[bot])
- Remove FilterExec from CoalesceBatches optimization rule, add fetch support [#18630](https://github.com/apache/datafusion/pull/18630) (Dandandan)
- minor: refactor with `assert_or_internal_err!()` in `datafusion/datasource` [#18697](https://github.com/apache/datafusion/pull/18697) (kumarUjjawal)
- chore: Enforce lint rule `clippy::needless_pass_by_value` to datafusion-datasource [#18682](https://github.com/apache/datafusion/pull/18682) (AryanBagade)
- [main] Update changelog for 51.0.0 RC2 [#18710](https://github.com/apache/datafusion/pull/18710) (alamb)
- Refactor Spark crc32/sha1 signatures [#18662](https://github.com/apache/datafusion/pull/18662) (Jefffrey)
- CI: try free up space in `Rust / cargo test (amd64)` action [#18709](https://github.com/apache/datafusion/pull/18709) (Jefffrey)
- chore: enforce clippy lint needless_pass_by_value to datafusion-proto [#18715](https://github.com/apache/datafusion/pull/18715) (foskey51)
- chore: enforce clippy lint needless_pass_by_value to datafusion-spark [#18714](https://github.com/apache/datafusion/pull/18714) (foskey51)
- minor: refactor with `assert_or_internal_err!()` in `datafusion/optimizer` [#18699](https://github.com/apache/datafusion/pull/18699) (kumarUjjawal)
- minor: refactor with `assert_or_internal_err!()` in `datafusion/functions` [#18700](https://github.com/apache/datafusion/pull/18700) (kumarUjjawal)
- minor: refactor with `assert_or_internal_err!()` in `datafusion/expr-common` [#18702](https://github.com/apache/datafusion/pull/18702) (kumarUjjawal)
- minor: refactor with `assert_or_internal_err!()` in `datafusion/functions-aggregate` [#18716](https://github.com/apache/datafusion/pull/18716) (kumarUjjawal)
- chore: enforce clippy lint needless_pass_by_value to datafusion-execution [#18723](https://github.com/apache/datafusion/pull/18723) (foskey51)
- minor: refactor with `assert_or_internal_err!()` in `datafusion/functions-nested` [#18724](https://github.com/apache/datafusion/pull/18724) (kumarUjjawal)
- chore: enforce clippy lint needless_pass_by_value to datafusion-substrait [#18703](https://github.com/apache/datafusion/pull/18703) (foskey51)
- chore: Refactor with assert_or_internal_err!() in datafusion/spark. [#18674](https://github.com/apache/datafusion/pull/18674) (codetyri0n)
- Minor: Add docs to release/README.md about rate limits [#18704](https://github.com/apache/datafusion/pull/18704) (alamb)
- Consolidate query planning examples (#18142) [#18690](https://github.com/apache/datafusion/pull/18690) (cj-zhukov)
- minor: refactor with `assert_or_internal_err!()` in `datafusion/physical-expr-common` [#18735](https://github.com/apache/datafusion/pull/18735) (kumarUjjawal)
- minor: refactor with `assert_or_internal_err!()` in `datafusion/physical-expr` [#18736](https://github.com/apache/datafusion/pull/18736) (kumarUjjawal)
- Consolidate ArrowFileSource and ArrowStreamFileSource [#18720](https://github.com/apache/datafusion/pull/18720) (adriangb)
- minor: refactor with `assert_or_internal_err!()` in `datafusion/physical-optimizer` [#18732](https://github.com/apache/datafusion/pull/18732) (kumarUjjawal)
- refactor: reduce duplication in make_udf_function macro [#18733](https://github.com/apache/datafusion/pull/18733) (shashidhar-bm)
- minor: refactor with `assert_or_internal_err!()` in `datafusion/physical-plan` [#18730](https://github.com/apache/datafusion/pull/18730) (kumarUjjawal)
- chore: enforce clippy lint needless_pass_by_value to datafusion-functions-aggregate-common [#18741](https://github.com/apache/datafusion/pull/18741) (foskey51)
- Optimize NullState::build [#18737](https://github.com/apache/datafusion/pull/18737) (Dandandan)
- chore: enforce clippy lint needless_pass_by_value to datafusion-datasource-parquet [#18695](https://github.com/apache/datafusion/pull/18695) (foskey51)
- minor: refactor with `assert_or_internal_err!()` in `datafusion/expr` [#18731](https://github.com/apache/datafusion/pull/18731) (kumarUjjawal)
- minor: Fix an example in the `PruningPredicate` documentation [#18742](https://github.com/apache/datafusion/pull/18742) (2010YOUY01)
- chore(deps): bump indicatif from 0.18.2 to 0.18.3 [#18756](https://github.com/apache/datafusion/pull/18756) (dependabot[bot])
- Fix map_query_sql benchmark duplicate key error [#18427](https://github.com/apache/datafusion/pull/18427) (atheendre130505)
- minor: enforce lint rule clippy::needless_pass_by_value to datafusion-ffi [#18764](https://github.com/apache/datafusion/pull/18764) (Standing-Man)
- Rename boolean `Interval` constants to match `NullableInterval` [#18654](https://github.com/apache/datafusion/pull/18654) (pepijnve)
- chore(deps): bump bytes from 1.10.1 to 1.11.0 [#18755](https://github.com/apache/datafusion/pull/18755) (dependabot[bot])
- CI: Fix `main` branch CI test failure [#18792](https://github.com/apache/datafusion/pull/18792) (2010YOUY01)
- chore: Enforce 'clippy::needless_pass_by_value' to datafusion-expr-common [#18775](https://github.com/apache/datafusion/pull/18775) (petern48)
- chore: Finish refactor with `assert_or_internal_err!()` [#18790](https://github.com/apache/datafusion/pull/18790) (2010YOUY01)
- Switch from xz2 to liblzma to reduce duplicate dependencies [#17509](https://github.com/apache/datafusion/pull/17509) (timsaucer)
- chore(deps): bump taiki-e/install-action from 2.62.51 to 2.62.53 [#18796](https://github.com/apache/datafusion/pull/18796) (dependabot[bot])
- chore(deps): bump actions/checkout from 5.0.0 to 5.0.1 [#18797](https://github.com/apache/datafusion/pull/18797) (dependabot[bot])
- Misc improvements to ProjectionExprs [#18719](https://github.com/apache/datafusion/pull/18719) (adriangb)
- Fix incorrect link for sql_query.rs example in README [#18807](https://github.com/apache/datafusion/pull/18807) (kondamudikarthik)
- Adds prefix filtering for table URLs [#18780](https://github.com/apache/datafusion/pull/18780) (BlakeOrth)
- Refactor InListExpr to support structs by re-using existing hashing infrastructure [#18449](https://github.com/apache/datafusion/pull/18449) (adriangb)
- chore: Add script to protect RC branches during the release [#18660](https://github.com/apache/datafusion/pull/18660) (comphead)
- Prevent overflow and panics when casting DATE to TIMESTAMP by validating bounds [#18761](https://github.com/apache/datafusion/pull/18761) (kosiew)
- chore(deps): bump taiki-e/install-action from 2.62.53 to 2.62.54 [#18815](https://github.com/apache/datafusion/pull/18815) (dependabot[bot])
- CI : Enforce clippy: :needless_pass_by_value rule to datafusion-functions-aggregate [#18805](https://github.com/apache/datafusion/pull/18805) (codetyri0n)
- Consolidate sql operations examples (#18142) [#18743](https://github.com/apache/datafusion/pull/18743) (cj-zhukov)
- Move `GuaranteeRewriter` to datafusion_expr [#18821](https://github.com/apache/datafusion/pull/18821) (pepijnve)
- Refactor state management in `HashJoinExec` and use CASE expressions for more precise filters [#18451](https://github.com/apache/datafusion/pull/18451) (adriangb)
- Refactor avg & sum signatures away from user defined [#18769](https://github.com/apache/datafusion/pull/18769) (Jefffrey)
- Hash UnionArrays [#18718](https://github.com/apache/datafusion/pull/18718) (friendlymatthew)
- CI: add clippy::needless_pass_by_value rule to datafusion-functions-window crate [#18838](https://github.com/apache/datafusion/pull/18838) (codetyri0n)
- Add field to DynamicPhysicalExpr to indicate when the filter is complete or updated [#18799](https://github.com/apache/datafusion/pull/18799) (LiaCastaneda)
- #17801 Improve nullability reporting of case expressions [#17813](https://github.com/apache/datafusion/pull/17813) (pepijnve)
- Consolidate execution monitoring examples (#18142) [#18846](https://github.com/apache/datafusion/pull/18846) (cj-zhukov)
- Implement CatalogProviderList in FFI [#18657](https://github.com/apache/datafusion/pull/18657) (timsaucer)
- Removed incorrect union check in enforce_sorting and updated tests [#18661](https://github.com/apache/datafusion/pull/18661) (gene-bordegaray)
- chore(deps): bump actions/checkout from 5.0.1 to 6.0.0 [#18865](https://github.com/apache/datafusion/pull/18865) (dependabot[bot])
- Remove unnecessary bit counting code from spark `bit_count` [#18841](https://github.com/apache/datafusion/pull/18841) (pepijnve)
- Fix async_udf batch size behaviour [#18819](https://github.com/apache/datafusion/pull/18819) (shivbhatia10)
- Fix Partial AggregateExec correctness issue dropping rows [#18712](https://github.com/apache/datafusion/pull/18712) (xanderbailey)
- chore: Add missing boolean tests to `bit_count` Spark function [#18871](https://github.com/apache/datafusion/pull/18871) (comphead)
- Consolidate proto examples (#18142) [#18861](https://github.com/apache/datafusion/pull/18861) (cj-zhukov)
- Use logical null count in `case_when_with_expr` [#18872](https://github.com/apache/datafusion/pull/18872) (pepijnve)
- chore: enforce `clippy::needless_pass_by_value` to `datafusion-physical-plan` [#18864](https://github.com/apache/datafusion/pull/18864) (2010YOUY01)
- Refactor spark `bit_get()` signature away from user defined [#18836](https://github.com/apache/datafusion/pull/18836) (Jefffrey)
- minor: enforce lint rule clippy::needless_pass_by_value to datafusion-functions [#18768](https://github.com/apache/datafusion/pull/18768) (Standing-Man)
- chore: enforce clippy lint needless_pass_by_value to datafusion-functions-nested [#18839](https://github.com/apache/datafusion/pull/18839) (foskey51)
- chore: fix CI on main [#18876](https://github.com/apache/datafusion/pull/18876) (Jefffrey)
- chore: update Repartition DisplayAs to indicate maintained sort order [#18673](https://github.com/apache/datafusion/pull/18673) (ruchirK)
- implement sum for durations [#18853](https://github.com/apache/datafusion/pull/18853) (logan-keede)
- Consolidate dataframe examples (#18142) [#18862](https://github.com/apache/datafusion/pull/18862) (cj-zhukov)
- Avoid the need to rewrite expressions when evaluating logical case nullability [#18849](https://github.com/apache/datafusion/pull/18849) (pepijnve)
- Avoid skew in Roundrobin repartition [#18880](https://github.com/apache/datafusion/pull/18880) (Dandandan)
- Add benchmark for array_has/array_has_all/array_has_any [#18729](https://github.com/apache/datafusion/pull/18729) (zhuqi-lucas)
- chore(deps): bump taiki-e/install-action from 2.62.54 to 2.62.56 [#18899](https://github.com/apache/datafusion/pull/18899) (dependabot[bot])
- chore(deps): bump indicatif from 0.18.0 to 0.18.3 [#18897](https://github.com/apache/datafusion/pull/18897) (dependabot[bot])
- chore(deps): bump tokio-util from 0.7.16 to 0.7.17 [#18898](https://github.com/apache/datafusion/pull/18898) (dependabot[bot])
- Support Non-Literal Expressions in Substrait VirtualTable Values and Improve Round-Trip Robustness [#18866](https://github.com/apache/datafusion/pull/18866) (kosiew)
- chore(deps): bump indexmap from 2.12.0 to 2.12.1 [#18895](https://github.com/apache/datafusion/pull/18895) (dependabot[bot])
- chore(deps): bump aws-config from 1.8.7 to 1.8.11 [#18896](https://github.com/apache/datafusion/pull/18896) (dependabot[bot])
- chore(deps): bump flate2 from 1.1.4 to 1.1.5 [#18900](https://github.com/apache/datafusion/pull/18900) (dependabot[bot])
- Add iter() method to `Extensions` [#18887](https://github.com/apache/datafusion/pull/18887) (gabotechs)
- chore: Enforce `clippy::needless_pass_by_value` globally across the workspace [#18904](https://github.com/apache/datafusion/pull/18904) (2010YOUY01)
- Consolidate external dependency examples (#18142) [#18747](https://github.com/apache/datafusion/pull/18747) (cj-zhukov)
- Optimize planning for projected nested union [#18713](https://github.com/apache/datafusion/pull/18713) (logan-keede)
- chore(deps): bump taiki-e/install-action from 2.62.56 to 2.62.57 [#18927](https://github.com/apache/datafusion/pull/18927) (dependabot[bot])
- chore(deps): bump actions/setup-python from 6.0.0 to 6.1.0 [#18925](https://github.com/apache/datafusion/pull/18925) (dependabot[bot])
- Fix `map` function alias handling in SQL planner [#18914](https://github.com/apache/datafusion/pull/18914) (friendlymatthew)
- minor: add builder setting `NdJsonReadOptions::schema_infer_max_records` [#18920](https://github.com/apache/datafusion/pull/18920) (Jefffrey)
- Implement Substrait Support for `GROUPING SET CUBE` [#18798](https://github.com/apache/datafusion/pull/18798) (kosiew)
- chore: unify common dependencies as workspace dependencies [#18665](https://github.com/apache/datafusion/pull/18665) (Jefffrey)
- Fix bug where binary types were incorrectly being casted for coercible signatures [#18750](https://github.com/apache/datafusion/pull/18750) (Jefffrey)
- Refactor approx_median signature & support f16 [#18647](https://github.com/apache/datafusion/pull/18647) (Jefffrey)
- Refactor `to_local_time()` signature away from user_defined [#18707](https://github.com/apache/datafusion/pull/18707) (Jefffrey)
- chore(deps-dev): bump node-forge from 1.3.1 to 1.3.2 in /datafusion/wasmtest/datafusion-wasm-app [#18958](https://github.com/apache/datafusion/pull/18958) (dependabot[bot])
- Support LikeMatch, ILikeMatch, NotLikeMatch, NotILikeMatch operators in protobuf serialization [#18961](https://github.com/apache/datafusion/pull/18961) (zhuqi-lucas)
- chore: cargo fmt to fix CI [#18969](https://github.com/apache/datafusion/pull/18969) (Jefffrey)
- chore(deps): bump Swatinem/rust-cache from 2.8.1 to 2.8.2 [#18963](https://github.com/apache/datafusion/pull/18963) (dependabot[bot])
- chore(deps): bump taiki-e/install-action from 2.62.57 to 2.62.58 [#18964](https://github.com/apache/datafusion/pull/18964) (dependabot[bot])
- chore(deps): bump crate-ci/typos from 1.39.2 to 1.40.0 [#18965](https://github.com/apache/datafusion/pull/18965) (dependabot[bot])
- [Minor] Refactor `traverse_chain` macro to function [#18951](https://github.com/apache/datafusion/pull/18951) (Dandandan)
- Enable clippy::allow_attributes lint for datafusion-catalog [#18973](https://github.com/apache/datafusion/pull/18973) (chakkk309)
- chore: update group of crates to rust 2024 edition [#18915](https://github.com/apache/datafusion/pull/18915) (timsaucer)
- chore(deps): bump taiki-e/install-action from 2.62.58 to 2.62.59 [#18978](https://github.com/apache/datafusion/pull/18978) (dependabot[bot])
- Simplify percentile_cont for 0/1 percentiles [#18837](https://github.com/apache/datafusion/pull/18837) (kumarUjjawal)
- chore: enforce clippy::allow_attributes for functions-\* crates [#18986](https://github.com/apache/datafusion/pull/18986) (carlosahs)
- chore: enforce clippy::allow_attributes for common crates [#18988](https://github.com/apache/datafusion/pull/18988) (chakkk309)
- Fix predicate_rows_pruned & predicate_rows_matched metrics [#18980](https://github.com/apache/datafusion/pull/18980) (xudong963)
- Allocate a buffer of the correct length for ScalarValue::FixedSizeBinary in ScalarValue::to_array_of_size [#18903](https://github.com/apache/datafusion/pull/18903) (tobixdev)
- Fix error planning aggregates with duplicated names in select list [#18831](https://github.com/apache/datafusion/pull/18831) (tshauck)
- chore: remove `deny`s of `needless_pass_by_value` in `lib.rs` files [#18996](https://github.com/apache/datafusion/pull/18996) (Jefffrey)
- Add Explicit Error Handling for Unsupported SQL `FETCH` Clause in Planner and CLI [#18691](https://github.com/apache/datafusion/pull/18691) (kosiew)
- chore(deps): bump criterion from 0.7.0 to 0.8.0 [#19009](https://github.com/apache/datafusion/pull/19009) (dependabot[bot])
- chore(deps): bump syn from 2.0.108 to 2.0.111 [#19011](https://github.com/apache/datafusion/pull/19011) (dependabot[bot])
- chore(deps): bump taiki-e/install-action from 2.62.59 to 2.62.60 [#19012](https://github.com/apache/datafusion/pull/19012) (dependabot[bot])
- chore: remove redundant clone code [#18997](https://github.com/apache/datafusion/pull/18997) (Smith-Cruise)
- Update to `arrow`, `parquet` to `57.1.0` [#18820](https://github.com/apache/datafusion/pull/18820) (alamb)
- deny on allow_attributes lint in physical-plan [#18983](https://github.com/apache/datafusion/pull/18983) (YuraLitvinov)
- Add additional test coverage of multi-value PartitionPruningStats [#19021](https://github.com/apache/datafusion/pull/19021) (alamb)
- Fix tpch benchmark harness [#19033](https://github.com/apache/datafusion/pull/19033) (alamb)
- Fix data for tpch_csv and tpch_csv10 [#19034](https://github.com/apache/datafusion/pull/19034) (alamb)
- chore: update group of 3 crates to rust 2024 edition [#19001](https://github.com/apache/datafusion/pull/19001) (timsaucer)
- chore(deps-dev): bump express from 4.21.2 to 4.22.1 in /datafusion/wasmtest/datafusion-wasm-app [#19040](https://github.com/apache/datafusion/pull/19040) (dependabot[bot])
- Allow repartitioning on files with ranges [#18948](https://github.com/apache/datafusion/pull/18948) (Samyak2)
- Support simplify not for physical expr [#18970](https://github.com/apache/datafusion/pull/18970) (xudong963)
- dev: Add typos check to the local `dev/rust_lint.sh` [#17863](https://github.com/apache/datafusion/pull/17863) (2010YOUY01)
- Implement FFI_PhysicalExpr and the structs it needs to support it. [#18916](https://github.com/apache/datafusion/pull/18916) (timsaucer)
- chore(deps): bump actions/setup-node from 6.0.0 to 6.1.0 [#19063](https://github.com/apache/datafusion/pull/19063) (dependabot[bot])
- chore(deps): bump taiki-e/install-action from 2.62.60 to 2.62.61 [#19062](https://github.com/apache/datafusion/pull/19062) (dependabot[bot])
- chore(deps): bump actions/stale from 10.1.0 to 10.1.1 [#19061](https://github.com/apache/datafusion/pull/19061) (dependabot[bot])
- chore: merge make_array and spark array [#19006](https://github.com/apache/datafusion/pull/19006) (jizezhang)
- chore(deps): bump actions/checkout from 6.0.0 to 6.0.1 [#19060](https://github.com/apache/datafusion/pull/19060) (dependabot[bot])
- Add documentation example for `PartitionPruningStatistics` [#19020](https://github.com/apache/datafusion/pull/19020) (alamb)
- chore: upgrade expr and execution crates to rust 2024 edition [#19047](https://github.com/apache/datafusion/pull/19047) (timsaucer)
- refactor: Refactor spark make_interval signature away from user defined [#19027](https://github.com/apache/datafusion/pull/19027) (kumarUjjawal)
- Fix: Align sort_merge_join filter output with join schema to fix right-anti panic [#18800](https://github.com/apache/datafusion/pull/18800) (kumarUjjawal)
- Support Substrait Round-Trip of `EmptyRelation` Including `produce_one_row` Semantics [#18842](https://github.com/apache/datafusion/pull/18842) (kosiew)
- chore(deps): bump taiki-e/install-action from 2.62.61 to 2.62.62 [#19081](https://github.com/apache/datafusion/pull/19081) (dependabot[bot])
- chore: enforce clippy::allow_attributes for datasource crates [#19068](https://github.com/apache/datafusion/pull/19068) (chakkk309)
- common: Add hashing support for REE arrays [#18981](https://github.com/apache/datafusion/pull/18981) (brancz)
- Use `tpchgen-cli` to generate tpch data in bench.sh [#19035](https://github.com/apache/datafusion/pull/19035) (alamb)
- Update aggregate probe to be locked only if skipping aggregation [#18766](https://github.com/apache/datafusion/pull/18766) (hareshkh)
- Fix function doc CI check [#19093](https://github.com/apache/datafusion/pull/19093) (alamb)
- Fix Schema Duplication Errors in Self‑Referential INTERSECT/EXCEPT by Requalifying Input Sides [#18814](https://github.com/apache/datafusion/pull/18814) (kosiew)
- run cargo fmt to fix after #18998 [#19102](https://github.com/apache/datafusion/pull/19102) (adriangb)
- bench: set test_util as required feature for aggregate_vectorized [#19101](https://github.com/apache/datafusion/pull/19101) (rluvaton)
- use ProjectionExprs:project_statistics in FileScanConfig [#19094](https://github.com/apache/datafusion/pull/19094) (adriangb)
- Temporarily ignore test_cache_with_ttl_and_lru test [#19115](https://github.com/apache/datafusion/pull/19115) (alamb)
- refactor: move human readable display utilities to `datafusion-common` crate [#19080](https://github.com/apache/datafusion/pull/19080) (2010YOUY01)
- Always remove unecessary software from github runners for all jobs (fix intermittent out of space on runners) [#19122](https://github.com/apache/datafusion/pull/19122) (alamb)
- [datafusion-spark]: Refactor make_dt_interval's signature away from user defined [#19083](https://github.com/apache/datafusion/pull/19083) (codetyri0n)
- fix deprecation notes with incorrect versions from #13083 [#19135](https://github.com/apache/datafusion/pull/19135) (adriangb)
- Run the examples in the new format [#18946](https://github.com/apache/datafusion/pull/18946) (cj-zhukov)
- Add constant expression evaluator to physical expression simplifier [#19130](https://github.com/apache/datafusion/pull/19130) (adriangb)
- Fix shuffle function to report nullability correctly [#19184](https://github.com/apache/datafusion/pull/19184) (harshitsaini17)
- chore: enforce clippy::allow_attributes for physical crates [#19185](https://github.com/apache/datafusion/pull/19185) (carlosahs)
- Update 5 crates to rust 2024 edition [#19091](https://github.com/apache/datafusion/pull/19091) (timsaucer)
- Coalesce batches inside hash join, reuse indices buffer [#18972](https://github.com/apache/datafusion/pull/18972) (Dandandan)
- slt test coverage for `CASE` exprs with constant value lookup tables [#19143](https://github.com/apache/datafusion/pull/19143) (alamb)
- Fix fmt after logical conflict [#19208](https://github.com/apache/datafusion/pull/19208) (alamb)
- chore: Add TPCDS benchmarks [#19138](https://github.com/apache/datafusion/pull/19138) (comphead)
- Arc partition values in TableSchema [#19137](https://github.com/apache/datafusion/pull/19137) (adriangb)
- Add sorted data benchmark. [#19042](https://github.com/apache/datafusion/pull/19042) (zhuqi-lucas)
- Refactor PhysicalExprSimplfier to &self instead of &mut self [#19212](https://github.com/apache/datafusion/pull/19212) (adriangb)
- chore(deps): bump uuid from 1.18.1 to 1.19.0 [#19199](https://github.com/apache/datafusion/pull/19199) (dependabot[bot])
- chore(deps): bump async-compression from 0.4.34 to 0.4.35 [#19201](https://github.com/apache/datafusion/pull/19201) (dependabot[bot])
- chore(deps): bump taiki-e/install-action from 2.62.62 to 2.62.63 [#19198](https://github.com/apache/datafusion/pull/19198) (dependabot[bot])
- chore(deps): bump tracing-subscriber from 0.3.20 to 0.3.22 [#19200](https://github.com/apache/datafusion/pull/19200) (dependabot[bot])
- chore(deps): bump wasm-bindgen-test from 0.3.55 to 0.3.56 [#19202](https://github.com/apache/datafusion/pull/19202) (dependabot[bot])
- bench: add dedicated Utf8View benchmarks for InList [#19211](https://github.com/apache/datafusion/pull/19211) (geoffreyclaude)
- Fix PruningPredicate interaction with DynamicFilterPhysicalExpr that references partition columns [#19129](https://github.com/apache/datafusion/pull/19129) (adriangb)
- Implement physical and logical codecs in FFI [#19079](https://github.com/apache/datafusion/pull/19079) (timsaucer)
- refactor: Refactor spark width bucket signature away from user defined [#19065](https://github.com/apache/datafusion/pull/19065) (kumarUjjawal)
- Sort Merge Join: Reduce batch concatenation, use `BatchCoalescer`, new benchmarks (TPC-H Q21 SMJ up to ~4000x faster) [#18875](https://github.com/apache/datafusion/pull/18875) (mbutrovich)
- Add relation planner extension support to customize SQL planning [#17843](https://github.com/apache/datafusion/pull/17843) (geoffreyclaude)
- Add additional tests for InListExpr [#19050](https://github.com/apache/datafusion/pull/19050) (adriangb)
- chore(deps): bump taiki-e/install-action from 2.62.63 to 2.62.64 [#19226](https://github.com/apache/datafusion/pull/19226) (dependabot[bot])
- Use strum in the examples (#19126) [#19205](https://github.com/apache/datafusion/pull/19205) (cj-zhukov)
- [Proto]: Serialization support for `AsyncFuncExec` [#19118](https://github.com/apache/datafusion/pull/19118) (mach-kernel)
- chore: add test case for decimal overflow [#19255](https://github.com/apache/datafusion/pull/19255) (Jefffrey)
- chore(deps): bump taiki-e/install-action from 2.62.64 to 2.62.65 [#19251](https://github.com/apache/datafusion/pull/19251) (dependabot[bot])
- chore: update 6 crates to rust edition 2024 [#19196](https://github.com/apache/datafusion/pull/19196) (timsaucer)
- Implement FFI_Session [#19223](https://github.com/apache/datafusion/pull/19223) (timsaucer)
- Feat: Add an option for fast tests by gating slow tests to extended_tests feature [#19237](https://github.com/apache/datafusion/pull/19237) (Yuvraj-cyborg)
- chore: enforce clippy::allow_attributes for 7 crates [#19133](https://github.com/apache/datafusion/pull/19133) (chakkk309)
- dev: Add CI doc prettier check to local `rust_lint.sh` [#19254](https://github.com/apache/datafusion/pull/19254) (2010YOUY01)
- bug: Eliminate dead round-robin insertion in enforce distribution [#19132](https://github.com/apache/datafusion/pull/19132) (gene-bordegaray)
- Automatically download tpcds benchmark data to the right place [#19244](https://github.com/apache/datafusion/pull/19244) (alamb)
- [datafusion-spark]: Refactor hex's signature away from user_defined [#19235](https://github.com/apache/datafusion/pull/19235) (codetyri0n)
- fix : correct nullability propagation for spark.bitwise_not [#19224](https://github.com/apache/datafusion/pull/19224) (shifluxxc)
- added custom nullability for char [#19268](https://github.com/apache/datafusion/pull/19268) (skushagra)
- replace HashTableLookupExpr with lit(true) in proto serialization [#19300](https://github.com/apache/datafusion/pull/19300) (adriangb)
- chore: fix return_field_from_args doc [#19307](https://github.com/apache/datafusion/pull/19307) (xumingming)
- chore: enforce clippy::allow_attributes for spark,sql,sustrait [#19309](https://github.com/apache/datafusion/pull/19309) (kumarUjjawal)
- Simplify make_date & fix null handling [#19296](https://github.com/apache/datafusion/pull/19296) (Jefffrey)
- Allow base64 encoding of fixedsizebinary arrays [#18950](https://github.com/apache/datafusion/pull/18950) (maxburke)
- chore: update 11 crates to Rust 2024 edition [#19258](https://github.com/apache/datafusion/pull/19258) (timsaucer)
- Minor: remove unnecessary unit tests for fixed size binary [#19318](https://github.com/apache/datafusion/pull/19318) (alamb)
- Populate partition column statistics for PartitionedFile [#19284](https://github.com/apache/datafusion/pull/19284) (adriangb)
- refactor: move metrics module to `datafusion-common` crate [#19247](https://github.com/apache/datafusion/pull/19247) (2010YOUY01)
- chore(deps): bump taiki-e/install-action from 2.62.65 to 2.62.67 [#19295](https://github.com/apache/datafusion/pull/19295) (dependabot[bot])
- chore(deps): bump ctor from 0.6.1 to 0.6.3 [#19328](https://github.com/apache/datafusion/pull/19328) (dependabot[bot])
- Refactor `power()` signature away from user defined [#18968](https://github.com/apache/datafusion/pull/18968) (Jefffrey)
- chore: enforce `clippy::allow_attributes` for optimizer and macros [#19310](https://github.com/apache/datafusion/pull/19310) (kumarUjjawal)
- chore(deps): bump taiki-e/install-action from 2.62.67 to 2.63.3 [#19349](https://github.com/apache/datafusion/pull/19349) (dependabot[bot])
- chore(deps): bump clap from 4.5.50 to 4.5.53 [#19326](https://github.com/apache/datafusion/pull/19326) (dependabot[bot])
- chore(deps): bump insta from 1.43.2 to 1.44.3 [#19327](https://github.com/apache/datafusion/pull/19327) (dependabot[bot])
- remove repartition exec from coalesce batches optimizer [#19239](https://github.com/apache/datafusion/pull/19239) (jizezhang)
- minor: cleanup unnecessary config in `decimal.slt` [#19352](https://github.com/apache/datafusion/pull/19352) (Jefffrey)
- Fix panic for `GROUPING SETS(())` and handle empty-grouping aggregates [#19252](https://github.com/apache/datafusion/pull/19252) (kosiew)
- Update datafusion-core crate to Rust 2024 edition [#19332](https://github.com/apache/datafusion/pull/19332) (timsaucer)
- Update 4 crates to rust 2024 edition [#19357](https://github.com/apache/datafusion/pull/19357) (timsaucer)
- preserve Field metadata in first_value/last_value [#19335](https://github.com/apache/datafusion/pull/19335) (adriangb)
- Fix flaky SpillPool channel test by synchronizing reader and writer tasks [#19110](https://github.com/apache/datafusion/pull/19110) (kosiew)
- [minor] Upgrade rust version [#19363](https://github.com/apache/datafusion/pull/19363) (Dandandan)
- Minor: fix cargo fmt [#19368](https://github.com/apache/datafusion/pull/19368) (zhuqi-lucas)
- chore: enforce clippy::allow_attributes for proto, pruning, session [#19350](https://github.com/apache/datafusion/pull/19350) (kumarUjjawal)
- Update remaining crates to rust 2024 edition [#19361](https://github.com/apache/datafusion/pull/19361) (timsaucer)
- Minor: Make `ProjectionExpr::new` easier to use with constants [#19343](https://github.com/apache/datafusion/pull/19343) (alamb)
- Feat: DefaultListFilesCache prefix-aware for partition pruning optimization [#19298](https://github.com/apache/datafusion/pull/19298) (Yuvraj-cyborg)
- Extend in_list benchmark coverage [#19376](https://github.com/apache/datafusion/pull/19376) (geoffreyclaude)
- [datafusion-cli] Implement average LIST duration for object store profiling [#19127](https://github.com/apache/datafusion/pull/19127) (peterxcli)
- chore(deps): bump taiki-e/install-action from 2.63.3 to 2.64.0 [#19382](https://github.com/apache/datafusion/pull/19382) (dependabot[bot])
- update insta snapshots [#19381](https://github.com/apache/datafusion/pull/19381) (kosiew)
- Fix regression for negative-scale decimal128 in log [#19315](https://github.com/apache/datafusion/pull/19315) (shifluxxc)
- Fix input handling for encoding functions & various refactors [#18754](https://github.com/apache/datafusion/pull/18754) (Jefffrey)
- Fix ORDER BY positional reference regression with aliased aggregates [#19412](https://github.com/apache/datafusion/pull/19412) (adriangb)
- Implement disk spilling for all grouping ordering modes in GroupedHashAggregateStream [#19287](https://github.com/apache/datafusion/pull/19287) (pepijnve)
- refactor: add ParquetOpenerBuilder to reduce test code duplication [#19405](https://github.com/apache/datafusion/pull/19405) (shashidhar-bm)
- bench: add `range_and_generate_series` [#19428](https://github.com/apache/datafusion/pull/19428) (rluvaton)
- chore: use extend instead of manual loop in multi group by [#19429](https://github.com/apache/datafusion/pull/19429) (rluvaton)
- chore(deps): bump taiki-e/install-action from 2.64.0 to 2.64.2 [#19399](https://github.com/apache/datafusion/pull/19399) (dependabot[bot])
- Add recursive protection on planner's `create_physical_expr` [#19299](https://github.com/apache/datafusion/pull/19299) (rgehan)
- chore(deps): bump aws-config from 1.8.11 to 1.8.12 [#19453](https://github.com/apache/datafusion/pull/19453) (dependabot[bot])
- chore(deps): bump log from 0.4.28 to 0.4.29 [#19452](https://github.com/apache/datafusion/pull/19452) (dependabot[bot])
- chore(deps): bump taiki-e/install-action from 2.64.2 to 2.65.1 [#19451](https://github.com/apache/datafusion/pull/19451) (dependabot[bot])
- chore(deps): bump insta from 1.44.3 to 1.45.0 [#19454](https://github.com/apache/datafusion/pull/19454) (dependabot[bot])
- added support for negative scale for log decimal32/64 and power [#19409](https://github.com/apache/datafusion/pull/19409) (shifluxxc)
- Remove core dependency from ffi [#19422](https://github.com/apache/datafusion/pull/19422) (timsaucer)
- bench: increase in_list benchmark coverage [#19443](https://github.com/apache/datafusion/pull/19443) (geoffreyclaude)
- Use SortMergeJoinExec name consistently in physical plan outputs [#19246](https://github.com/apache/datafusion/pull/19246) (xavlee)
- Fix panic during spill to disk in clickbench query [#19421](https://github.com/apache/datafusion/pull/19421) (alamb)
- Optimize memory footprint of view arrays from `ScalarValue::to_array_of_size` [#19441](https://github.com/apache/datafusion/pull/19441) (Jefffrey)
- minor: refactoring of some `ScalarValue` code [#19439](https://github.com/apache/datafusion/pull/19439) (Jefffrey)
- Refactor Spark crc32 & sha1 to remove unnecessary scalar argument check [#19466](https://github.com/apache/datafusion/pull/19466) (Jefffrey)
- Add link to arrow-rs ticket in comments [#19479](https://github.com/apache/datafusion/pull/19479) (alamb)
- chore(deps): bump taiki-e/install-action from 2.65.1 to 2.65.2 [#19474](https://github.com/apache/datafusion/pull/19474) (dependabot[bot])
- Improve plan_to_sql handling of empty projections with dialect-specific SELECT list support [#19221](https://github.com/apache/datafusion/pull/19221) (kosiew)
- examples: replace sql_dialect with custom_sql_parser example [#19383](https://github.com/apache/datafusion/pull/19383) (geoffreyclaude)
- Replace custom merge operator with arrow-rs implementation [#19424](https://github.com/apache/datafusion/pull/19424) (pepijnve)
- Implement nested recursive CTEs [#18956](https://github.com/apache/datafusion/pull/18956) (Tpt)
- Add: PI upper/lower bound f16 constants to ScalarValue [#19497](https://github.com/apache/datafusion/pull/19497) (xonx4l)
- chore: enforce clippy::allow_attributes for datafusion-ffi crate [#19480](https://github.com/apache/datafusion/pull/19480) (chakkk309)
- Add CI check to ensure examples are documented in README [#19371](https://github.com/apache/datafusion/pull/19371) (cj-zhukov)
- fix : snapshot to the modern multiline format [#19517](https://github.com/apache/datafusion/pull/19517) (Nachiket-Roy)
- chore(deps): bump taiki-e/install-action from 2.65.2 to 2.65.3 [#19499](https://github.com/apache/datafusion/pull/19499) (dependabot[bot])
- docs : clarify unused test utility [#19508](https://github.com/apache/datafusion/pull/19508) (Nachiket-Roy)
- Date / time / interval arithmetic improvements [#19460](https://github.com/apache/datafusion/pull/19460) (Omega359)
- Preserve ORDER BY in Unparser for projection -> order by pattern [#19483](https://github.com/apache/datafusion/pull/19483) (adriangb)
- Redesign the try_reverse_output to support more cases [#19446](https://github.com/apache/datafusion/pull/19446) (zhuqi-lucas)
- refactor: Spark `ascii` signature away from `user_defined` [#19513](https://github.com/apache/datafusion/pull/19513) (kumarUjjawal)
- Fix: SparkAscii nullability to depend on input nullability [#19531](https://github.com/apache/datafusion/pull/19531) (Yuvraj-cyborg)
- chore(deps): bump tracing from 0.1.41 to 0.1.43 [#19543](https://github.com/apache/datafusion/pull/19543) (dependabot[bot])
- chore(deps): bump substrait from 0.62.0 to 0.62.2 [#19542](https://github.com/apache/datafusion/pull/19542) (dependabot[bot])
- chore(deps): bump taiki-e/install-action from 2.65.3 to 2.65.6 [#19541](https://github.com/apache/datafusion/pull/19541) (dependabot[bot])
- minor: run all examples by default [#19506](https://github.com/apache/datafusion/pull/19506) (theirix)
- Refactor TopKHashTable to use HashTable API [#19464](https://github.com/apache/datafusion/pull/19464) (Dandandan)
- Revert Spark Elt nullability change [#19510](https://github.com/apache/datafusion/pull/19510) (Jefffrey)
- minor: implement more arms for `get_data_types()` for `NativeType` [#19449](https://github.com/apache/datafusion/pull/19449) (Jefffrey)
- Upgrade hashbrown to 0.16 [#19554](https://github.com/apache/datafusion/pull/19554) (Dandandan)
- minor : add crypto function benchmark [#19539](https://github.com/apache/datafusion/pull/19539) (getChan)
- chore(deps): bump taiki-e/install-action from 2.65.6 to 2.65.8 [#19559](https://github.com/apache/datafusion/pull/19559) (dependabot[bot])
- bugfix: preserve schema metadata for record batch in FFI [#19293](https://github.com/apache/datafusion/pull/19293) (timsaucer)
- refactor: extract the data generate out of aggregate_topk benchmark [#19523](https://github.com/apache/datafusion/pull/19523) (haohuaijin)
- Compute Dynamic Filters only when a consumer supports them [#19546](https://github.com/apache/datafusion/pull/19546) (LiaCastaneda)
- Various refactors to string functions [#19402](https://github.com/apache/datafusion/pull/19402) (Jefffrey)
- Implement `partition_statistics` API for `NestedLoopJoinExec` [#19468](https://github.com/apache/datafusion/pull/19468) (kumarUjjawal)
- Replace deprecated structopt with clap in datafusion-benchmarks [#19492](https://github.com/apache/datafusion/pull/19492) (Yuvraj-cyborg)
- Refactor duplicate code in `type_coercion/functions.rs` [#19518](https://github.com/apache/datafusion/pull/19518) (Jefffrey)
- chore(deps): bump taiki-e/install-action from 2.65.8 to 2.65.10 [#19578](https://github.com/apache/datafusion/pull/19578) (dependabot[bot])
- perf: Improve performance of hex encoding in spark functions [#19586](https://github.com/apache/datafusion/pull/19586) (shashidhar-bm)
- Add left function benchmark [#19600](https://github.com/apache/datafusion/pull/19600) (viirya)
- chore: Add TPCDS benchmark comparison for PR [#19552](https://github.com/apache/datafusion/pull/19552) (comphead)
- chore(deps): bump taiki-e/install-action from 2.65.10 to 2.65.11 [#19601](https://github.com/apache/datafusion/pull/19601) (dependabot[bot])
- chore: bump testcontainers-modules to 0.14 and remove testcontainers dep [#19620](https://github.com/apache/datafusion/pull/19620) (Jefffrey)
- Validate parquet writer version [#19515](https://github.com/apache/datafusion/pull/19515) (AlyAbdelmoneim)
- chore(deps): bump insta from 1.45.0 to 1.46.0 [#19643](https://github.com/apache/datafusion/pull/19643) (dependabot[bot])
- chore(deps): bump taiki-e/install-action from 2.65.11 to 2.65.13 [#19646](https://github.com/apache/datafusion/pull/19646) (dependabot[bot])
- chore(deps): bump tracing from 0.1.43 to 0.1.44 [#19644](https://github.com/apache/datafusion/pull/19644) (dependabot[bot])
- chore(deps): bump syn from 2.0.111 to 2.0.113 [#19645](https://github.com/apache/datafusion/pull/19645) (dependabot[bot])
- Refactor `percentile_cont` to clarify support input types [#19611](https://github.com/apache/datafusion/pull/19611) (Jefffrey)
- Add a protection to release candidate branch 52 [#19660](https://github.com/apache/datafusion/pull/19660) (xudong963)
- Downgrade aws-smithy-runtime, update `rust_decimal`, ignore RUSTSEC-2026-0001 to get clean CI [#19657](https://github.com/apache/datafusion/pull/19657) (alamb)
- Update dependencies [#19667](https://github.com/apache/datafusion/pull/19667) (alamb)
- Refactor PartitionedFile: add ordering field and new_from_meta constructor [#19596](https://github.com/apache/datafusion/pull/19596) (adriangb)
- Remove coalesce batches rule and deprecate CoalesceBatchesExec [#19622](https://github.com/apache/datafusion/pull/19622) (feniljain)
- Perf: Optimize `substring_index` via single-byte fast path and direct indexing [#19590](https://github.com/apache/datafusion/pull/19590) (lyne7-sc)
- refactor: Use `Signature::coercible` for isnan/iszero [#19604](https://github.com/apache/datafusion/pull/19604) (kumarUjjawal)
- Parquet: Push down supported list predicates (array_has/any/all) during decoding [#19545](https://github.com/apache/datafusion/pull/19545) (kosiew)
- Remove dependency on `rust_decimal`, remove ignore of `RUSTSEC-2026-0001` [#19666](https://github.com/apache/datafusion/pull/19666) (alamb)
- Store example data directly inside the datafusion-examples (#19141) [#19319](https://github.com/apache/datafusion/pull/19319) (cj-zhukov)
- minor: More comments to `ParquetOpener::open()` [#19677](https://github.com/apache/datafusion/pull/19677) (2010YOUY01)
- Feat: Allow pow with negative & non-integer exponent on decimals [#19369](https://github.com/apache/datafusion/pull/19369) (Yuvraj-cyborg)
- chore(deps): bump taiki-e/install-action from 2.65.13 to 2.65.15 [#19676](https://github.com/apache/datafusion/pull/19676) (dependabot[bot])
- Refactor cache APIs to support ordering information [#19597](https://github.com/apache/datafusion/pull/19597) (adriangb)
- Record sort order when writing Parquet with WITH ORDER [#19595](https://github.com/apache/datafusion/pull/19595) (adriangb)
- implement var distinct [#19706](https://github.com/apache/datafusion/pull/19706) (thinh2)

## Credits

Thank you to everyone who contributed to this release. Here is a breakdown of commits (PRs merged) per contributor.

```
    67	dependabot[bot]
    38	Andrew Lamb
    36	Jeffrey Vo
    35	Kumar Ujjawal
    34	Adrian Garcia Badaracco
    22	Tim Saucer
    19	Yongting You
    13	Sergey Zhukov
    11	Pepijn Van Eeckhoudt
    11	kosiew
    10	Daniël Heres
    10	Dhanush
    10	Oleks V
     8	Geoffrey Claude
     8	Raz Luvaton
     7	Andy Grove
     7	Liang-Chi Hsieh
     7	Qi Zhu
     6	Peter Nguyen
     6	Shashidhar B M
     5	Alan Tang
     5	Alex Huang
     5	Bruce Ritchie
     5	Gene Bordegaray
     5	Nuno Faria
     5	Sriram Sundar
     4	Blake Orth
     4	Thomas Tanon
     4	Yuvraj
     4	theirix
     3	Aryan Bagade
     3	Chakkk
     3	Emily Matheys
     3	Huaijin
     3	Khanh Duong
     3	Kushagra S
     3	Vedic Chawla
     3	feniljain
     3	harshit saini
     3	jizezhang
     3	shifluxxc
     3	xonx
     3	xudong.w
     2	Carlos Hurtado
     2	Chen Chongchen
     2	Cora Sutton
     2	Haresh Khanna
     2	Lía Adriana
     2	Manish Kumar
     2	Martin Grigorov
     2	Matthew Kim
     2	Namgung Chan
     2	Nimalan
     2	Nithurshen
     2	Rosai
     2	Shubham Yadav
     2	Trent Hauck
     2	Vegard Stikbakke
     2	Vrishabh
     2	Xander
     2	chakkk309
     2	mag1c1an1
     2	nlimpid
     2	yqrz
     1	Adam Curtis
     1	Aly Abdelmoneim
     1	Andrey Velichkevich
     1	Arpit Bandejiya
     1	Bharathwaj G
     1	Bipul Lamsal
     1	Clement de Groc
     1	Congxian Qiu
     1	David López
     1	David Stancu
     1	Devanshu
     1	Dongpo Liu
     1	EeshanBembi
     1	Eshaan Gupta
     1	Ethan Urbanski
     1	Frederic Branczyk
     1	Gabriel
     1	Gohlub
     1	Heran Lin
     1	James Xu
     1	Jatin Kumar singh
     1	Karan Pradhan
     1	Karthik Kondamudi
     1	Kazantsev Maksim
     1	Marco Neumann
     1	Matt Butrovich
     1	Max Burke
     1	Michele Vigilante
     1	Mikhail Zabaluev
     1	Mohit rao
     1	Ning Sun
     1	Peter Lee
     1	Quoc Anh
     1	Ram
     1	Randy
     1	Renan GEHAN
     1	Ruchir Khaitan
     1	Samyak Sarnayak
     1	Shiv Bhatia
     1	Smith Cruise
     1	Smotrov Oleksii
     1	Solari Systems
     1	Suhail
     1	T2MIX
     1	Tal Glanzman
     1	Tamar
     1	Tim-53
     1	Tobias Schwarzinger
     1	Ujjwal Kumar Tiwari
     1	Willem Verstraeten
     1	YuraLitvinov
     1	bubulalabu
     1	delamarch3
     1	hsiang-c
     1	r1b
     1	rin
     1	xavlee
```

Thank you also to everyone who contributed in other ways such as filing issues, reviewing PRs, and providing feedback on this release.
