<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements.  See the NOTICE file
distributed with this work for additional information
regarding copyright ownership.  The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License.  You may obtain a copy of the License at

  http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied.  See the License for the
specific language governing permissions and limitations
under the License.
-->

# DataFusion Comet 0.16.0 Changelog

This release consists of 127 commits from 17 contributors. See credits at the end of this changelog for more information.

**Fixed bugs:**

- fix: report task output metrics in Spark UI [#3999](https://github.com/apache/datafusion-comet/pull/3999) (0lai0)
- fix: cast to and from timestamp_ntz [#4008](https://github.com/apache/datafusion-comet/pull/4008) (parthchandra)
- fix: support to_json on Spark 4.0 [#4036](https://github.com/apache/datafusion-comet/pull/4036) (andygrove)
- fix: enable arrays_overlap [#3901](https://github.com/apache/datafusion-comet/pull/3901) (kazuyukitanimura)
- fix: Iceberg reflection for current() on TableOperations hierarchy [#3895](https://github.com/apache/datafusion-comet/pull/3895) (karuppayya)
- fix: fall back to Spark for shuffle/sort/aggregate on non-default collated strings [Spark 4] [#4035](https://github.com/apache/datafusion-comet/pull/4035) (andygrove)
- fix: scalar subquery pushdown and reuse for CometNativeScanExec (SPARK-43402) [#4053](https://github.com/apache/datafusion-comet/pull/4053) (mbutrovich)
- fix: fall back for shredded Variant scans on Spark 4.0 [#4084](https://github.com/apache/datafusion-comet/pull/4084) (andygrove)
- fix: enable Spark 4 SQL tests previously ignored for issues #3313 and #3314 [#4092](https://github.com/apache/datafusion-comet/pull/4092) (andygrove)
- fix: fall back to Spark for hash join and sort-merge join on non-default collated string keys [Spark 4] [#4095](https://github.com/apache/datafusion-comet/pull/4095) (0lai0)
- fix: reject string/binary read as numeric in native_datafusion scan [#4091](https://github.com/apache/datafusion-comet/pull/4091) (andygrove)
- fix: reject incompatible decimal precision/scale in native_datafusion scan [#4090](https://github.com/apache/datafusion-comet/pull/4090) (andygrove)
- fix: throw SchemaColumnConvertNotSupportedException from native_datafusion schema mismatch [#4117](https://github.com/apache/datafusion-comet/pull/4117) (andygrove)
- fix: substring with negative start index [#4017](https://github.com/apache/datafusion-comet/pull/4017) (kazuyukitanimura)
- fix: honor strictFloatingPoint in RangePartitioning [#4167](https://github.com/apache/datafusion-comet/pull/4167) (0lai0)
- fix: [Spark 4.1.1] preserve stored allowDecimalPrecisionLoss in DecimalPrecision rule [#4179](https://github.com/apache/datafusion-comet/pull/4179) (andygrove)
- fix: [Spark 4.1.1] preserve parent struct nullness when all requested fields missing in Parquet [#4190](https://github.com/apache/datafusion-comet/pull/4190) (andygrove)
- fix: support Spark 4.1 BloomFilter V2 format and bit-scattering [#4196](https://github.com/apache/datafusion-comet/pull/4196) (andygrove)
- fix: JNI local reference cleanup in JVMClasses::with_env [#4225](https://github.com/apache/datafusion-comet/pull/4225) (0lai0)
- fix: broadcast exchange bypasses AQE partition coalescing [#4163](https://github.com/apache/datafusion-comet/pull/4163) (andygrove)
- fix: resolve Scala compiler warnings for auto-tupling and bare try [#4227](https://github.com/apache/datafusion-comet/pull/4227) (andygrove)
- fix: [Spark 4.1] preserve union output partitioning in CometUnionExec [#4207](https://github.com/apache/datafusion-comet/pull/4207) (andygrove)
- fix: re-enable tests skipped for Spark 4.1 (issue #4098) [#4253](https://github.com/apache/datafusion-comet/pull/4253) (andygrove)
- fix: cargo clean before release build to avoid stale native libs [#4257](https://github.com/apache/datafusion-comet/pull/4257) (andygrove)

**Performance related:**

- perf: avoid redundant columnar shuffle when both parent and child are non-Comet [#4010](https://github.com/apache/datafusion-comet/pull/4010) (andygrove)
- perf: reduce per-node allocations in to_native_metric_node [#4075](https://github.com/apache/datafusion-comet/pull/4075) (andygrove)

**Implemented enhancements:**

- feat: enable native Iceberg reader by default [#3819](https://github.com/apache/datafusion-comet/pull/3819) (andygrove)
- feat: support `collect_set` [#3954](https://github.com/apache/datafusion-comet/pull/3954) (comphead)
- feat: non-AQE DPP for native Parquet scans, broadcast exchange reuse for DPP subqueries [#4011](https://github.com/apache/datafusion-comet/pull/4011) (mbutrovich)
- feat: add support for array_position expression [#3172](https://github.com/apache/datafusion-comet/pull/3172) (andygrove)
- feat: Cast string to timestamp_ntz [#4034](https://github.com/apache/datafusion-comet/pull/4034) (parthchandra)
- feat: Add TimestampNTZType support for unix_timestamp [#4039](https://github.com/apache/datafusion-comet/pull/4039) (parthchandra)
- feat: fix array_compact for Spark 4.0 and correct return type metadata [#3796](https://github.com/apache/datafusion-comet/pull/3796) (andygrove)
- feat: task-level input metrics (bytesRead) for Iceberg native scan [#4128](https://github.com/apache/datafusion-comet/pull/4128) (mbutrovich)
- feat: add MapSort expression support for Spark 4.0 [#4076](https://github.com/apache/datafusion-comet/pull/4076) (andygrove)
- feat: Support Spark expression `str_to_map` [#3654](https://github.com/apache/datafusion-comet/pull/3654) (unknowntpo)
- feat: add support for timestamp_seconds expression [#3146](https://github.com/apache/datafusion-comet/pull/3146) (andygrove)
- feat: add config to gate converting Spark shuffle to Comet shuffle when child is non-Comet plan [#4166](https://github.com/apache/datafusion-comet/pull/4166) (andygrove)
- feat: AQE DPP for native Parquet scans with broadcast reuse [#4112](https://github.com/apache/datafusion-comet/pull/4112) (mbutrovich)
- feat: support regular BuildRight+LeftAnti hash join [#4073](https://github.com/apache/datafusion-comet/pull/4073) (viirya)
- feat: add bug-triage Claude skill [#4109](https://github.com/apache/datafusion-comet/pull/4109) (andygrove)
- feat: support `PartialMerge` aggregation mode [#4003](https://github.com/apache/datafusion-comet/pull/4003) (comphead)
- feat: add encode time tracking for shuffle operations [#4068](https://github.com/apache/datafusion-comet/pull/4068) (0lai0)
- feat: Add support for Spark ToDegrees and ToRadians math expressions [#3786](https://github.com/apache/datafusion-comet/pull/3786) (rafafrdz)
- feat: Add support for Spark Acosh, Asinh, Atanh math expressions [#3787](https://github.com/apache/datafusion-comet/pull/3787) (rafafrdz)
- feat: Add support for Spark Cbrt math expression [#3788](https://github.com/apache/datafusion-comet/pull/3788) (rafafrdz)
- feat: Add support for Spark Pi math expression [#3789](https://github.com/apache/datafusion-comet/pull/3789) (rafafrdz)
- feat: support Parquet field ID matching in native_datafusion scan [#4216](https://github.com/apache/datafusion-comet/pull/4216) (mbutrovich)
- feat: support AQE DPP broadcast reuse for Iceberg native scans [#4215](https://github.com/apache/datafusion-comet/pull/4215) (mbutrovich)
- feat: add support for url_encode, url_decode, and try_url_decode [#4231](https://github.com/apache/datafusion-comet/pull/4231) (parthchandra)
- feat: support TimestampType join keys in SortMergeJoin [#3986](https://github.com/apache/datafusion-comet/pull/3986) (andygrove)

**Documentation updates:**

- docs: Add changelog for 0.15.0 [#4000](https://github.com/apache/datafusion-comet/pull/4000) (andygrove)
- docs: Update README and benchmark results for 0.15.0 release [#3995](https://github.com/apache/datafusion-comet/pull/3995) (andygrove)
- docs: fix errors in benchmark pages [#4001](https://github.com/apache/datafusion-comet/pull/4001) (andygrove)
- docs: split compatibility guide into multiple pages [#4055](https://github.com/apache/datafusion-comet/pull/4055) (andygrove)
- docs: Generate expression compatibility docs from code [#4057](https://github.com/apache/datafusion-comet/pull/4057) (andygrove)
- doc: update documentation for cast and datetime functions [#4058](https://github.com/apache/datafusion-comet/pull/4058) (parthchandra)
- docs: add compatibility documentation to all expressions [#4067](https://github.com/apache/datafusion-comet/pull/4067) (andygrove)
- docs: rename SQL File Tests to Comet SQL Tests [#4108](https://github.com/apache/datafusion-comet/pull/4108) (andygrove)
- docs: add Understanding Comet Plans user guide page [#4086](https://github.com/apache/datafusion-comet/pull/4086) (andygrove)
- docs: support conditional content for snapshot vs release builds [#4030](https://github.com/apache/datafusion-comet/pull/4030) (andygrove)
- docs: update Spark version support and add version compatibility page [#4138](https://github.com/apache/datafusion-comet/pull/4138) (andygrove)
- docs: improve review skill and contributor guide for serde patterns [#4132](https://github.com/apache/datafusion-comet/pull/4132) (andygrove)
- docs: Fix errors in list of supported Spark versions [#4141](https://github.com/apache/datafusion-comet/pull/4141) (andygrove)
- docs: Update roadmap in contributor guide [#4144](https://github.com/apache/datafusion-comet/pull/4144) (andygrove)
- docs: add implement-comet-expression Claude skill [#4158](https://github.com/apache/datafusion-comet/pull/4158) (andygrove)
- docs: add roadmap items for spillable hash join, UDF support, memory management, and 1.0.0 [#4171](https://github.com/apache/datafusion-comet/pull/4171) (andygrove)
- docs: start Spark 4.1 known-limitations section, seeded with #4199 [#4202](https://github.com/apache/datafusion-comet/pull/4202) (andygrove)
- docs: document Spark 4 IntelliJ setup [#4198](https://github.com/apache/datafusion-comet/pull/4198) (yuboxx)
- docs: refresh Gluten comparison with ANSI, Spark 4, and Iceberg coverage [#4169](https://github.com/apache/datafusion-comet/pull/4169) (andygrove)
- docs: check off 53 implemented expressions in support doc [#4147](https://github.com/apache/datafusion-comet/pull/4147) (andygrove)
- docs: replace project logos with updated branding [#4230](https://github.com/apache/datafusion-comet/pull/4230) (pranamya123)
- docs: Documentation updates in preparation for 0.16 release [#4244](https://github.com/apache/datafusion-comet/pull/4244) (andygrove)

**Other:**

- chor: enable array_distinct [#3987](https://github.com/apache/datafusion-comet/pull/3987) (kazuyukitanimura)
- chore: Improve shuffle fallback logic [#3989](https://github.com/apache/datafusion-comet/pull/3989) (andygrove)
- chore: Start 0.16 development [#4016](https://github.com/apache/datafusion-comet/pull/4016) (andygrove)
- chore: update compatibility guide for primitive to string casts [#4012](https://github.com/apache/datafusion-comet/pull/4012) (parthchandra)
- test: add sql-file test confirming fallback on parquet variant reads [#4021](https://github.com/apache/datafusion-comet/pull/4021) (andygrove)
- chore: skip Iceberg and Spark SQL test workflows on test-only changes [#4023](https://github.com/apache/datafusion-comet/pull/4023) (andygrove)
- ci: force GNU ld on Linux to fix -ljvm linker error [#4024](https://github.com/apache/datafusion-comet/pull/4024) (andygrove)
- chore: update pending PR filter [#4025](https://github.com/apache/datafusion-comet/pull/4025) (comphead)
- test: run more Spark 4 tests [#4013](https://github.com/apache/datafusion-comet/pull/4013) (kazuyukitanimura)
- chore: update documentation links for 0.15.0 release [#4029](https://github.com/apache/datafusion-comet/pull/4029) (andygrove)
- test: run more JVM tests [#4026](https://github.com/apache/datafusion-comet/pull/4026) (kazuyukitanimura)
- feature: Support Spark expression: arrays_zip [#3643](https://github.com/apache/datafusion-comet/pull/3643) (hsiang-c)
- test: fix SparkToColumnar plan-shape assertions on Spark 4 [#4032](https://github.com/apache/datafusion-comet/pull/4032) (andygrove)
- test: unignore DynamicPartitionPruning static scan metrics test [#4038](https://github.com/apache/datafusion-comet/pull/4038) (andygrove)
- test: add non-AQE DPP edge case coverage for native Parquet scans [#4037](https://github.com/apache/datafusion-comet/pull/4037) (mbutrovich)
- test: unignore passing Spark 4.0 #3321 tests, retag remaining failures [#4041](https://github.com/apache/datafusion-comet/pull/4041) (andygrove)
- test: unignore passing DPP test, retag remaining failures [#4046](https://github.com/apache/datafusion-comet/pull/4046) (andygrove)
- chore: enable array_union [#4043](https://github.com/apache/datafusion-comet/pull/4043) (kazuyukitanimura)
- test: add date_trunc DST regression test in non-UTC session timezone [#4040](https://github.com/apache/datafusion-comet/pull/4040) (andygrove)
- ci: skip heavy test workflows for GenerateDocs and changelog changes [#4056](https://github.com/apache/datafusion-comet/pull/4056) (andygrove)
- chore(deps): bump the all-other-cargo-deps group in /native with 3 updates [#4061](https://github.com/apache/datafusion-comet/pull/4061) (dependabot[bot])
- ci: add concurrency blocks to more workflows to cancel on new commit [#4064](https://github.com/apache/datafusion-comet/pull/4064) (mbutrovich)
- chore: gitignore generated docs directories [#4066](https://github.com/apache/datafusion-comet/pull/4066) (andygrove)
- chore: promote array_insert to compatible and consolidate expression docs [#4065](https://github.com/apache/datafusion-comet/pull/4065) (andygrove)
- test: re-enable sql_hive-1 for Spark 4.0 and fix two small failures [#4047](https://github.com/apache/datafusion-comet/pull/4047) (andygrove)
- ci: stopping running PR builds with 3.5.5/3.5.6 [#4103](https://github.com/apache/datafusion-comet/pull/4103) (andygrove)
- chore: Bump Spark 4.0.1 to 4.0.2 [#4114](https://github.com/apache/datafusion-comet/pull/4114) (andygrove)
- build: add spark-4.1 Maven profile and shim sources [#4097](https://github.com/apache/datafusion-comet/pull/4097) (andygrove)
- refactor: consolidate identical spark-4.0 and spark-4.1 shims into spark-4.x [#4118](https://github.com/apache/datafusion-comet/pull/4118) (andygrove)
- ci: enable Spark 4.1 PR test matrix [#4106](https://github.com/apache/datafusion-comet/pull/4106) (andygrove)
- build: add spark-4.2 Maven profile targeting 4.2.0-preview4 [#4119](https://github.com/apache/datafusion-comet/pull/4119) (andygrove)
- ci: reduce macOS PR matrix to single Spark 4.0 profile [#4104](https://github.com/apache/datafusion-comet/pull/4104) (andygrove)
- chore: audit array_intersect and expand SQL test coverage [#4071](https://github.com/apache/datafusion-comet/pull/4071) (andygrove)
- chore: support Spark 4.1/4.2 for golden files generation [#4134](https://github.com/apache/datafusion-comet/pull/4134) (comphead)
- ci: add Spark 4.0 / JDK 21 profile [#4060](https://github.com/apache/datafusion-comet/pull/4060) (james-willis)
- ci: Enable Comet PR test matrix and TPCDS plan-stability for Spark 4.2 [#4126](https://github.com/apache/datafusion-comet/pull/4126) (andygrove)
- test: support fallback chain in CometPlanStabilitySuite, dedupe existing goldens [#4129](https://github.com/apache/datafusion-comet/pull/4129) (andygrove)
- chore: Update `spark_expressions_support.md` doc [#4165](https://github.com/apache/datafusion-comet/pull/4165) (comphead)
- build: Enable Spark SQL tests for Spark 4.1.1 [#4093](https://github.com/apache/datafusion-comet/pull/4093) (andygrove)
- test: unignore InjectRuntimeFilterSuite tests gated on issue #242 [#4178](https://github.com/apache/datafusion-comet/pull/4178) (andygrove)
- chore(deps): bump cc from 1.2.60 to 1.2.61 in /native in the all-other-cargo-deps group [#4168](https://github.com/apache/datafusion-comet/pull/4168) (dependabot[bot])
- test: [Spark 4.1.1] unignore SPARK-52921 union partitioning tests [#4195](https://github.com/apache/datafusion-comet/pull/4195) (andygrove)
- test: [Spark 4.1.1] unignore CachedBatchSerializerNoUnwrapSuite [#4204](https://github.com/apache/datafusion-comet/pull/4204) (andygrove)
- test: relax bytesRead ratio assertion for Spark 4.1+ [#4197](https://github.com/apache/datafusion-comet/pull/4197) (andygrove)
- deps: Bump OpenDAL to 0.56.0 [#4217](https://github.com/apache/datafusion-comet/pull/4217) (mbutrovich)
- ci: pin JDK per Spark version in Iceberg workflow matrix [#4120](https://github.com/apache/datafusion-comet/pull/4120) (dmefs)
- test: skip flaky StateStoreSuite under Comet and disambiguate JDK matrix names [#4226](https://github.com/apache/datafusion-comet/pull/4226) (andygrove)
- Map ProfileCredentialsProvider to profile credential chain [#4213](https://github.com/apache/datafusion-comet/pull/4213) (karuppayya)
- test: add explicit negative-count cases for array_repeat [#4182](https://github.com/apache/datafusion-comet/pull/4182) (slavlotski)
- test: add unhex dictionary coverage [#4222](https://github.com/apache/datafusion-comet/pull/4222) (yuboxx)
- chore: Enable GitHub project view [#4247](https://github.com/apache/datafusion-comet/pull/4247) (andygrove)
- test: dedupe redundant Spark 4.2 TPC-DS plan-stability golden files [#4243](https://github.com/apache/datafusion-comet/pull/4243) (andygrove)
- build: change default Maven profile to Spark 4.1 / Scala 2.13 [#4140](https://github.com/apache/datafusion-comet/pull/4140) (andygrove)
- chore: remove legacy ENABLE_COMET_SCAN_ONLY and ENABLE_COMET_ANSI_MODE env vars from Spark diffs [#4252](https://github.com/apache/datafusion-comet/pull/4252) (parthchandra)

## Credits

Thank you to everyone who contributed to this release. Here is a breakdown of commits (PRs merged) per contributor.

```
    78	Andy Grove
     9	Matt Butrovich
     7	Parth Chandra
     6	KAZUYUKI TANIMURA
     5	ChenChen Lai
     5	Oleks V
     4	Rafael Fernández
     2	Karuppayya
     2	Yubo Xu
     2	dependabot[bot]
     1	Eric Chang
     1	James Willis
     1	Kuo-Hao Huang
     1	Liang-Chi Hsieh
     1	Pranamya Vadlamani
     1	Vladislav Zabolotsky
     1	hsiang-c
```

Thank you also to everyone who contributed in other ways such as filing issues, reviewing PRs, and providing feedback on this release.
