feat(storage): default parquet codec to zstd for flink and spark 3.5+

[cshuo]

Describe the issue this Pull Request addresses

Closes #19684.

Flink and Spark 3.5+ currently default Parquet base and native log files to GZIP, which provides a good compression ratio but adds significant CPU overhead to the write path. ZSTD offers a better balance of compression throughput and storage efficiency. Spark 3.3/3.4 require a compatibility exception because Hudi's non-vectorized file-group reader can still encounter the PARQUET-2160 off-heap leak when reading ZSTD files.

This implements the engine- and runtime-specific behavior discussed in #19615.

Summary and Changelog

• Set the storage-level Parquet codec default to ZSTD while deferring materialization until the write engine is known.
• Default Flink and Spark 3.5+ writes to ZSTD; retain GZIP for Spark 3.3/3.4, Java clients, and Spark version detection failures.
• Preserve explicit hoodie.parquet.compression.codec values in DataSource/SQL, direct clients, metadata table services, and derived configurations.
• Include zstd-jni in the Flink bundle and continue relying on the Spark runtime dependency without adding it to Spark bundles.
• Add coverage for engine defaults, explicit overrides, partial storage configurations, metadata write configs, and actual ZSTD Parquet output.

Impact

Upgrading changes the default codec for writes without an explicit codec under Flink and Spark 3.5+. Existing Parquet files remain readable and a table can contain files using different codecs. Spark 3.3/3.4 and Java write defaults remain GZIP.

The Flink bundle gains the profile-compatible zstd-jni classes and native libraries. Spark bundles do not duplicate zstd-jni; supported Spark runtimes provide it through Spark's dependency set. Explicit codec configuration continues to take precedence.

Risk Level

Medium. This intentionally changes a write default and adds a native dependency to the Flink bundle. The risk is mitigated by retaining GZIP on affected Spark versions and Java clients, preserving explicit overrides, verifying Spark 3.3 and 3.5 profiles independently, testing the Flink write-client path, and writing a Parquet file whose footer reports ZSTD compression.

Validation performed:

• Targeted HoodieStorageConfig, HoodieWriteConfig, and Hadoop Parquet writer tests.
• TestDataSourceUtils#testSparkVersionSpecificParquetCompressionCodecDefault with Spark 3.3 and Spark 3.5 profiles.
• TestFlinkWriteClients#testParquetCompressionCodecDefaultAndOverride with the Flink 2.1 profile.
• Dependency-tree checks for Hadoop, Java, Flink, and Spark zstd-jni resolution.
• Bundle-content verification that the Flink bundle contains ZSTD classes/native libraries and the Spark bundle does not contain a duplicate copy.

Documentation Update

Updated the root and Spark datasource documentation, plus the configuration description, to describe the version-specific defaults and warn that Spark 3.3/3.4 Hudi file-group readers may encounter PARQUET-2160 when reading ZSTD Parquet files. Spark 3.5+ is recommended.

Contributor's checklist

• Read through contributor's guide
• Enough context is provided in the sections above
• Adequate tests were added if applicable

[codecov-commenter]

Codecov Report

❌ Patch coverage is 93.10345% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 77.84%. Comparing base (42e885f) to head (3293ab8).
⚠️ Report is 5 commits behind head on master.

Files with missing lines	Patch %	Lines
...java/org/apache/hudi/config/HoodieWriteConfig.java	88.88%	0 Missing and 2 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             master   #19685    +/-   ##
==========================================
  Coverage     77.83%   77.84%            
- Complexity    33064    33143    +79     
==========================================
  Files          2525     2527     +2     
  Lines        139670   139963   +293     
  Branches      16800    17164   +364     
==========================================
+ Hits         108709   108949   +240     
- Misses        23347    23395    +48     
- Partials       7614     7619     +5     

Components	Coverage Δ
hudi-common	83.35% <100.00%> (+0.04%)	⬆️
hudi-client	82.84% <89.47%> (+0.02%)	⬆️
hudi-flink	85.71% <ø> (-0.04%)	⬇️
hudi-spark-datasource	72.27% <ø> (+0.02%)	⬆️
hudi-utilities	74.03% <ø> (-0.05%)	⬇️
hudi-cli	15.06% <ø> (-0.20%)	⬇️
hudi-hadoop	69.08% <100.00%> (+0.05%)	⬆️
hudi-sync	75.55% <ø> (+0.02%)	⬆️
hudi-io	79.90% <ø> (+0.48%)	⬆️
hudi-timeline-service	84.32% <ø> (+0.58%)	⬆️
hudi-cloud	64.33% <ø> (ø)
hudi-kafka-connect	53.20% <ø> (ø)

Flag	Coverage Δ
common-and-other-modules	50.99% <89.65%> (+0.02%)	⬆️
flink-integration-tests	49.13% <86.20%> (-0.01%)	⬇️
hadoop-mr-java-client	43.92% <89.65%> (+<0.01%)	⬆️
integration-tests	13.65% <72.41%> (+0.02%)	⬆️
spark-client-hadoop-common	50.64% <93.10%> (-0.01%)	⬇️
spark-java-tests	51.95% <93.10%> (+0.07%)	⬆️
spark-scala-tests	46.47% <93.10%> (+<0.01%)	⬆️
utilities	36.62% <89.65%> (+0.04%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
...apache/hudi/metadata/HoodieMetadataWriteUtils.java	94.44% <100.00%> (+0.02%)	⬆️
...va/org/apache/hudi/common/config/HoodieConfig.java	95.41% <100.00%> (+0.12%)	⬆️
...apache/hudi/common/config/HoodieStorageConfig.java	90.70% <100.00%> (+0.61%)	⬆️
...io/storage/hadoop/HoodieAvroFileWriterFactory.java	92.98% <100.00%> (ø)
...java/org/apache/hudi/config/HoodieWriteConfig.java	92.07% <88.88%> (+0.03%)	⬆️

... and 59 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[hudi-bot]

CI report:

• 3293ab8 Azure: FAILURE

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build