Skip to content

Add optional Parquet page index writes - #3829

Open
adonm wants to merge 1 commit into
apache:mainfrom
adonm:feat/parquet-page-index
Open

Add optional Parquet page index writes#3829
adonm wants to merge 1 commit into
apache:mainfrom
adonm:feat/parquet-page-index

Conversation

@adonm

@adonm adonm commented Aug 21, 2026

Copy link
Copy Markdown

Related: #3847

Rationale for this change

PyArrow supports page-index writing, but PyIceberg did not expose it through table properties. Readers such as ClickHouse can use these indexes to avoid decoding nonmatching pages for selective queries.

What changes?

Adds an opt-in write.parquet.page-index-enabled table property. When enabled, PyIceberg passes write_page_index=True to PyArrow's ParquetWriter, producing Parquet column and offset indexes for page-level predicate pruning.

The default is false, preserving existing output and file-size behavior.

Are these changes tested?

  • Added a direct unit test for default/opt-in writer kwargs
  • Extended the existing Parquet writer-property integration parameterization
  • PYTHONPATH=. uv run pytest tests/io/test_pyarrow.py -k parquet_page_index_writer_property -q — passed
  • ruff check on changed Python files — passed
  • git diff --check — passed

Are there any user-facing changes?

Yes. A new opt-in write.parquet.page-index-enabled table property (default false), documented in mkdocs/docs/configuration.md. Tables with the property enabled write Parquet column and offset indexes. Changelog label requested.

Tooling note: developed with assistance from DS v4 Pro. I reviewed and verified the changes.

@adonm

adonm commented Aug 25, 2026

Copy link
Copy Markdown
Author

Synced with latest main (13 commits, including the pyarrow 24→25 upgrade). No conflicts.

Verified end-to-end against a REST catalog: with write.parquet.page-index-enabled=true the written Parquet has column+offset indexes (has_column_index/has_offset_index true), and default tables still write without them. Unit tests, ruff, and mypy (zero new errors vs main) pass.

@adonm
adonm force-pushed the feat/parquet-page-index branch from 9b4d96a to 06bf26d Compare August 25, 2026 04:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant