Skip to content

Avro: Cache Avro schema conversion in AvroFileHeader.get_schema (#3662) - #3844

Open
hedger9487 wants to merge 1 commit into
apache:mainfrom
hedger9487:fix/cache-avro-schema-conversion-3662
Open

Avro: Cache Avro schema conversion in AvroFileHeader.get_schema (#3662)#3844
hedger9487 wants to merge 1 commit into
apache:mainfrom
hedger9487:fix/cache-avro-schema-conversion-3662

Conversation

@hedger9487

Copy link
Copy Markdown

Description

Fixes #3662.

Every manifest file under a partition spec embeds an identical Avro schema string (avro.schema). Previously, AvroFileHeader.get_schema() in pyiceberg/avro/file.py performed a full json.loads() and recursive AvroSchemaConversion().avro_to_iceberg() conversion on every manifest read during scan planning.

This PR adds an LRU cache for Avro schema string parsing via _parse_avro_schema, avoiding redundant conversions across repeated manifest reads during scan planning (e.g. accelerating scan().plan_files() by ~1.8x on tables with many manifests).

Testing

  • Added unit test test_get_schema_is_cached in tests/avro/test_file.py.
  • All 49 Avro and manifest tests pass locally.

Copilot AI lite review requested due to automatic review settings August 24, 2026 21:09

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR optimizes Avro manifest/schema handling by caching the Avro-schema-string → Iceberg Schema conversion, avoiding repeated json.loads() + recursive conversion work across many manifest reads during scan planning.

Changes:

  • Added an @lru_cache(maxsize=128)-backed _parse_avro_schema() helper to cache schema parsing/conversion by schema string.
  • Updated AvroFileHeader.get_schema() to use the cached helper.
  • Added a unit test asserting schema conversion results are reused (object identity) across headers with the same embedded schema string.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File Description
pyiceberg/avro/file.py Introduces an LRU-cached schema parsing/conversion helper and routes get_schema() through it.
tests/avro/test_file.py Adds a regression test to ensure schema conversion is cached and reused across calls.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Avro schema is re-converted to an Iceberg schema on every manifest read during scan planning

2 participants