equality_ids written as list<long>, but spec and all other implementations use list<int>

[HaoXuAI]

Apache Iceberg version

main (development)

Please describe the bug 🐞

equality_ids is declared as a list of long in the manifest-entry Avro schema, but the Iceberg spec, the Java reference implementation, and iceberg-cpp all specify int. pyiceberg is the only implementation that disagrees, so manifests it writes are rejected outright by spec-conformant readers.

Evidence

source	element type of field 136
Iceberg spec, format/spec.md ("Manifests" table)	list<136: int>
Java, api/src/main/java/org/apache/iceberg/DataFile.java	ListType.ofRequired(136, IntegerType.get())
iceberg-cpp, src/iceberg/manifest_entry.h	iceberg::int32()
pyiceberg, pyiceberg/manifest.py:298 and :393	LongType()

Both the v1 and v2 manifest-entry schemas are affected. Note that pyiceberg's own type hint already says int:

# pyiceberg/manifest.py:528
def equality_ids(self) -> list[int] | None:

So only the Avro schema disagrees; the Python side already treats these as ints.

Impact

A spec-conformant reader cannot read a pyiceberg-written manifest at all. iceberg-cpp fails with:

Cannot read Iceberg type: int from Avro type: "long"

because its kInt case accepts only AVRO_INT (Avro permits int → long promotion, not the reverse). Any Java or C++ consumer of pyiceberg-written metadata hits the same wall.

This is not limited to equality-delete tables: the schema is written into every manifest, so an ordinary append-only table with no delete files is equally unreadable.

Reproducer

import fastavro

# any table written by pyiceberg
with open("<some>-m0.avro", "rb") as f:
    schema = fastavro.reader(f).writer_schema

# -> equality_ids: ["null", {"element-id": 136, "type": "array", "items": "long"}]
#    spec says items should be "int"

The obvious fix breaks backwards compatibility

Changing LongType() → IntegerType() in those two places makes pyiceberg unable to read every manifest it has previously written:

ResolveError: Cannot promote long to int

Two reasons compound here:

1. pyiceberg/schema.py registers promote for IntegerType (int → long is legal) but has no LongType registration, so long → int falls through to the generic case and raises (schema.py:1660).
2. MANIFEST_ENTRY_SCHEMAS is used for both directions — the writer takes file_schema from it and the reader takes record_schema from it — so changing the constant changes the read path too.

Verified empirically: patching field 135's element type to IntegerType and then reading a manifest written by the current release raises the above.

Possible directions

A correct fix has to write int while continuing to read long. Options, roughly:

• Split the write and read schemas so the writer emits the spec type while the reader keeps accepting the historical one. Probably the cleanest, and it changes no promotion rules.
• Allow long → int narrowing for this field specifically. Smaller diff, but it widens the resolver in a direction the Avro and Iceberg promotion rules deliberately exclude.

Deliberately not proposing a patch — which trade-off is acceptable seems like a maintainer call, and there may be a preferred pattern for schema-compatibility shims that I am not aware of.

Environment

• pyiceberg 0.11.1 (current release) and main, both affected
• Cross-checked against the Iceberg spec, the Java reference implementation, and iceberg-cpp
• Found while making iceberg-cpp read a pyiceberg-written table

Status checks

Still present on main. Verified against main on 2026-08-24: manifest.py:298 and :393 both still declare LongType(). Separately, no commit in the repository's history mentions equality_ids at all, and the recent commits touching manifest.py are unrelated (manifest cache configuration, min sequence number, ManifestEntry.snapshot_id setter, lint rules).

No prior report found, though this is the weaker of the two claims. Searching this repository for equality_ids returns 13 issues and PRs, all of which mention it incidentally (equality-delete tests, manifest entry status, the files metadata table) rather than discussing its declared type. Searches for "Cannot promote long to int" and "Cannot read Iceberg type" return nothing. It is entirely possible this has been raised somewhere I did not look — the dev mailing list or Slack, or phrased in a way my queries missed.

Related but distinct: #3744 (Metadata inspection APIs fail with struct.error after int→long / float→double type promotion) also concerns int/long handling, but it is about promotion of user table columns breaking inspect.* on pre-promotion data files. It does not involve the manifest-entry schema or equality_ids, so the two should not be conflated.

Willingness to contribute

• I can contribute a fix for this bug independently
• I would be willing to contribute a fix for this bug with guidance from the Iceberg community
• I cannot contribute a fix for this bug at this time