PyArrow: Support dictionary-encoded columns in projection and upsert (#3835)

[hedger9487]

Description

Fixes #3835.

When appending, overwriting, or upserting dataframes containing dictionary-encoded columns (pa.DictionaryArray), ArrowProjectionVisitor._cast_if_needed did not cast dictionary arrays to the target schema type (e.g. pa.string()). As a result, newly written data files retain dictionary types while existing files contain plain strings, causing subsequent table.scan().to_arrow() or upsert table concats to fail with:

pyarrow.lib.ArrowTypeError: Unable to merge: Field name has incompatible types: dictionary<values=string, indices=int32, ordered=0> vs string

This PR updates ArrowProjectionVisitor._cast_if_needed in pyiceberg/io/pyarrow.py to cast DictionaryArray columns to the target type when the target is not a dictionary type.

Testing

• Added unit test test_upsert_dictionary_encoded_columns in tests/table/test_upsert.py.
• All 24 upsert tests and full test suites pass locally.

[Copilot]

Pull request overview

This PR fixes a PyArrow interoperability issue where dictionary-encoded columns (pa.DictionaryArray) were not consistently cast to the table’s target schema type during projection, leading to incompatible Arrow schemas across data files and failures when reading/merging results (e.g., table.scan().to_arrow() after upserts).

Changes:

• Cast dictionary-encoded primitive columns to the target (non-dictionary) PyArrow type in ArrowProjectionVisitor._cast_if_needed.
• Add a regression test covering upsert behavior with a dictionary-encoded string column and verifying successful read-back via scan().to_arrow().

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File	Description
pyiceberg/io/pyarrow.py	Ensures dictionary-encoded arrays are cast to the non-dictionary target type during schema projection to avoid incompatible Arrow field types across files.
tests/table/test_upsert.py	Adds a unit test reproducing the dictionary-encoding upsert scenario and validating clean Arrow read-back.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.