Skip to content

Support assume-role for Glue and DynamoDB catalogs - #3838

Open
lemon0333 wants to merge 1 commit into
apache:mainfrom
lemon0333:feat/assume-role-glue-dynamodb-catalog
Open

Support assume-role for Glue and DynamoDB catalogs#3838
lemon0333 wants to merge 1 commit into
apache:mainfrom
lemon0333:feat/assume-role-glue-dynamodb-catalog

Conversation

@lemon0333

Copy link
Copy Markdown

Closes #2747

Rationale for this change

At the moment, assume-role (client.role-arn / client.role-session-name, via STS) is only honored by the S3 FileIO (pyiceberg/io/pyarrow.py). The AWS catalogs (Glue, DynamoDB) build their own boto3.Session and ignore these properties, so users who authenticate to AWS by assuming a role cannot use it to reach the catalog — even though the other client.* properties (region, access key, etc.) are already shared between S3 FileIO and the catalogs.

This PR extends the same assume-role behavior to the GlueCatalog and DynamoDbCatalog boto3 sessions:

  • A small shared helper _get_aws_session_with_assumed_role() (in pyiceberg/catalog/__init__.py) performs the STS AssumeRole from the base session and returns a new session using the returned temporary credentials.
  • Each catalog resolves the role config with the same precedence already used for the other client.* properties: a service-prefixed override falls back to the unified property:
    • glue.role-arn / dynamodb.role-arnclient.role-arn
    • glue.role-session-name / dynamodb.role-session-nameclient.role-session-name
  • When no role ARN is configured, behavior is unchanged.

I've kept the new service-prefixed property names consistent with the existing glue.* / dynamodb.*client.* convention, but I'm happy to adjust the naming (e.g. unified-only) to match maintainer preference — feedback welcome.

Are these changes tested?

Yes. New moto-based tests (mock STS + Glue + DynamoDB) in tests/catalog/test_glue.py and tests/catalog/test_dynamodb.py cover:

  • assume-role is triggered via the unified client.role-arn / client.role-session-name;
  • service-prefixed properties (glue.role-arn / dynamodb.role-arn) take precedence over the unified ones;
  • no assume-role call is made when no role ARN is set (default behavior unchanged);
  • the helper returns a session backed by temporary STS credentials (session token present).
uv run python -m pytest tests/catalog/test_glue.py tests/catalog/test_dynamodb.py -k "role or assume or session_properties"
# 11 passed

uv run python -m pytest tests/catalog/test_glue.py tests/catalog/test_dynamodb.py
# 98 passed

uv run prek run --files pyiceberg/catalog/__init__.py pyiceberg/catalog/glue.py pyiceberg/catalog/dynamodb.py \
  tests/catalog/test_glue.py tests/catalog/test_dynamodb.py
# ruff / ruff-format / mypy / pydocstyle / codespell: all Passed

Are there any user-facing changes?

Yes — new opt-in configuration properties for assuming an IAM role when connecting to the Glue/DynamoDB catalogs: client.role-arn / client.role-session-name (now honored by the catalogs too), plus service-specific glue.role-arn / glue.role-session-name and dynamodb.role-arn / dynamodb.role-session-name. Documented in mkdocs/docs/configuration.md. No change to default behavior.

Extend the existing S3 FileIO assume-role mechanism (client.role-arn /
client.role-session-name) to the boto3 sessions built by GlueCatalog and
DynamoDbCatalog. When a role ARN is configured, an STS AssumeRole is
performed from the base session and the resulting temporary credentials
are used for the Glue/DynamoDB client.

Role configuration follows the same precedence already used for the other
client.* properties: a service-prefixed override (glue.role-arn /
dynamodb.role-arn, glue.role-session-name / dynamodb.role-session-name)
falls back to the unified client.role-arn / client.role-session-name.
Default behavior (no role ARN configured) is unchanged.

Closes apache#2747

Signed-off-by: lemon0333 <147061193+lemon0333@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Allow to assume role when connecting to AWS catalogs (glue/dynamodb)

1 participant