Skip to content

Add support for bucket expression to table scans #3839

Description

@psavalle

Feature Request / Improvement

For time partitioning, we can express time range expressions and they lead to partition pruning when planning an Arrow scan:

 scan = table.scan(
      row_filter=And(
          GreaterThanOrEqual("event_ts", start),
          LessThan("event_ts", end),
      )
  )

  arrow_table = scan.to_arrow()

It would be useful to be able to filter by other hidden partitioning transforms, such as buckets -- e.g. filtering on bucket[16](user_id) in {0, 1, 2, 3}, and getting partition pruning whenever possible based on the underlying table partitioning specs (falling back to filtering rows when files cannot be pruned).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions