Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-7127 Fetch-on-demand metadata for the impalad-side catalog
  3. IMPALA-7141

Extract interfaces for partition pruning prior to fetching partitions

Attach filesAttach ScreenshotVotersWatch issueWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: Impala 2.13.0, Impala 3.1.0
    • Component/s: None
    • Labels:
      None
    • Epic Color:
      ghx-label-7

      Description

      In the LocalCatalog, we want to only fetch the partitions that are referenced by a query – i.e. we must prune partitions based only on the partition names and not the entire partition objects. However, the PartitionPruner implementation currently expect to be able to fetch the full map of HdfsPartition objects from the table and work on them as is.

      This JIRA is to do some refactorings such that the PartitionPruner interacts with a slightly more restricted interface that only exposes the minimal interaction points with the table and the partition map. Once it has computed a list of remaining partitions, it can then instruct the table to fully load them to yield the resulting full Partition objects.

        Attachments

          Activity

            People

            • Assignee:
              tlipcon Todd Lipcon
              Reporter:
              tlipcon Todd Lipcon

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment