Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-9042 Support reading full-ACID ORC tables
  3. IMPALA-9512

Milestone 2: Validate each row against the valid write id list

Attach filesAttach ScreenshotVotersWatch issueWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • Impala 4.0.0
    • None
    • ghx-label-11

    Description

      Minor compactions can compact several delta directories into a single delta directory. The current directory filtering algorithm needs to be modified to handle minor compacted directories and prefer those over plain delta directories.

      On top of that, in minor compacted directories we need to filter out rows we cannot see. E.g. we can have the following delta directory:

      full_acid/delta_0000001_0000010_0000/0000 # minWriteId: 1
                                                # maxWriteId: 10
      

      So this delta dir contains rows with write ids between 1 and 10. But maybe we are only allowed to see write ids less than 5. Therefore we need to check the ACID write id column (named originalTransaction) for each row to decide whether this row is valid or not.

      There are several ways to optimize this. E.g. based on the min/max write ids of the delta directory, and the validWriteIdList, we can decide whether we need to validate the rows at all. Or, when we reach the high watermark (that tells us the max valid write id) we can stop the scanner since rows are ordered based on record ID.

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            boroknagyz Zoltán Borók-Nagy
            boroknagyz Zoltán Borók-Nagy
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment