Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-10656

Fire insert events before commit

Attach filesAttach ScreenshotAdd voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: In Progress
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Backend, Frontend
    • Labels:
      None
    • Epic Color:
      ghx-label-12

      Description

      Currently Impala commits an insert first, then reloads the table from HMS, and generates the insert events based on the difference between the two snapshots. (e.g. which file was not present in the old snapshot but are there in the new). Hive replication expects the insert events before the commit, so this may potentially lead to issues there,

      The solution is to collect the new files during the insert in the backend, and send the insert events based on this file set.

        Attachments

        Issue Links

          Activity

            People

            • Assignee:
              csringhofer Csaba Ringhofer
              Reporter:
              csringhofer Csaba Ringhofer

              Dates

              • Created:
                Updated:

                Issue deployment