Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-10656

Fire insert events before commit

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: In Progress
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Backend, Frontend
    • Labels:
      None
    • Epic Color:
      ghx-label-12

      Description

      Currently Impala commits an insert first, then reloads the table from HMS, and generates the insert events based on the difference between the two snapshots. (e.g. which file was not present in the old snapshot but are there in the new). Hive replication expects the insert events before the commit, so this may potentially lead to issues there,

      The solution is to collect the new files during the insert in the backend, and send the insert events based on this file set.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                csringhofer Csaba Ringhofer
                Reporter:
                csringhofer Csaba Ringhofer
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated: