Uploaded image for project: 'Apache Hudi'
  1. Apache Hudi
  2. HUDI-2286

Handle the case of failed deltacommit on the metadata table.

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.9.0
    • Component/s: None

      Description

      Assume the current timeline state is as follows:
      Dataset: C10
      Metadata Table: DC10

       

      Next ingestion run attempts a commit which succeeds. But syncing to metadata table fails. This is the new timeline state:
      Dataset: C10  C11
      Metadata Table: DC10   DC11.inflight

       

      Next ingestion run attempts to sync metadata table in preWrite(). This will do the following:
      1. Get list of instants to sync (C11)
      2. MetadataTable.startCommitWithTime(C11)
      3. autoRollback DC11.inflight

                       At this time the timelines will be:
                          Dataset: C10. C11
                          Metadata Table: DC10. Rollback12

      4. startCommitWithTime fails here with the following exception:
      21/07/17 15:59:43 ERROR client.AbstractHoodieWriteClient: Cannot start a new commit at time 20210717141448 as there are future commits present: [[==>20210717141448__deltacommit__REQUESTED]]

       

      Next ingestion run attempts a commit which succeeds. This is the new timeline state just after the commit has finished:
               Dataset: C10  C11 C13
                Metadata Table: DC10. Rollback12

      Metadata table sync will be called. It will pickup all instants to sync after the last timestamp on the metadata table - C13 (last timestamp on metadata = 12). Sync will succeed leading to this final timeline state:
      Dataset: C10. C11 C13
      Metadata Table: DC10. Rollback12 DC13

      So C11 never got committed to the metadata table.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                pwason Prashant Wason
                Reporter:
                pwason Prashant Wason
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: