Uploaded image for project: 'Apache Hudi'
  1. Apache Hudi
  2. HUDI-1763

DefaultHoodieRecordPayload does not honor ordering value when records within multiple log files are merged

    XMLWordPrintableJSON

    Details

      Description

      While creating HoodieRecordPayloads from log files in case of MOR tables, the payloads are created without any orderingVal (even if specified while writing data). Due to this the precombine function could result in any payload irrespective of its orderingVal.

      Attaching a sample script to reproduce the issue.

      In this example, for key "key1", 1st insert is with ts=1000. Then we update with ts=2000. Thenn we updated with ts=500. Ideally after last update if we snnapshot query the table, we must get key1 with ts=2000 (since our ordering field is ts). However it shows entry of ts=1000 because from logs it ignores ts=2000 and only picks up ts=500.

      Also AFAIU, the same flow will be used while compaction and then we might lose data forever.

       

      More info: https://github.com/apache/hudi/issues/2756

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                shivnarayan sivabalan narayanan
                Reporter:
                shivnarayan sivabalan narayanan
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated: