Uploaded image for project: 'Apache Hudi'
  1. Apache Hudi
  2. HUDI-2535

Late arriving records and global index with partition path update set to true

    XMLWordPrintableJSON

Details

    Description

      incase of a global index, we have a config to update partition path. if this is set to true, if there is an incoming record to a newer partition compared to whats in storage, older record will be deleted and new incoming record will be routed to new partition. 

      But it could run into issues if new incoming is a late arriving record. Expected behavior is, old record is retained and new one is discarded it it has lower preCombine value. But in this case, we may not honor that. 

       

      Attachments

        Issue Links

          Activity

            People

              shivnarayan sivabalan narayanan
              shivnarayan sivabalan narayanan
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated: