Uploaded image for project: 'Apache Hudi'
  1. Apache Hudi
  2. HUDI-5863

Fix HoodieMetadataFileSystemView serving stale view at the timeline server

    XMLWordPrintableJSON

Details

    Description

      We observe that for MOR table, occasionally (<10% for large tables with frequent updates and compactions), the deltacommit after the compaction commit may add a new log file to the old file slice, not the latest file slice, in the corresponding file group.  This happens when both the metadata table and timeline server are enabled.  If either is disabled, the problem does not show up.

      Deeper analysis of the code surfaces that the file system view at the timeline server may serve the stale view, causing the issue.  This is because the sync of HoodieMetadataFileSystemView is not atomic when the metadata table is enabled:

       

      AbstractTableFileSystemView:
      
      @Override
      public void sync() {
        HoodieTimeline oldTimeline = getTimeline();
        HoodieTimeline newTimeline = metaClient.reloadActiveTimeline().filterCompletedOrMajorOrMinorCompactionInstants();
        try {
          writeLock.lock();
          runSync(oldTimeline, newTimeline);
        } finally {
          writeLock.unlock();
        }
      }
      
      HoodieMetadataFileSystemView:
      
      @Override
        public void sync() {
          super.sync();
          tableMetadata.reset();
        }
      

      The
      tableMetadata.reset()
      happens outside the write lock, causing the issue during race conditions (reading the inconsistent state while the table metadata is being reset).

       

       

      Attachments

        Issue Links

          Activity

            People

              guoyihua Ethan Guo
              guoyihua Ethan Guo
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: