Uploaded image for project: 'Apache Hudi'
  1. Apache Hudi
  2. HUDI-309

General Redesign of Archived Timeline for efficient scan and management

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 1.0.0-beta1
    • Common Core
    • None

    Description

      As designed by Vinoth:

      Goals

      1. Archived Metadata should be scannable in the same way as data
      2. Provides more safety by always serving committed data independent of timeframe when the corresponding commit action was tried. Currently, we implicitly assume a data file to be valid if its commit time is older than the earliest time in the active timeline. While this works ok, any inherent bugs in rollback could inadvertently expose a possibly duplicate file when its commit timestamp becomes older than that of any commits in the timeline.
      3. We had to deal with lot of corner cases because of the way we treat a "commit" as special after it gets archived. Examples also include Savepoint handling logic by cleaner.
      4. Small Files : For Cloud stores, archiving simply moves fils from one directory to another causing the archive folder to grow. We need a way to efficiently compact these files and at the same time be friendly to scans

      Design:

       The basic file-group abstraction for managing file versions for data files can be extended to managing archived commit metadata. The idea is to use an optimal format (like HFile) for storing compacted version of <commitTime, Metadata> pairs. Every archiving run will read <commitTime, Metadata> pairs from active timeline and append to indexable log files. We will run periodic minor compactions to merge multiple log files to a compacted HFile storing metadata for a time-range. It should be also noted that we will partition by the action types (commit/clean).  This design would allow for the archived timeline to be queryable for determining whether a timeline is valid or not.

      Attachments

        1. Archive TImeline Notes by Vinoth 1.jpg
          102 kB
          Balaji Varadarajan
        2. Archived Timeline Notes by Vinoth 2.jpg
          92 kB
          Balaji Varadarajan

        Issue Links

          Activity

            People

              danny0405 Danny Chen
              vbalaji Balaji Varadarajan
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: