Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-14520

WASB: Block compaction for Azure Block Blobs

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.7.4
    • Fix Version/s: 2.9.0, 3.0.0-beta1
    • Component/s: fs/azure
    • Labels:
      None
    • Release Note:
      Block Compaction for Azure Block Blobs. When the number of blocks in a block blob is above 32000, the process of compaction replaces a sequence of small blocks with with one big block.

      Description

      Block Compaction for WASB allows uploading new blocks for every hflush/hsync call. When the number of blocks is above 32000, next hflush/hsync triggers the block compaction process. Block compaction replaces a sequence of blocks with one block. From all the sequences with total length less than 4M, compaction chooses the longest one. It is a greedy algorithm that preserve all potential candidates for the next round. Block Compaction for WASB increases data durability and allows using block blobs instead of page blobs. By default, block compaction is disabled. Similar to the configuration for page blobs, the client needs to specify HDFS folders where block compaction over block blobs is enabled.

      Results for HADOOP_14520_07.patch
      tested endpoint: fs.azure.account.key.hdfs4.blob.core.windows.net
      Tests run: 777, Failures: 0, Errors: 0, Skipped: 155

        Attachments

        1. HADOOP_14520_07.patch
          87 kB
          Georgi Chalakov
        2. HADOOP_14520_08.patch
          88 kB
          Georgi Chalakov
        3. HADOOP_14520_09.patch
          88 kB
          Georgi Chalakov
        4. HADOOP_14520_10.patch
          91 kB
          Georgi Chalakov
        5. HADOOP-14520-006.patch
          93 kB
          Thomas Marquardt
        6. HADOOP-14520-008.patch
          90 kB
          Steve Loughran
        7. HADOOP-14520-009.patch
          92 kB
          Steve Loughran
        8. HADOOP-14520-05.patch
          104 kB
          Georgi Chalakov
        9. hadoop-14520-branch-2-010.patch
          92 kB
          Georgi Chalakov
        10. HADOOP-14520-patch-07-08.diff
          19 kB
          Steve Loughran
        11. HADOOP-14520-patch-07-09.diff
          25 kB
          Steve Loughran

          Issue Links

            Activity

              People

              • Assignee:
                Georgi Georgi Chalakov
                Reporter:
                Georgi Georgi Chalakov
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: