Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-21810

bulkload support set hfile compression on client

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.3.3, 1.4.9, 2.0.4, 2.1.3, 1.2.11
    • Fix Version/s: 3.0.0-alpha-1, 1.5.0, 2.2.0, 2.3.0
    • Component/s: mapreduce
    • Labels:
      None
    • Release Note:
      Hide
      bulkload (HFileOutputFormat2) support config the compression on client ,you can set the job configuration "hbase.mapreduce.hfileoutputformat.compression" override the auto-detection of the target table's compression

      Show
      bulkload (HFileOutputFormat2) support config the compression on client ,you can set the job configuration "hbase.mapreduce.hfileoutputformat.compression" override the auto-detection of the target table's compression

      Description

      hbase bulkload (HFileOutputFormat2) generate hfile ,the compression from the table(cf) compression,

      if the compression can be set on client ,sometimes,it's useful,

      some case in our production:

      1、hfile bulkload replication between the data center with bandwidth limit, we can set the compression of the bulkload hfile not changing the table compression

      2、bulkload hfile not set  compression ,but the table compression is gz/zstd/snappy... ,can reduce the hfile created time and compaction will make the hfile to compression finally

      3、somethings the yarn nodes (hfile created by reduce) /dobulkload client has no compression lib,but the hbase cluster has,it's useful for this case

        Attachments

        1. HBASE-21810.branch-1.001.patch
          3 kB
          Yechao Chen
        2. HBASE-21810.branch-1.002.patch
          5 kB
          Yechao Chen
        3. HBASE-21810.branch-1.003.patch
          5 kB
          Yechao Chen
        4. HBASE-21810.branch-1.2.001.patch
          3 kB
          Yechao Chen
        5. HBASE-21810.branch-1.2.002.patch
          5 kB
          Yechao Chen
        6. HBASE-21810.branch-1.2.002.patch
          5 kB
          Yechao Chen
        7. HBASE-21810.branch-1.2.003.patch
          5 kB
          Yechao Chen
        8. HBASE-21810.branch-2.001.patch
          3 kB
          Yechao Chen
        9. HBASE-21810.branch-2.002.patch
          5 kB
          Yechao Chen
        10. HBASE-21810.master.001.patch
          3 kB
          Guanghao Zhang
        11. HBASE-21810.master.001.patch
          3 kB
          Yechao Chen
        12. HBASE-21810.master.002.patch
          5 kB
          Yechao Chen
        13. HBASE-21810.master.003.patch
          5 kB
          Yechao Chen
        14. HBASE-21810.master.003.patch
          5 kB
          Yechao Chen

          Activity

            People

            • Assignee:
              chenyechao Yechao Chen
              Reporter:
              chenyechao Yechao Chen

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment