[HBASE-21810] bulkload support set hfile compression on client

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.3.3, 1.4.9, 2.0.4, 2.1.3, 1.2.11
• Fix Version/s: 3.0.0-alpha-1, 1.5.0, 2.2.0, 2.3.0
• Component/s: mapreduce

Release Note

bulkload (HFileOutputFormat2) support config the compression on client ,you can set the job configuration "hbase.mapreduce.hfileoutputformat.compression" override the auto-detection of the target table's compression

• Assignee: Yechao Chen
• Reporter: Yechao Chen
• Created: 30/Jan/19 09:18
• Updated: 13/Mar/19 01:50
• Resolved: 12/Mar/19 10:31

Description

hbase bulkload (HFileOutputFormat2) generate hfile ,the compression from the table(cf) compression,

if the compression can be set on client ,sometimes,it's useful,

some case in our production:

1、hfile bulkload replication between the data center with bandwidth limit, we can set the compression of the bulkload hfile not changing the table compression

2、bulkload hfile not set  compression ,but the table compression is gz/zstd/snappy... ,can reduce the hfile created time and compaction will make the hfile to compression finally

3、somethings the yarn nodes (hfile created by reduce) /dobulkload client has no compression lib,but the hbase cluster has,it's useful for this case