[HBASE-12596] bulkload needs to follow locality

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.98.8
• Fix Version/s: 0.98.14, 1.3.0, 2.0.0
• Component/s: HFile, regionserver

Environment

hadoop-2.3.0, hbase-0.98.8, jdk1.7

• Hadoop Flags: Reviewed
• Assignee: Victor Xu
• Reporter: Victor Xu
• Created: 27/Nov/14 23:01
• Updated: 17/Jun/22 18:36
• Resolved: 15/Jul/15 00:03

Description

Normally, we have 2 steps to perform a bulkload: 1. use a job to write HFiles to be loaded; 2. Move these HFiles to the right hdfs directory. However, the locality could be loss during the first step. Why not just write the HFiles directly into the right place? We can do this easily because StoreFile.WriterBuilder has the "withFavoredNodes" method, and we just need to call it in HFileOutputFormat's getNewWriter().
This feature is enabled by default, and we could use 'hbase.bulkload.locality.sensitive.enabled=false' to disable it.