[CRUNCH-545] Writing to HFiles starts a job per column family

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.13.0
• Assignee: Gabriel Reid
• Reporter: Gabriel Reid
• Created: 19/Jul/15 14:12
• Updated: 06/Aug/15 04:39
• Resolved: 20/Jul/15 13:27

Description

When writing to HFiles via HFileUtils.writeToHFilesForIncrementalLoad, a separate MR job is started up per column family defined for the table, regardless of whether or not there is any data for each of these column families.

Each of the column family jobs runs over the full set of Cells, filters for the desired column family, and then partitions the data.

For tables with multiple column families, it would be a lot more efficient to sort/partition all of the data together, and then split it out per column family afterwards.