[HBASE-5140] TableInputFormat subclass to allow N number of splits per region during MR jobs

• Type: New Feature
• Status: Closed
• Priority: Trivial
• Resolution: Won't Fix
• Affects Version/s: 0.90.4
• Component/s: mapreduce
• Labels: mapreduce split
• Release Note: Used the 0.90 branch for the patch but code looks compatible in trunk as well (with one deprecated method)
• Tags: mapreduce splits tableinputformat
• Assignee: Unassigned
• Reporter: Josh Wymer
• Created: 06/Jan/12 18:59
• Updated: 12/Jun/22 20:03
• Resolved: 08/Jun/14 21:40

Description

In regards to HBASE-5138 I am working on a patch for the TableInputFormat class that overrides getSplits in order to generate N number of splits per regions and/or N number of splits per job. The idea is to convert the startKey and endKey for each region from byte[] to BigDecimal, take the difference, divide by N, convert back to byte[] and generate splits on the resulting values. Assuming your keys are fully distributed this should generate splits at nearly the same number of rows per split. Any suggestions on this issue are welcome.

Issue Links

• relates to: HBASE-4063 Improve TableInputFormat to allow application to configure the number of mappers