[HBASE-3996] Support multiple tables and scanners as input to the mapper in map/reduce jobs - ASF JIRA

Voters

Watch issue

Watchers

Create sub-task

Link

Clone

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: Improvement
Status: Closed
Priority: Critical
Resolution: Fixed
Affects Version/s: None
Fix Version/s: 0.94.5, 0.95.0
Component/s: mapreduce
Labels:
None

Hadoop Flags:

Reviewed
Release Note:

Hide
Adds MultiTableInputFormat.

Usage example:

{code}
Scan scan1 = new Scan();
scan1.setStartRow(start1);
scan1.setStopRow(end1);
Scan scan2 = new Scan();
scan2.setStartRow(start2);
scan2.setStopRow(end2);
MultiTableInputCollection mtic = new MultiTableInputCollection();
mtic.Add(tableName1, scan1);
mtic.Add(tableName2, scan2);
TableMapReduceUtil.initTableMapperJob(mtic, TestTableMapper.class, Text.class, IntWritable.class, job1);
{code}

Show
Adds MultiTableInputFormat. Usage example: {code} Scan scan1 = new Scan(); scan1.setStartRow(start1); scan1.setStopRow(end1); Scan scan2 = new Scan(); scan2.setStartRow(start2); scan2.setStopRow(end2); MultiTableInputCollection mtic = new MultiTableInputCollection(); mtic.Add(tableName1, scan1); mtic.Add(tableName2, scan2); TableMapReduceUtil.initTableMapperJob(mtic, TestTableMapper.class, Text.class, IntWritable.class, job1); {code}

Description

It seems that in many cases feeding data from multiple tables or multiple scanners on a single table can save a lot of time when running map/reduce jobs.
I propose a new MultiTableInputFormat class that would allow doing this.