[HBASE-18090] Improve TableSnapshotInputFormat to allow more multiple mappers per region - ASF JIRA

Voters

Watch issue

Watchers

Create sub-task

Link

Clone

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: Improvement
Status: Resolved
Priority: Major
Resolution: Fixed
Affects Version/s: None
Fix Version/s: 1.4.0, 1.3.2, 2.0.0-beta-1, 2.0.0
Component/s: mapreduce
Labels:
None

Hadoop Flags:

Reviewed
Release Note:

Hide
In this task, we make it possible to run multiple mappers per region in the table snapshot. The following code is primary table snapshot mapper initializatio:

TableMapReduceUtil.initTableSnapshotMapperJob(
          snapshotName, // The name of the snapshot (of a table) to read from
          scan, // Scan instance to control CF and attribute selection
          mapper, // mapper
          outputKeyClass, // mapper output key
          outputValueClass, // mapper output value
          job, // The current job to adjust
          true, // upload HBase jars and jars for any of the configured job classes via the distributed cache (tmpjars)
          restoreDir, // a temporary directory to copy the snapshot files into
);

The job only run one map task per region in the table snapshot. With this feature, client can specify the desired num of mappers when init table snapshot mapper job：

TableMapReduceUtil.initTableSnapshotMapperJob(
          snapshotName, // The name of the snapshot (of a table) to read from
          scan, // Scan instance to control CF and attribute selection
          mapper, // mapper
          outputKeyClass, // mapper output key
          outputValueClass, // mapper output value
          job, // The current job to adjust
          true, // upload HBase jars and jars for any of the configured job classes via the distributed cache (tmpjars)
          restoreDir, // a temporary directory to copy the snapshot files into
          splitAlgorithm, // splitAlgo algorithm to split, current split algorithms support RegionSplitter.UniformSplit() and RegionSplitter.HexStringSplit()
          n // how many input splits to generate per one region
);

Show
In this task, we make it possible to run multiple mappers per region in the table snapshot. The following code is primary table snapshot mapper initializatio: TableMapReduceUtil.initTableSnapshotMapperJob(           snapshotName, // The name of the snapshot (of a table) to read from           scan, // Scan instance to control CF and attribute selection           mapper, // mapper           outputKeyClass, // mapper output key           outputValueClass, // mapper output value           job, // The current job to adjust           true, // upload HBase jars and jars for any of the configured job classes via the distributed cache (tmpjars)           restoreDir, // a temporary directory to copy the snapshot files into ); The job only run one map task per region in the table snapshot. With this feature, client can specify the desired num of mappers when init table snapshot mapper job： TableMapReduceUtil.initTableSnapshotMapperJob(           snapshotName, // The name of the snapshot (of a table) to read from           scan, // Scan instance to control CF and attribute selection           mapper, // mapper           outputKeyClass, // mapper output key           outputValueClass, // mapper output value           job, // The current job to adjust           true, // upload HBase jars and jars for any of the configured job classes via the distributed cache (tmpjars)           restoreDir, // a temporary directory to copy the snapshot files into           splitAlgorithm, // splitAlgo algorithm to split, current split algorithms support RegionSplitter.UniformSplit() and RegionSplitter.HexStringSplit()           n // how many input splits to generate per one region );

Description

TableSnapshotInputFormat runs one map task per region in the table snapshot. This places unnecessary restriction that the region layout of the original table needs to take the processing resources available to MR job into consideration. Allowing to run multiple mappers per region (assuming reasonably even key distribution) would be useful.