HBase
  1. HBase
  2. HBASE-3996

Support multiple tables and scanners as input to the mapper in map/reduce jobs

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.94.5, 0.95.0
    • Component/s: mapreduce
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      Adds MultiTableInputFormat.

      Usage example:

      {code}
      Scan scan1 = new Scan();
      scan1.setStartRow(start1);
      scan1.setStopRow(end1);
      Scan scan2 = new Scan();
      scan2.setStartRow(start2);
      scan2.setStopRow(end2);
      MultiTableInputCollection mtic = new MultiTableInputCollection();
      mtic.Add(tableName1, scan1);
      mtic.Add(tableName2, scan2);
      TableMapReduceUtil.initTableMapperJob(mtic, TestTableMapper.class, Text.class, IntWritable.class, job1);
      {code}
      Show
      Adds MultiTableInputFormat. Usage example: {code} Scan scan1 = new Scan(); scan1.setStartRow(start1); scan1.setStopRow(end1); Scan scan2 = new Scan(); scan2.setStartRow(start2); scan2.setStopRow(end2); MultiTableInputCollection mtic = new MultiTableInputCollection(); mtic.Add(tableName1, scan1); mtic.Add(tableName2, scan2); TableMapReduceUtil.initTableMapperJob(mtic, TestTableMapper.class, Text.class, IntWritable.class, job1); {code}

      Description

      It seems that in many cases feeding data from multiple tables or multiple scanners on a single table can save a lot of time when running map/reduce jobs.
      I propose a new MultiTableInputFormat class that would allow doing this.

      1. HBase-3996.patch
        66 kB
        Eran Kutner
      2. 3996-v2.txt
        43 kB
        Ted Yu
      3. 3996-v3.txt
        42 kB
        Ted Yu
      4. 3996-v4.txt
        40 kB
        Ted Yu
      5. 3996-v5.txt
        40 kB
        Eran Kutner
      6. 3996-v6.txt
        40 kB
        Ted Yu
      7. 3996-v7.txt
        42 kB
        Ted Yu
      8. 3996-v8.txt
        32 kB
        Bryan Baugher
      9. 3996-v9.txt
        32 kB
        Bryan Baugher
      10. 3996-v10.txt
        33 kB
        Bryan Baugher
      11. 3996-v11.txt
        33 kB
        Bryan Baugher
      12. 3996-v12.txt
        33 kB
        Bryan Baugher
      13. 3996-v13.txt
        33 kB
        Bryan Baugher
      14. 3996-v14.txt
        33 kB
        Bryan Baugher
      15. 3996-0.94.txt
        33 kB
        Lars Hofhansl
      16. 3996-v15.txt
        34 kB
        Lars Hofhansl

        Issue Links

          Activity

            People

            • Assignee:
              Bryan Baugher
              Reporter:
              Eran Kutner
            • Votes:
              10 Vote for this issue
              Watchers:
              15 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development