Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-3996

Support multiple tables and scanners as input to the mapper in map/reduce jobs

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.94.5, 0.95.0
    • Component/s: mapreduce
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      Adds MultiTableInputFormat.

      Usage example:

      {code}
      Scan scan1 = new Scan();
      scan1.setStartRow(start1);
      scan1.setStopRow(end1);
      Scan scan2 = new Scan();
      scan2.setStartRow(start2);
      scan2.setStopRow(end2);
      MultiTableInputCollection mtic = new MultiTableInputCollection();
      mtic.Add(tableName1, scan1);
      mtic.Add(tableName2, scan2);
      TableMapReduceUtil.initTableMapperJob(mtic, TestTableMapper.class, Text.class, IntWritable.class, job1);
      {code}
      Show
      Adds MultiTableInputFormat. Usage example: {code} Scan scan1 = new Scan(); scan1.setStartRow(start1); scan1.setStopRow(end1); Scan scan2 = new Scan(); scan2.setStartRow(start2); scan2.setStopRow(end2); MultiTableInputCollection mtic = new MultiTableInputCollection(); mtic.Add(tableName1, scan1); mtic.Add(tableName2, scan2); TableMapReduceUtil.initTableMapperJob(mtic, TestTableMapper.class, Text.class, IntWritable.class, job1); {code}

      Description

      It seems that in many cases feeding data from multiple tables or multiple scanners on a single table can save a lot of time when running map/reduce jobs.
      I propose a new MultiTableInputFormat class that would allow doing this.

        Attachments

        1. 3996-v15.txt
          34 kB
          Lars Hofhansl
        2. 3996-0.94.txt
          33 kB
          Lars Hofhansl
        3. 3996-v14.txt
          33 kB
          Bryan Baugher
        4. 3996-v13.txt
          33 kB
          Bryan Baugher
        5. 3996-v12.txt
          33 kB
          Bryan Baugher
        6. 3996-v11.txt
          33 kB
          Bryan Baugher
        7. 3996-v10.txt
          33 kB
          Bryan Baugher
        8. 3996-v9.txt
          32 kB
          Bryan Baugher
        9. 3996-v8.txt
          32 kB
          Bryan Baugher
        10. 3996-v7.txt
          42 kB
          Ted Yu
        11. 3996-v6.txt
          40 kB
          Ted Yu
        12. 3996-v5.txt
          40 kB
          Eran Kutner
        13. 3996-v4.txt
          40 kB
          Ted Yu
        14. 3996-v3.txt
          42 kB
          Ted Yu
        15. 3996-v2.txt
          43 kB
          Ted Yu
        16. HBase-3996.patch
          66 kB
          Eran Kutner

          Issue Links

            Activity

              People

              • Assignee:
                bbaugher Bryan Baugher
                Reporter:
                erank Eran Kutner
              • Votes:
                10 Vote for this issue
                Watchers:
                14 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: