HBase
  1. HBase
  2. HBASE-4907 Port 89-fb changes to trunk
  3. HBASE-4914

Enhance MapReduce TableInputFormat to Support N-mappers per Region

    Details

    • Type: Sub-task Sub-task
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Client, regionserver
    • Labels:
      None

      Description

      Current TableInputFormat based MR jobs create exactly one mapper per region where each mapper sets one Scan with appropriate start/stop row keys. This change allows jobs to be run with any number of mappers per region, so that when a mapper fails, there will be less data to be reprocessed.

        Issue Links

          Activity

          Hide
          Nicolas Spiegelberg added a comment -

          Need to port from 89-fb. See SVN #1181607

          Show
          Nicolas Spiegelberg added a comment - Need to port from 89-fb. See SVN #1181607
          Hide
          stack added a comment -

          Made blocker on 0.94.0 at Nicolas's suggestion

          Show
          stack added a comment - Made blocker on 0.94.0 at Nicolas's suggestion
          Hide
          Cosmin Lehene added a comment -

          Hadoop 0.20 doesn't behave well with large number of map tasks, so we implemented a N-Regions per map (through a splits_per_map property).

          I guess ideally we should be able to specify a min/max number of map tasks as well and have these two happen implicitly, perhaps with some sane thresholds.

          Show
          Cosmin Lehene added a comment - Hadoop 0.20 doesn't behave well with large number of map tasks, so we implemented a N-Regions per map (through a splits_per_map property). I guess ideally we should be able to specify a min/max number of map tasks as well and have these two happen implicitly, perhaps with some sane thresholds.
          Hide
          Lars Hofhansl added a comment -

          Again, this does not look like a 0.94 blocker.

          Show
          Lars Hofhansl added a comment - Again, this does not look like a 0.94 blocker.
          Hide
          Lars Hofhansl added a comment -

          Please comment today on why this is a blocker for 0.94. Otherwise I'll move this out of 0.94.

          Show
          Lars Hofhansl added a comment - Please comment today on why this is a blocker for 0.94. Otherwise I'll move this out of 0.94.
          Hide
          Lars Hofhansl added a comment -

          Moving out of 0.94. Pull back if you disagree.

          Show
          Lars Hofhansl added a comment - Moving out of 0.94. Pull back if you disagree.
          Hide
          Ted Yu added a comment -

          No assignee.
          Lowering priority

          Show
          Ted Yu added a comment - No assignee. Lowering priority
          Hide
          stack added a comment -

          Moving issue w/ no recent movement out of 0.95

          Show
          stack added a comment - Moving issue w/ no recent movement out of 0.95

            People

            • Assignee:
              Unassigned
              Reporter:
              Nicolas Spiegelberg
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:

                Development