Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-379

Implement a Map-Reduce application which can be used to reliably launch speculative tasks

    Details

    • Type: Test Test
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      It would be very useful to have a reliable test case to help launch speculative tasks (maps and/or reduces) to help debug problems or fine-tune strategies for speculative execution.

      I propose we implement some along the lines of the SmallJobsBenchmark and put it in the hadoop-test.jar. I imagine it could be used like:

      $ bin/hadoop jar hadoop-${version}-test.jar speculativejob -nMaps 1000 -nReduces 50 -specMaps 10 -specReduces 5
      

      The application should be minutely aware of the current strategies for launching speculative tasks, and throttle requisite no. of maps/reduces to ensure that the speculative tasks are launched in a reliable and repeatable manner.

      Thoughts?

        Issue Links

          Activity

          Hide
          Augusto Souza added a comment -

          Hello, is this still relevant? I would like to help with it, can someone point me to a starting point?

          Show
          Augusto Souza added a comment - Hello, is this still relevant? I would like to help with it, can someone point me to a starting point?
          Hide
          Arun C Murthy added a comment -

          I'm moving this to 0.17.0.

          Show
          Arun C Murthy added a comment - I'm moving this to 0.17.0.
          Hide
          Doug Cutting added a comment -

          +1 This would be great to have. It would be nice to have a simple version that runs in tens of seconds that could be run as a unit test too.

          Show
          Doug Cutting added a comment - +1 This would be great to have. It would be nice to have a simple version that runs in tens of seconds that could be run as a unit test too.

            People

            • Assignee:
              Unassigned
              Reporter:
              Arun C Murthy
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:

                Development