Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-379

Implement a Map-Reduce application which can be used to reliably launch speculative tasks

    Details

    • Type: Test Test
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      It would be very useful to have a reliable test case to help launch speculative tasks (maps and/or reduces) to help debug problems or fine-tune strategies for speculative execution.

      I propose we implement some along the lines of the SmallJobsBenchmark and put it in the hadoop-test.jar. I imagine it could be used like:

      $ bin/hadoop jar hadoop-${version}-test.jar speculativejob -nMaps 1000 -nReduces 50 -specMaps 10 -specReduces 5
      

      The application should be minutely aware of the current strategies for launching speculative tasks, and throttle requisite no. of maps/reduces to ensure that the speculative tasks are launched in a reliable and repeatable manner.

      Thoughts?

        Issue Links

          Activity

          aditya sundar made changes -
          Link This issue is cloned as MAPREDUCE-2270 [ MAPREDUCE-2270 ]
          Owen O'Malley made changes -
          Project Hadoop Common [ 12310240 ] Hadoop Map/Reduce [ 12310941 ]
          Key HADOOP-2214 MAPREDUCE-379
          Component/s mapred [ 12310690 ]
          Component/s test [ 12311440 ]
          Robert Chansler made changes -
          Fix Version/s 0.17.0 [ 12312913 ]
          Arun C Murthy made changes -
          Field Original Value New Value
          Fix Version/s 0.16.0 [ 12312740 ]
          Component/s mapred [ 12310690 ]
          Fix Version/s 0.17.0 [ 12312913 ]
          Arun C Murthy created issue -

            People

            • Assignee:
              Unassigned
              Reporter:
              Arun C Murthy
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:

                Development