Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1687

Stress submission policy does not always stress the cluster.

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: contrib/gridmix
    • Labels:
      None

      Description

      Currently, the rough idea of stress submission policy is to continue submitting jobs until the pending map tasks reach 2x of the cluster capacity. This proves to be inadequate and we saw a large job could monopolize the whole cluster.

        Issue Links

          Activity

          Hide
          Hong Tang added a comment -

          Preliminary patch that caps the maximum number of incomplete map tasks for each job in cluster overload calculation.

          Show
          Hong Tang added a comment - Preliminary patch that caps the maximum number of incomplete map tasks for each job in cluster overload calculation.
          Hide
          Hong Tang added a comment -

          Patch that fine tunes the overload determination heuristics.

          Show
          Hong Tang added a comment - Patch that fine tunes the overload determination heuristics.
          Hide
          Hong Tang added a comment -

          Fixed a minor bug in StressReaderThread.run() where the condition of the while loop waiting for cluster to be underloaded is incorrect.

          Show
          Hong Tang added a comment - Fixed a minor bug in StressReaderThread.run() where the condition of the while loop waiting for cluster to be underloaded is incorrect.
          Hide
          Amar Kamat added a comment -

          This got incorporated as part of our 0.20 to trunk forward-porting effort.

          Show
          Amar Kamat added a comment - This got incorporated as part of our 0.20 to trunk forward-porting effort.

            People

            • Assignee:
              Amar Kamat
              Reporter:
              Hong Tang
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development