Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1687

Stress submission policy does not always stress the cluster.

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: contrib/gridmix
    • Labels:
      None

      Description

      Currently, the rough idea of stress submission policy is to continue submitting jobs until the pending map tasks reach 2x of the cluster capacity. This proves to be inadequate and we saw a large job could monopolize the whole cluster.

        Issue Links

          Activity

          Hong Tang created issue -
          Hide
          Hong Tang added a comment -

          Preliminary patch that caps the maximum number of incomplete map tasks for each job in cluster overload calculation.

          Show
          Hong Tang added a comment - Preliminary patch that caps the maximum number of incomplete map tasks for each job in cluster overload calculation.
          Hong Tang made changes -
          Field Original Value New Value
          Attachment mr-1687-yhadoop-20.1xx-20100416.patch [ 12442026 ]
          Hide
          Hong Tang added a comment -

          Patch that fine tunes the overload determination heuristics.

          Show
          Hong Tang added a comment - Patch that fine tunes the overload determination heuristics.
          Hong Tang made changes -
          Attachment mr-1687-yhadoop-20.1xx-20100423.patch [ 12442653 ]
          Hide
          Hong Tang added a comment -

          Fixed a minor bug in StressReaderThread.run() where the condition of the while loop waiting for cluster to be underloaded is incorrect.

          Show
          Hong Tang added a comment - Fixed a minor bug in StressReaderThread.run() where the condition of the while loop waiting for cluster to be underloaded is incorrect.
          Hong Tang made changes -
          Attachment mr-1687-yhadoop-20.1xx-20100423-2.patch [ 12442692 ]
          Hong Tang made changes -
          Assignee Hong Tang [ hong.tang ]
          Hong Tang made changes -
          Assignee Hong Tang [ hong.tang ] rahul k singh [ rksingh ]
          Amar Kamat made changes -
          Link This issue is related to MAPREDUCE-3481 [ MAPREDUCE-3481 ]
          Amar Kamat made changes -
          Assignee rahul k singh [ rksingh ] Amar Kamat [ amar_kamat ]
          Hide
          Amar Kamat added a comment -

          This got incorporated as part of our 0.20 to trunk forward-porting effort.

          Show
          Amar Kamat added a comment - This got incorporated as part of our 0.20 to trunk forward-porting effort.
          Amar Kamat made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Amar Kamat made changes -
          Link This issue relates to MAPREDUCE-3769 [ MAPREDUCE-3769 ]
          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Resolved Resolved
          722d 21h 36m 1 Amar Kamat 01/Apr/12 06:00

            People

            • Assignee:
              Amar Kamat
              Reporter:
              Hong Tang
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development