Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.21.0, 0.22.0
    • Fix Version/s: 0.22.0
    • Component/s: contrib/mumak
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Tags:
      mumak capacity-scheduler

      Description

      In order to make the capacity-scheduler work in the mumak simulation environment, we have to replace the job-initialization threads of the capacity scheduler with classes that perform event-based initialization. We propose to use aspectj to disable the threads of the JobInitializationPoller class used by the Capacity Scheduler, and then perform the corresponding initialization tasks through a simulation job-initialization class that receives periodic wake-up calls from the simulator engine.

        Issue Links

          Activity

          Hide
          Mahadev konar added a comment -

          +1 for the patch. I just committed this. ant test for mumak pass.

          Show
          Mahadev konar added a comment - +1 for the patch. I just committed this. ant test for mumak pass.
          Hide
          Hong Tang added a comment -

          I have reviewed Anirban's earlier and I forgot to comment with +1.

          Show
          Hong Tang added a comment - I have reviewed Anirban's earlier and I forgot to comment with +1.
          Hide
          Hong Tang added a comment -

          Both test-patch and mumak unit tests pass on my local machine (I forgot to copy the console output of test-patch results).

          test:
               [echo] contrib: mumak
             [delete] Deleting directory /grid/0/htang/hadoop-mapreduce/build/contrib/mumak/test/logs
              [mkdir] Created dir: /grid/0/htang/hadoop-mapreduce/build/contrib/mumak/test/logs
              [junit] WARNING: multiple versions of ant detected in path for junit
              [junit]          jar:file:/homes/htang/installed/ant/lib/ant.jar!/org/apache/tools/ant/Project.class
              [junit]      and jar:file:/homes/htang/.ivy2/cache/ant/ant/jars/ant-1.6.5.jar!/org/apache/tools/ant/Project.class
              [junit] Running org.apache.hadoop.mapred.TestRemoveIpsFromLoggedNetworkTopology
              [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 0.586 sec
              [junit] Running org.apache.hadoop.mapred.TestSimulatorDeterministicReplay
              [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 38.469 sec
              [junit] Running org.apache.hadoop.mapred.TestSimulatorEndToEnd
              [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 107.055 sec
              [junit] Running org.apache.hadoop.mapred.TestSimulatorEngine
              [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 0.162 sec
              [junit] Running org.apache.hadoop.mapred.TestSimulatorEventQueue
              [junit] Tests run: 4, Failures: 0, Errors: 0, Time elapsed: 0.02 sec
              [junit] Running org.apache.hadoop.mapred.TestSimulatorJobClient
              [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.063 sec
              [junit] Running org.apache.hadoop.mapred.TestSimulatorJobTracker
              [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 2.123 sec
              [junit] Running org.apache.hadoop.mapred.TestSimulatorSerialJobSubmission
              [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 139.603 sec
              [junit] Running org.apache.hadoop.mapred.TestSimulatorStressJobSubmission
              [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 100.492 sec
              [junit] Running org.apache.hadoop.mapred.TestSimulatorTaskTracker
              [junit] Tests run: 11, Failures: 0, Errors: 0, Time elapsed: 0.397 sec
          
          Show
          Hong Tang added a comment - Both test-patch and mumak unit tests pass on my local machine (I forgot to copy the console output of test-patch results). test: [echo] contrib: mumak [delete] Deleting directory /grid/0/htang/hadoop-mapreduce/build/contrib/mumak/test/logs [mkdir] Created dir: /grid/0/htang/hadoop-mapreduce/build/contrib/mumak/test/logs [junit] WARNING: multiple versions of ant detected in path for junit [junit] jar:file:/homes/htang/installed/ant/lib/ant.jar!/org/apache/tools/ant/Project.class [junit] and jar:file:/homes/htang/.ivy2/cache/ant/ant/jars/ant-1.6.5.jar!/org/apache/tools/ant/Project.class [junit] Running org.apache.hadoop.mapred.TestRemoveIpsFromLoggedNetworkTopology [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 0.586 sec [junit] Running org.apache.hadoop.mapred.TestSimulatorDeterministicReplay [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 38.469 sec [junit] Running org.apache.hadoop.mapred.TestSimulatorEndToEnd [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 107.055 sec [junit] Running org.apache.hadoop.mapred.TestSimulatorEngine [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 0.162 sec [junit] Running org.apache.hadoop.mapred.TestSimulatorEventQueue [junit] Tests run: 4, Failures: 0, Errors: 0, Time elapsed: 0.02 sec [junit] Running org.apache.hadoop.mapred.TestSimulatorJobClient [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.063 sec [junit] Running org.apache.hadoop.mapred.TestSimulatorJobTracker [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 2.123 sec [junit] Running org.apache.hadoop.mapred.TestSimulatorSerialJobSubmission [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 139.603 sec [junit] Running org.apache.hadoop.mapred.TestSimulatorStressJobSubmission [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 100.492 sec [junit] Running org.apache.hadoop.mapred.TestSimulatorTaskTracker [junit] Tests run: 11, Failures: 0, Errors: 0, Time elapsed: 0.397 sec
          Hide
          Hong Tang added a comment -

          Fixed a findbugs warning.

          Show
          Hong Tang added a comment - Fixed a findbugs warning.
          Hide
          Hong Tang added a comment -

          All mumak unit tests passed on my local machine after applying patch mr-1834-20100727.patch from MAPREDUCE-1834.

          Show
          Hong Tang added a comment - All mumak unit tests passed on my local machine after applying patch mr-1834-20100727.patch from MAPREDUCE-1834 .
          Hide
          Anirban Dasgupta added a comment -

          I am attaching a partial patch at this point. In offline testing I am observing some javadoc and findbugs warnings. It seems that the main build.xml has to be changed to remove these.

          Show
          Anirban Dasgupta added a comment - I am attaching a partial patch at this point. In offline testing I am observing some javadoc and findbugs warnings. It seems that the main build.xml has to be changed to remove these.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          No idea how the Mumak test code is written, but various test-cases in TestCapacityScheduler already replace the initialization thread with a ControlledInitializationPoller for the sake of deterministic testing. Please see testJobInitialization() if it helps.

          Show
          Vinod Kumar Vavilapalli added a comment - No idea how the Mumak test code is written, but various test-cases in TestCapacityScheduler already replace the initialization thread with a ControlledInitializationPoller for the sake of deterministic testing. Please see testJobInitialization() if it helps.

            People

            • Assignee:
              Anirban Dasgupta
              Reporter:
              Anirban Dasgupta
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - 672h
                672h
                Remaining:
                Remaining Estimate - 672h
                672h
                Logged:
                Time Spent - Not Specified
                Not Specified

                  Development