Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-628

TestJobInProgress brings up MinMR/DFS clusters for every test

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      TestJobInProgress brings up MiniMR clusters in setUp and brings it down in tearDown methods. Since these methods are called before each test, the test brings up/down the cluster several times. Instead, the cluster should just be brought up once, all tests run and then brought down

      1. hadoop-5970.patch
        7 kB
        Jothi Padmanabhan
      2. mapred-628.patch
        13 kB
        Jothi Padmanabhan
      3. mapred-628-v1.patch
        20 kB
        Jothi Padmanabhan

        Issue Links

          Activity

          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Patch Available Patch Available Open Open
          25d 22h 53m 2 Jothi Padmanabhan 09/Jul/09 11:34
          Open Open Patch Available Patch Available
          28d 10h 24m 3 Jothi Padmanabhan 28/Jul/09 16:19
          Patch Available Patch Available Resolved Resolved
          18h 48m 1 Devaraj Das 29/Jul/09 11:07
          Resolved Resolved Closed Closed
          391d 11h 6m 1 Tom White 24/Aug/10 22:13
          Tom White made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Hide
          Jothi Padmanabhan added a comment -

          Modifes TestJobInProgress to use Fake Objects to test the functionality instead of running a full fledged job using the MiniMR clusters.

          Show
          Jothi Padmanabhan added a comment - Modifes TestJobInProgress to use Fake Objects to test the functionality instead of running a full fledged job using the MiniMR clusters.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #33 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/33/)
          . Improves the execution time of TestJobInProgress. Contributed by Jothi Padmanabhan.

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #33 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/33/ ) . Improves the execution time of TestJobInProgress. Contributed by Jothi Padmanabhan.
          Devaraj Das made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags [Reviewed]
          Fix Version/s 0.21.0 [ 12314045 ]
          Resolution Fixed [ 1 ]
          Hide
          Devaraj Das added a comment -

          I just committed this. Thanks, Jothi!

          Show
          Devaraj Das added a comment - I just committed this. Thanks, Jothi!
          Hide
          Jothi Padmanabhan added a comment -

          Test Patch and all mapred tests passed with this patch

          Show
          Jothi Padmanabhan added a comment - Test Patch and all mapred tests passed with this patch
          Jothi Padmanabhan made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Jothi Padmanabhan made changes -
          Attachment mapred-628-v1.patch [ 12414749 ]
          Hide
          Jothi Padmanabhan added a comment -

          Made this test as an unit test.
          This test creates data-local, rack-local and non-local maps and tests the internal data structures as the original test, but instead using the mock objects.
          As a part of this test, also refactored the initTasks of FakeJobInProgress.initTasks to mimic initTasks of JobInProgress.

          Show
          Jothi Padmanabhan added a comment - Made this test as an unit test. This test creates data-local, rack-local and non-local maps and tests the internal data structures as the original test, but instead using the mock objects. As a part of this test, also refactored the initTasks of FakeJobInProgress.initTasks to mimic initTasks of JobInProgress.
          Jothi Padmanabhan made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12412480/mapred-628.patch
          against trunk revision 790869.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/346/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/346/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/346/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/346/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12412480/mapred-628.patch against trunk revision 790869. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/346/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/346/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/346/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/346/console This message is automatically generated.
          Jothi Padmanabhan made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Jothi Padmanabhan made changes -
          Attachment mapred-628.patch [ 12412480 ]
          Hide
          Jothi Padmanabhan added a comment -

          Patch that changes this test into a unit test

          Show
          Jothi Padmanabhan added a comment - Patch that changes this test into a unit test
          Chris Douglas made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Chris Douglas added a comment -

          Good catch.

          UtilsForTests::signalTasks(FileSystem, boolean, String, String) added in this patch has no callers...

          Show
          Chris Douglas added a comment - Good catch. UtilsForTests::signalTasks(FileSystem, boolean, String, String) added in this patch has no callers...
          Owen O'Malley made changes -
          Project Hadoop Common [ 12310240 ] Hadoop Map/Reduce [ 12310941 ]
          Key HADOOP-5970 MAPREDUCE-628
          Issue Type Test [ 6 ] Bug [ 1 ]
          Component/s mapred [ 12310690 ]
          Component/s test [ 12311440 ]
          Fix Version/s 0.21.0 [ 12313563 ]
          Jothi Padmanabhan made changes -
          Component/s mapred [ 12310690 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12409855/hadoop-5970.patch
          against trunk revision 782083.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/475/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/475/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/475/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/475/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12409855/hadoop-5970.patch against trunk revision 782083. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 Eclipse classpath. The patch retains Eclipse classpath integrity. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/475/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/475/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/475/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/475/console This message is automatically generated.
          Jothi Padmanabhan made changes -
          Link This issue is cloned as HADOOP-5972 [ HADOOP-5972 ]
          Jothi Padmanabhan made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Jothi Padmanabhan made changes -
          Field Original Value New Value
          Attachment hadoop-5970.patch [ 12409855 ]
          Hide
          Jothi Padmanabhan added a comment -

          Attached is a path that does
          1. Bring up/down the MiniMR Cluster only once
          2. Change to use MiniMR with localFS instead of DFS

          The run time of the test came down by about 100 seconds with these changes

          Show
          Jothi Padmanabhan added a comment - Attached is a path that does 1. Bring up/down the MiniMR Cluster only once 2. Change to use MiniMR with localFS instead of DFS The run time of the test came down by about 100 seconds with these changes
          Jothi Padmanabhan created issue -

            People

            • Assignee:
              Jothi Padmanabhan
              Reporter:
              Jothi Padmanabhan
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development