Hadoop Common
  1. Hadoop Common
  2. HADOOP-5771

Create unit test for LinuxTaskController

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: security, test
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Added unit tests for verifying LinuxTaskController functionality.
    • Tags:
      ygridqa

      Description

      Add unit tests to test LinuxTaskController functionality introduced by HADOOP-4490

      1. 5771.20S.patch
        30 kB
        Ravi Gummadi
      2. HADOOP-5771-1.patch
        22 kB
        Sreekanth Ramakrishnan
      3. HADOOP-5771-2.patch
        23 kB
        Sreekanth Ramakrishnan
      4. HADOOP-5771-3.patch
        25 kB
        Sreekanth Ramakrishnan
      5. HADOOP-5771-4.txt
        27 kB
        Vinod Kumar Vavilapalli
      6. HADOOP-5771-5.patch
        26 kB
        Sreekanth Ramakrishnan
      7. hadoop-5771-ydist.patch
        30 kB
        Sreekanth Ramakrishnan
      8. hadoop-5771-ydist.patch
        31 kB
        Sreekanth Ramakrishnan

        Activity

        Sreekanth Ramakrishnan created issue -
        Hide
        Sreekanth Ramakrishnan added a comment -

        Attaching patch which adds unit test for HADOOP-4490.

        Following is approach used in running unit tests. HADOOP-4490 unit tests require following extra parameter to be passed alongwith the ant test target.

        • taskcontroller-path - path to built task controller executable
        • taskcontroller-user - ugi of the user as who we should be executing the task.

        If any of the above two parameter is not passed the HADOOP-4490 test cases would not be executed and test case would be successful.

        An example for running a HADOOP-4490 test case is as follows

         ant test -Dtestcase=TestJobExecutionAsDifferentUser -Dtaskcontroller-path=/tmp/hadoop/bin -Dtaskcontroller-user=mapred,users
        

        For pipes related test cases you would have to pass compile.c++ flag.

        Following are the test cases added to the patch:

        • TestJobExecutionAsDifferentUser :- Core test case which launches a SleepJob and WordCount job as a different user.
        • TestKillSubProcessesWithTaskController :- Test case which checks if the process trees of launched tasks are cleaned up properly. Currently, fails with trunk when HADOOP-5420 is commited would work fine.
        • TestPipesAsDifferentUser :- Runs a pipes job as a different user.
        • TestStreamingAsDifferentUser :- Runs a streaming job as different user.
        Show
        Sreekanth Ramakrishnan added a comment - Attaching patch which adds unit test for HADOOP-4490 . Following is approach used in running unit tests. HADOOP-4490 unit tests require following extra parameter to be passed alongwith the ant test target. taskcontroller-path - path to built task controller executable taskcontroller-user - ugi of the user as who we should be executing the task. If any of the above two parameter is not passed the HADOOP-4490 test cases would not be executed and test case would be successful. An example for running a HADOOP-4490 test case is as follows ant test -Dtestcase=TestJobExecutionAsDifferentUser -Dtaskcontroller-path=/tmp/hadoop/bin -Dtaskcontroller-user=mapred,users For pipes related test cases you would have to pass compile.c++ flag. Following are the test cases added to the patch: TestJobExecutionAsDifferentUser :- Core test case which launches a SleepJob and WordCount job as a different user. TestKillSubProcessesWithTaskController :- Test case which checks if the process trees of launched tasks are cleaned up properly. Currently, fails with trunk when HADOOP-5420 is commited would work fine. TestPipesAsDifferentUser :- Runs a pipes job as a different user. TestStreamingAsDifferentUser :- Runs a streaming job as different user.
        Sreekanth Ramakrishnan made changes -
        Field Original Value New Value
        Attachment HADOOP-5771-1.patch [ 12407242 ]
        Hide
        Sreekanth Ramakrishnan added a comment -

        Attaching patch incorporating Vinod's offline comments:

        • Renamed MRTaskControllerCluster to ClusterWithLinuxTaskController
        • Added apache license header to all new test cases.
        • Cleaning up configuration file when test case completes.
        • Added documentation.
        Show
        Sreekanth Ramakrishnan added a comment - Attaching patch incorporating Vinod's offline comments: Renamed MRTaskControllerCluster to ClusterWithLinuxTaskController Added apache license header to all new test cases. Cleaning up configuration file when test case completes. Added documentation.
        Sreekanth Ramakrishnan made changes -
        Attachment HADOOP-5771-2.patch [ 12407353 ]
        Sreekanth Ramakrishnan made changes -
        Release Note  In order to run test cases utilizing LinuxTaskController please follow the steps mentioned in javadoc of ClusterWithLinuxTaskController base class which all test cases for LinuxTaskController extend.
        Hide
        Sreekanth Ramakrishnan added a comment -

        Attaching the new patch incorporating Vinod's offline comment.

        • Removed unused imports.
        • Added new assertion to check if the passed user actually ran the job.
        • Modified, TrApp to stop check if the local job runner and map output start, so that TrApp can be run on a MiniMRCluster
        • Changed documentation, added it to javadoc and release note of JIRA issue.
        Show
        Sreekanth Ramakrishnan added a comment - Attaching the new patch incorporating Vinod's offline comment. Removed unused imports. Added new assertion to check if the passed user actually ran the job. Modified, TrApp to stop check if the local job runner and map output start, so that TrApp can be run on a MiniMRCluster Changed documentation, added it to javadoc and release note of JIRA issue.
        Sreekanth Ramakrishnan made changes -
        Attachment HADOOP-5771-3.patch [ 12407450 ]
        Hide
        Vinod Kumar Vavilapalli added a comment -

        Patch looked good overall. Am uploading a new patch with slight modifications to the above patch with:

        • Better java code comments
        • Better code formatting and code-cleanup
        • And a small fix in TestStreamingAsDifferentUser because the earlier code was making the streaming jobs only run with LocalJobRunner.
        Show
        Vinod Kumar Vavilapalli added a comment - Patch looked good overall. Am uploading a new patch with slight modifications to the above patch with: Better java code comments Better code formatting and code-cleanup And a small fix in TestStreamingAsDifferentUser because the earlier code was making the streaming jobs only run with LocalJobRunner.
        Vinod Kumar Vavilapalli made changes -
        Attachment HADOOP-5771-4.txt [ 12407548 ]
        Vinod Kumar Vavilapalli made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Sreekanth Ramakrishnan added a comment -

        The changes to patch looks fine to me.

        +1 to patch.

        Show
        Sreekanth Ramakrishnan added a comment - The changes to patch looks fine to me. +1 to patch.
        Hide
        Sreekanth Ramakrishnan added a comment -

        Output from ant-test patch is:

             [exec]
             [exec] +1 overall.
             [exec]
             [exec]     +1 @author.  The patch does not contain any @author tags.
             [exec]
             [exec]     +1 tests included.  The patch appears to include 19 new or modified tests.
             [exec]
             [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
             [exec]
             [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
             [exec]
             [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
             [exec]
             [exec]     +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
             [exec]
             [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
        
        Show
        Sreekanth Ramakrishnan added a comment - Output from ant-test patch is: [exec] [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 19 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12407548/HADOOP-5771-4.txt
        against trunk revision 772960.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 19 new or modified tests.

        -1 patch. The patch command could not apply the patch.

        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/312/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12407548/HADOOP-5771-4.txt against trunk revision 772960. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 19 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/312/console This message is automatically generated.
        Sreekanth Ramakrishnan made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Hide
        Sreekanth Ramakrishnan added a comment -

        Attaching new patch merging with latest changes in trunk.

        Show
        Sreekanth Ramakrishnan added a comment - Attaching new patch merging with latest changes in trunk.
        Sreekanth Ramakrishnan made changes -
        Attachment HADOOP-5771-5.patch [ 12407731 ]
        Sreekanth Ramakrishnan made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12407731/HADOOP-5771-5.patch
        against trunk revision 772960.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 23 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/322/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/322/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/322/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/322/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12407731/HADOOP-5771-5.patch against trunk revision 772960. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 23 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 Eclipse classpath. The patch retains Eclipse classpath integrity. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/322/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/322/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/322/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/322/console This message is automatically generated.
        Hide
        Sreekanth Ramakrishnan added a comment -

        The test failure in capacity scheduler is not related to this patch.

        Show
        Sreekanth Ramakrishnan added a comment - The test failure in capacity scheduler is not related to this patch.
        Hide
        Hemanth Yamijala added a comment -

        I just committed this to trunk. Thanks, Sreekanth !

        Show
        Hemanth Yamijala added a comment - I just committed this to trunk. Thanks, Sreekanth !
        Hemanth Yamijala made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags [Reviewed]
        Resolution Fixed [ 1 ]
        Hide
        Hemanth Yamijala added a comment -

        Also, thanks to Vinod for review and modifications on the last patch.

        Show
        Hemanth Yamijala added a comment - Also, thanks to Vinod for review and modifications on the last patch.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-trunk #833 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/833/)
        . Implements unit tests for LinuxTaskController. Contributed by Sreekanth Ramakrishnan and Vinod Kumar Vavilapalli.

        Show
        Hudson added a comment - Integrated in Hadoop-trunk #833 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/833/ ) . Implements unit tests for LinuxTaskController. Contributed by Sreekanth Ramakrishnan and Vinod Kumar Vavilapalli.
        Nigel Daley made changes -
        Fix Version/s 0.21.0 [ 12313563 ]
        Nigel Daley made changes -
        Tags ygridqa
        Hide
        Sreekanth Ramakrishnan added a comment -

        Y! distribution patch.

        Show
        Sreekanth Ramakrishnan added a comment - Y! distribution patch.
        Sreekanth Ramakrishnan made changes -
        Attachment hadoop-5771-ydist.patch [ 12413079 ]
        Hide
        Robert Chansler added a comment -

        Editorial pass over all release notes prior to publication of 0.21. Routine.

        Show
        Robert Chansler added a comment - Editorial pass over all release notes prior to publication of 0.21. Routine.
        Robert Chansler made changes -
        Release Note  In order to run test cases utilizing LinuxTaskController please follow the steps mentioned in javadoc of ClusterWithLinuxTaskController base class which all test cases for LinuxTaskController extend.
        Hide
        Sreekanth Ramakrishnan added a comment -

        Attaching latest ydist patch with just testcases merged.

        Show
        Sreekanth Ramakrishnan added a comment - Attaching latest ydist patch with just testcases merged.
        Sreekanth Ramakrishnan made changes -
        Attachment hadoop-5771-ydist.patch [ 12426091 ]
        Ravi Gummadi made changes -
        Release Note Adds unit test for LinuxTaskController
        Hide
        Ravi Gummadi added a comment -

        Attaching patch for yahoo hadoop 20.

        Show
        Ravi Gummadi added a comment - Attaching patch for yahoo hadoop 20.
        Ravi Gummadi made changes -
        Attachment 5771.20S.patch [ 12429998 ]
        Vinod Kumar Vavilapalli made changes -
        Release Note Adds unit test for LinuxTaskController Added unit tests for verifying LinuxTaskController functionality.
        Component/s security [ 12312526 ]
        Component/s test [ 12311440 ]
        Tom White made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Sreekanth Ramakrishnan
            Reporter:
            Sreekanth Ramakrishnan
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development