Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1301

TestDebugScriptWithLinuxTaskController fails

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.21.0
    • Fix Version/s: 0.21.0
    • Component/s: test
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      After MAPREDUCE:879, TestDebugScriptWithLinuxTaskController fails with following exception :

      java.lang.NullPointerException
      at org.apache.hadoop.mapred.TestTaskTrackerLocalization.getFilePermissionAttrs(TestTaskTrackerLocalization.java:274)
      at org.apache.hadoop.mapred.TestTaskTrackerLocalization.checkFilePermissions(TestTaskTrackerLocalization.java:294)
      at org.apache.hadoop.mapred.TestDebugScript.verifyDebugScriptOutput(TestDebugScript.java:162)
      at org.apache.hadoop.mapred.TestDebugScriptWithLinuxTaskController.testDebugScriptExecutionAsDifferentUser(TestDebugScriptWithLinuxTaskController.java:50)

      1. patch-1301.txt
        0.7 kB
        Amareshwari Sriramadasu
      2. patch-1301-1.txt
        3 kB
        Amareshwari Sriramadasu

        Activity

        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #196 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/196/)

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #196 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/196/ )
        Hide
        Hemanth Yamijala added a comment -

        I just committed this to trunk and branch 0.21. Thanks, Amareshwari !

        Show
        Hemanth Yamijala added a comment - I just committed this to trunk and branch 0.21. Thanks, Amareshwari !
        Hide
        Hemanth Yamijala added a comment -

        +1.

        Show
        Hemanth Yamijala added a comment - +1.
        Hide
        Amareshwari Sriramadasu added a comment -

        -1 contrib tests.

        Is due to MAPREDUCE-1311 and MAPREDUCE-1312

        Show
        Amareshwari Sriramadasu added a comment - -1 contrib tests. Is due to MAPREDUCE-1311 and MAPREDUCE-1312
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12428695/patch-1301-1.txt
        against trunk revision 893055.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 6 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/233/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/233/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/233/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/233/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12428695/patch-1301-1.txt against trunk revision 893055. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/233/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/233/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/233/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/233/console This message is automatically generated.
        Hide
        Amareshwari Sriramadasu added a comment -

        I ran all the LinuxTaskController tests. Ran TestTrackerDistributedCacheManagerWithLinuxTaskController after applying the available patch on MAPREDUCE-1186. All of them passed except TestStreamingAsDifferentUser (due to MAPREDUCE-1322).

        Show
        Amareshwari Sriramadasu added a comment - I ran all the LinuxTaskController tests. Ran TestTrackerDistributedCacheManagerWithLinuxTaskController after applying the available patch on MAPREDUCE-1186 . All of them passed except TestStreamingAsDifferentUser (due to MAPREDUCE-1322 ).
        Hide
        Amareshwari Sriramadasu added a comment -

        Patch making "setting up permission dir" a static block

        Show
        Amareshwari Sriramadasu added a comment - Patch making "setting up permission dir" a static block
        Hide
        Amareshwari Sriramadasu added a comment -

        Shall we put setting up permission directory and files in a static block to achieve the above, instead of doing it in a method?

        Show
        Amareshwari Sriramadasu added a comment - Shall we put setting up permission directory and files in a static block to achieve the above, instead of doing it in a method?
        Hide
        Hemanth Yamijala added a comment -

        I tried to see why we failed to catch this in MAPREDUCE-879. The core issue here is that there are two APIs in TestTaskTrackerLocalization - checkFilePermissions and setupPermissionScriptDir(File) which are related to each other. I.e. the former requires the latter to have been called. But they are static and thus hide this dependency. In the review, it didn't occur to me that I should check for callers of checkFilePermissions and verify this dependency is satisfied by other callers (like TestDebugScriptWithLinuxTaskController), as the API was not touched in the patch.

        As a result of that, I am tending to think that others will make the same mistake in future as well. I.e, they may think there's no dependency between these APIs and call only checkFilePermissions without calling setupPermissionScriptDir.

        So, though the attached patch actually fixes the issue for this test case, I would like to see an approach that goes beyond the fix and prevents these problems in future as well. Makes sense ?

        Show
        Hemanth Yamijala added a comment - I tried to see why we failed to catch this in MAPREDUCE-879 . The core issue here is that there are two APIs in TestTaskTrackerLocalization - checkFilePermissions and setupPermissionScriptDir(File) which are related to each other. I.e. the former requires the latter to have been called. But they are static and thus hide this dependency. In the review, it didn't occur to me that I should check for callers of checkFilePermissions and verify this dependency is satisfied by other callers (like TestDebugScriptWithLinuxTaskController), as the API was not touched in the patch. As a result of that, I am tending to think that others will make the same mistake in future as well. I.e, they may think there's no dependency between these APIs and call only checkFilePermissions without calling setupPermissionScriptDir. So, though the attached patch actually fixes the issue for this test case, I would like to see an approach that goes beyond the fix and prevents these problems in future as well. Makes sense ?
        Hide
        Sreekanth Ramakrishnan added a comment -

        The changes look fine to me.

        +1 to the patch.

        Show
        Sreekanth Ramakrishnan added a comment - The changes look fine to me. +1 to the patch.
        Hide
        Amareshwari Sriramadasu added a comment -

        Test failures are not related to the patch

        Show
        Amareshwari Sriramadasu added a comment - Test failures are not related to the patch
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12428159/patch-1301.txt
        against trunk revision 891111.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/204/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/204/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/204/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/204/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12428159/patch-1301.txt against trunk revision 891111. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/204/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/204/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/204/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/204/console This message is automatically generated.
        Hide
        Amareshwari Sriramadasu added a comment -

        Patch fixing the test failure.

        Show
        Amareshwari Sriramadasu added a comment - Patch fixing the test failure.

          People

          • Assignee:
            Amareshwari Sriramadasu
            Reporter:
            Amareshwari Sriramadasu
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development