Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-568

TestServiceLevelAuthorization fails on latest build in Hudson

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.21.0
    • Fix Version/s: 0.21.0
    • Component/s: test
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The latest build in Hudson of Hadoop-hdfs fails org.apache.hadoop.security.authorize.TestServiceLevelAuthorization.testServiceLevelAuthorization. (http://hudson.zones.apache.org/hudson/view/Hadoop/job/Hadoop-Hdfs-trunk/61/testReport/)

      1. patch-568.txt
        0.5 kB
        Amareshwari Sriramadasu

        Issue Links

          Activity

          gary murry created issue -
          Hide
          gary murry added a comment -

          Error Message

          org/apache/hadoop/examples/QuasiMonteCarlo

          Stacktrace

          java.lang.NoClassDefFoundError: org/apache/hadoop/examples/QuasiMonteCarlo
          at org.apache.hadoop.mapred.TestMiniMRWithDFS.runPI(TestMiniMRWithDFS.java:176)
          at org.apache.hadoop.security.authorize.TestServiceLevelAuthorization.testServiceLevelAuthorization(TestServiceLevelAuthorization.java:63)
          Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.examples.QuasiMonteCarlo
          at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
          at java.security.AccessController.doPrivileged(Native Method)
          at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
          at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
          at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
          at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
          at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)

          Show
          gary murry added a comment - Error Message org/apache/hadoop/examples/QuasiMonteCarlo Stacktrace java.lang.NoClassDefFoundError: org/apache/hadoop/examples/QuasiMonteCarlo at org.apache.hadoop.mapred.TestMiniMRWithDFS.runPI(TestMiniMRWithDFS.java:176) at org.apache.hadoop.security.authorize.TestServiceLevelAuthorization.testServiceLevelAuthorization(TestServiceLevelAuthorization.java:63) Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.examples.QuasiMonteCarlo at java.net.URLClassLoader$1.run(URLClassLoader.java:200) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:188) at java.lang.ClassLoader.loadClass(ClassLoader.java:307) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:252) at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Oops, we have to update mapreduce jar.

          Show
          Tsz Wo Nicholas Sze added a comment - Oops, we have to update mapreduce jar.
          Tsz Wo Nicholas Sze made changes -
          Field Original Value New Value
          Link This issue relates to MAPREDUCE-874 [ MAPREDUCE-874 ]
          Hide
          Tsz Wo Nicholas Sze added a comment -

          I think the solution is to update hadoop-mapred-examples-0.21.0-dev.jar since MAPREDUCE-874 has changed it.

          Show
          Tsz Wo Nicholas Sze added a comment - I think the solution is to update hadoop-mapred-examples-0.21.0-dev.jar since MAPREDUCE-874 has changed it.
          Tsz Wo Nicholas Sze made changes -
          Assignee Tsz Wo (Nicholas), SZE [ szetszwo ]
          Fix Version/s 0.21.0 [ 12314046 ]
          Affects Version/s 0.21.0 [ 12314046 ]
          Tsz Wo Nicholas Sze made changes -
          Link This issue relates to HDFS-538 [ HDFS-538 ]
          Hide
          Tsz Wo Nicholas Sze added a comment -

          After updated hadoop-mapred-examples-0.21.0-dev.jar locally, passed TestServiceLevelAuthorization.

              [junit] Running org.apache.hadoop.security.authorize.TestServiceLevelAuthorization
              [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 110.037 sec
          

          No patch is needed for this issue.

          Show
          Tsz Wo Nicholas Sze added a comment - After updated hadoop-mapred-examples-0.21.0-dev.jar locally, passed TestServiceLevelAuthorization. [junit] Running org.apache.hadoop.security.authorize.TestServiceLevelAuthorization [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 110.037 sec No patch is needed for this issue.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          I have updated hadoop-mapred-examples-0.21.0-dev.jar on trunk.

          Show
          Tsz Wo Nicholas Sze added a comment - I have updated hadoop-mapred-examples-0.21.0-dev.jar on trunk.
          Tsz Wo Nicholas Sze made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Hide
          Tsz Wo Nicholas Sze added a comment -

          In build #87, TestServiceLevelAuthorization failed with a NullPointerException.

          java.lang.NullPointerException
          	at org.apache.hadoop.mapred.TestMiniMRWithDFS.runWordCount(TestMiniMRWithDFS.java:209)
          	at org.apache.hadoop.security.authorize.TestServiceLevelAuthorization.testServiceLevelAuthorization(TestServiceLevelAuthorization.java:64)
          
          Show
          Tsz Wo Nicholas Sze added a comment - In build #87 , TestServiceLevelAuthorization failed with a NullPointerException. java.lang.NullPointerException at org.apache.hadoop.mapred.TestMiniMRWithDFS.runWordCount(TestMiniMRWithDFS.java:209) at org.apache.hadoop.security.authorize.TestServiceLevelAuthorization.testServiceLevelAuthorization(TestServiceLevelAuthorization.java:64)
          Tsz Wo Nicholas Sze made changes -
          Resolution Fixed [ 1 ]
          Status Resolved [ 5 ] Reopened [ 4 ]
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Interestingly, TestServiceLevelAuthorization does not fail for the following command, which was used in my previous local test.

          ant run-test-hdfs-with-mr -Dtestcase=TestServiceLevelAuthorization
          

          However, it will fail with NPE if it is executed by

          ant test-core -Dtestcase=TestServiceLevelAuthorization
          
          Show
          Tsz Wo Nicholas Sze added a comment - Interestingly, TestServiceLevelAuthorization does not fail for the following command, which was used in my previous local test. ant run-test-hdfs-with-mr -Dtestcase=TestServiceLevelAuthorization However, it will fail with NPE if it is executed by ant test-core -Dtestcase=TestServiceLevelAuthorization
          Hide
          Tsz Wo Nicholas Sze added a comment -

          > Interestingly, TestServiceLevelAuthorization does not fail for the following command, ...
          Got a different story this morning. The test fails for both commands. I may have done some wrong previously.

          Show
          Tsz Wo Nicholas Sze added a comment - > Interestingly, TestServiceLevelAuthorization does not fail for the following command, ... Got a different story this morning. The test fails for both commands. I may have done some wrong previously.
          Hide
          Jakob Homan added a comment -

          I was seeing the same behavior.

          Show
          Jakob Homan added a comment - I was seeing the same behavior.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          TestServiceLevelAuthorization.testServiceLevelAuthorization() creates a MiniMRCluster and runs some jobs. I think it makes sense to move it to the mapreduce sub-project.

          Show
          Tsz Wo Nicholas Sze added a comment - TestServiceLevelAuthorization.testServiceLevelAuthorization() creates a MiniMRCluster and runs some jobs. I think it makes sense to move it to the mapreduce sub-project.
          Hide
          Jakob Homan added a comment -

          I looked into this and the problem seems to emanating from CompletedJobStatusStore::readCounters(), which is returning a null value and crashing the test. I don't believe it's related HDFS-538 and it's definitely not related to MAPREDUCE-874. Since the mapred jars don't get updated with every commit back into hdfs, it's not immediately obvious what may have caused the regression. I've spoken with Arun about it offline and he's looking into a possible cause.

          Show
          Jakob Homan added a comment - I looked into this and the problem seems to emanating from CompletedJobStatusStore::readCounters(), which is returning a null value and crashing the test. I don't believe it's related HDFS-538 and it's definitely not related to MAPREDUCE-874 . Since the mapred jars don't get updated with every commit back into hdfs, it's not immediately obvious what may have caused the regression. I've spoken with Arun about it offline and he's looking into a possible cause.
          Hide
          Amareshwari Sriramadasu added a comment -

          MAPREDUCE-870 retiresjobs immediately. So, mapred.job.tracker.retire.jobs is set to false in mapred-site.xml for mapreduce tests to run.
          I think the same thing should be done for hdfs also.

          Show
          Amareshwari Sriramadasu added a comment - MAPREDUCE-870 retiresjobs immediately. So, mapred.job.tracker.retire.jobs is set to false in mapred-site.xml for mapreduce tests to run. I think the same thing should be done for hdfs also.
          Hide
          Amareshwari Sriramadasu added a comment -

          Attaching patch mapred.job.tracker.retire.jobs to false in mapred-site.xml

          Show
          Amareshwari Sriramadasu added a comment - Attaching patch mapred.job.tracker.retire.jobs to false in mapred-site.xml
          Amareshwari Sriramadasu made changes -
          Attachment patch-568.txt [ 12417974 ]
          Hide
          Amareshwari Sriramadasu added a comment -

          submitting for hudson

          Show
          Amareshwari Sriramadasu added a comment - submitting for hudson
          Amareshwari Sriramadasu made changes -
          Status Reopened [ 4 ] Patch Available [ 10002 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12417974/patch-568.txt
          against trunk revision 808670.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          -1 release audit. The applied patch generated 148 release audit warnings (more than the trunk's current 147 warnings).

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/98/testReport/
          Release audit warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/98/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/98/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/98/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/98/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12417974/patch-568.txt against trunk revision 808670. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. -1 release audit. The applied patch generated 148 release audit warnings (more than the trunk's current 147 warnings). +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/98/testReport/ Release audit warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/98/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/98/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/98/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/98/console This message is automatically generated.
          Hide
          Arun C Murthy added a comment -

          The RAT warning is for the src/test/mapred-site.xml which has been added here. +1 for the patch.

          Show
          Arun C Murthy added a comment - The RAT warning is for the src/test/mapred-site.xml which has been added here. +1 for the patch.
          Tsz Wo Nicholas Sze made changes -
          Hadoop Flags [Reviewed]
          Assignee Tsz Wo (Nicholas), SZE [ szetszwo ] Amareshwari Sriramadasu [ amareshwari ]
          Priority Major [ 3 ] Minor [ 4 ]
          Hide
          Tsz Wo Nicholas Sze added a comment -

          I have committed this. Thanks, Amareshwari!

          Show
          Tsz Wo Nicholas Sze added a comment - I have committed this. Thanks, Amareshwari!
          Tsz Wo Nicholas Sze made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #6 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/6/)
          . Set mapred.job.tracker.retire.jobs to false in src/test/mapred-site.xml for mapreduce tests to run. Contributed by Amareshwari Sriramadasu

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #6 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/6/ ) . Set mapred.job.tracker.retire.jobs to false in src/test/mapred-site.xml for mapreduce tests to run. Contributed by Amareshwari Sriramadasu
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #65 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/65/)
          . Set mapred.job.tracker.retire.jobs to false in src/test/mapred-site.xml for mapreduce tests to run. Contributed by Amareshwari Sriramadasu

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #65 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/65/ ) . Set mapred.job.tracker.retire.jobs to false in src/test/mapred-site.xml for mapreduce tests to run. Contributed by Amareshwari Sriramadasu
          Tom White made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

            People

            • Assignee:
              Amareshwari Sriramadasu
              Reporter:
              gary murry
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development