Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-568

TestServiceLevelAuthorization fails on latest build in Hudson

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.21.0
    • Fix Version/s: 0.21.0
    • Component/s: test
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The latest build in Hudson of Hadoop-hdfs fails org.apache.hadoop.security.authorize.TestServiceLevelAuthorization.testServiceLevelAuthorization. (http://hudson.zones.apache.org/hudson/view/Hadoop/job/Hadoop-Hdfs-trunk/61/testReport/)

      1. patch-568.txt
        0.5 kB
        Amareshwari Sriramadasu

        Issue Links

          Activity

          Hide
          gary murry added a comment -

          Error Message

          org/apache/hadoop/examples/QuasiMonteCarlo

          Stacktrace

          java.lang.NoClassDefFoundError: org/apache/hadoop/examples/QuasiMonteCarlo
          at org.apache.hadoop.mapred.TestMiniMRWithDFS.runPI(TestMiniMRWithDFS.java:176)
          at org.apache.hadoop.security.authorize.TestServiceLevelAuthorization.testServiceLevelAuthorization(TestServiceLevelAuthorization.java:63)
          Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.examples.QuasiMonteCarlo
          at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
          at java.security.AccessController.doPrivileged(Native Method)
          at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
          at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
          at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
          at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
          at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)

          Show
          gary murry added a comment - Error Message org/apache/hadoop/examples/QuasiMonteCarlo Stacktrace java.lang.NoClassDefFoundError: org/apache/hadoop/examples/QuasiMonteCarlo at org.apache.hadoop.mapred.TestMiniMRWithDFS.runPI(TestMiniMRWithDFS.java:176) at org.apache.hadoop.security.authorize.TestServiceLevelAuthorization.testServiceLevelAuthorization(TestServiceLevelAuthorization.java:63) Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.examples.QuasiMonteCarlo at java.net.URLClassLoader$1.run(URLClassLoader.java:200) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:188) at java.lang.ClassLoader.loadClass(ClassLoader.java:307) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:252) at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Oops, we have to update mapreduce jar.

          Show
          Tsz Wo Nicholas Sze added a comment - Oops, we have to update mapreduce jar.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          I think the solution is to update hadoop-mapred-examples-0.21.0-dev.jar since MAPREDUCE-874 has changed it.

          Show
          Tsz Wo Nicholas Sze added a comment - I think the solution is to update hadoop-mapred-examples-0.21.0-dev.jar since MAPREDUCE-874 has changed it.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          After updated hadoop-mapred-examples-0.21.0-dev.jar locally, passed TestServiceLevelAuthorization.

              [junit] Running org.apache.hadoop.security.authorize.TestServiceLevelAuthorization
              [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 110.037 sec
          

          No patch is needed for this issue.

          Show
          Tsz Wo Nicholas Sze added a comment - After updated hadoop-mapred-examples-0.21.0-dev.jar locally, passed TestServiceLevelAuthorization. [junit] Running org.apache.hadoop.security.authorize.TestServiceLevelAuthorization [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 110.037 sec No patch is needed for this issue.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          I have updated hadoop-mapred-examples-0.21.0-dev.jar on trunk.

          Show
          Tsz Wo Nicholas Sze added a comment - I have updated hadoop-mapred-examples-0.21.0-dev.jar on trunk.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          In build #87, TestServiceLevelAuthorization failed with a NullPointerException.

          java.lang.NullPointerException
          	at org.apache.hadoop.mapred.TestMiniMRWithDFS.runWordCount(TestMiniMRWithDFS.java:209)
          	at org.apache.hadoop.security.authorize.TestServiceLevelAuthorization.testServiceLevelAuthorization(TestServiceLevelAuthorization.java:64)
          
          Show
          Tsz Wo Nicholas Sze added a comment - In build #87 , TestServiceLevelAuthorization failed with a NullPointerException. java.lang.NullPointerException at org.apache.hadoop.mapred.TestMiniMRWithDFS.runWordCount(TestMiniMRWithDFS.java:209) at org.apache.hadoop.security.authorize.TestServiceLevelAuthorization.testServiceLevelAuthorization(TestServiceLevelAuthorization.java:64)
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Interestingly, TestServiceLevelAuthorization does not fail for the following command, which was used in my previous local test.

          ant run-test-hdfs-with-mr -Dtestcase=TestServiceLevelAuthorization
          

          However, it will fail with NPE if it is executed by

          ant test-core -Dtestcase=TestServiceLevelAuthorization
          
          Show
          Tsz Wo Nicholas Sze added a comment - Interestingly, TestServiceLevelAuthorization does not fail for the following command, which was used in my previous local test. ant run-test-hdfs-with-mr -Dtestcase=TestServiceLevelAuthorization However, it will fail with NPE if it is executed by ant test-core -Dtestcase=TestServiceLevelAuthorization
          Hide
          Tsz Wo Nicholas Sze added a comment -

          > Interestingly, TestServiceLevelAuthorization does not fail for the following command, ...
          Got a different story this morning. The test fails for both commands. I may have done some wrong previously.

          Show
          Tsz Wo Nicholas Sze added a comment - > Interestingly, TestServiceLevelAuthorization does not fail for the following command, ... Got a different story this morning. The test fails for both commands. I may have done some wrong previously.
          Hide
          Jakob Homan added a comment -

          I was seeing the same behavior.

          Show
          Jakob Homan added a comment - I was seeing the same behavior.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          TestServiceLevelAuthorization.testServiceLevelAuthorization() creates a MiniMRCluster and runs some jobs. I think it makes sense to move it to the mapreduce sub-project.

          Show
          Tsz Wo Nicholas Sze added a comment - TestServiceLevelAuthorization.testServiceLevelAuthorization() creates a MiniMRCluster and runs some jobs. I think it makes sense to move it to the mapreduce sub-project.
          Hide
          Jakob Homan added a comment -

          I looked into this and the problem seems to emanating from CompletedJobStatusStore::readCounters(), which is returning a null value and crashing the test. I don't believe it's related HDFS-538 and it's definitely not related to MAPREDUCE-874. Since the mapred jars don't get updated with every commit back into hdfs, it's not immediately obvious what may have caused the regression. I've spoken with Arun about it offline and he's looking into a possible cause.

          Show
          Jakob Homan added a comment - I looked into this and the problem seems to emanating from CompletedJobStatusStore::readCounters(), which is returning a null value and crashing the test. I don't believe it's related HDFS-538 and it's definitely not related to MAPREDUCE-874 . Since the mapred jars don't get updated with every commit back into hdfs, it's not immediately obvious what may have caused the regression. I've spoken with Arun about it offline and he's looking into a possible cause.
          Hide
          Amareshwari Sriramadasu added a comment -

          MAPREDUCE-870 retiresjobs immediately. So, mapred.job.tracker.retire.jobs is set to false in mapred-site.xml for mapreduce tests to run.
          I think the same thing should be done for hdfs also.

          Show
          Amareshwari Sriramadasu added a comment - MAPREDUCE-870 retiresjobs immediately. So, mapred.job.tracker.retire.jobs is set to false in mapred-site.xml for mapreduce tests to run. I think the same thing should be done for hdfs also.
          Hide
          Amareshwari Sriramadasu added a comment -

          Attaching patch mapred.job.tracker.retire.jobs to false in mapred-site.xml

          Show
          Amareshwari Sriramadasu added a comment - Attaching patch mapred.job.tracker.retire.jobs to false in mapred-site.xml
          Hide
          Amareshwari Sriramadasu added a comment -

          submitting for hudson

          Show
          Amareshwari Sriramadasu added a comment - submitting for hudson
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12417974/patch-568.txt
          against trunk revision 808670.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          -1 release audit. The applied patch generated 148 release audit warnings (more than the trunk's current 147 warnings).

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/98/testReport/
          Release audit warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/98/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/98/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/98/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/98/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12417974/patch-568.txt against trunk revision 808670. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. -1 release audit. The applied patch generated 148 release audit warnings (more than the trunk's current 147 warnings). +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/98/testReport/ Release audit warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/98/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/98/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/98/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/98/console This message is automatically generated.
          Hide
          Arun C Murthy added a comment -

          The RAT warning is for the src/test/mapred-site.xml which has been added here. +1 for the patch.

          Show
          Arun C Murthy added a comment - The RAT warning is for the src/test/mapred-site.xml which has been added here. +1 for the patch.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          I have committed this. Thanks, Amareshwari!

          Show
          Tsz Wo Nicholas Sze added a comment - I have committed this. Thanks, Amareshwari!
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #6 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/6/)
          . Set mapred.job.tracker.retire.jobs to false in src/test/mapred-site.xml for mapreduce tests to run. Contributed by Amareshwari Sriramadasu

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #6 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/6/ ) . Set mapred.job.tracker.retire.jobs to false in src/test/mapred-site.xml for mapreduce tests to run. Contributed by Amareshwari Sriramadasu
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #65 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/65/)
          . Set mapred.job.tracker.retire.jobs to false in src/test/mapred-site.xml for mapreduce tests to run. Contributed by Amareshwari Sriramadasu

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #65 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/65/ ) . Set mapred.job.tracker.retire.jobs to false in src/test/mapred-site.xml for mapreduce tests to run. Contributed by Amareshwari Sriramadasu

            People

            • Assignee:
              Amareshwari Sriramadasu
              Reporter:
              gary murry
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development