Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: None
    • Labels:
      None

      Description

      TestDelegationTokenRenewal does not compile in trunk.
      The reason is that DelegationTokenSecretManager in hdfs requires namesystem in constructor.

      1. MR-1537.1.patch
        3 kB
        Jitendra Nath Pandey
      2. MR-1537.2.patch
        1.0 kB
        Jitendra Nath Pandey
      3. MR-1537.3.patch
        3 kB
        Jitendra Nath Pandey

        Activity

        Hide
        Devaraj Das added a comment -

        +1

        Show
        Devaraj Das added a comment - +1
        Hide
        Devaraj Das added a comment -

        I just committed this. Thanks, Jitendra!

        Show
        Devaraj Das added a comment - I just committed this. Thanks, Jitendra!
        Hide
        Devaraj Das added a comment -

        HADOOP-6573 & HDFS-984 seems to have broken other testcases in MR not just the compilation of the TestDelegationTokenRenewal. Reopening issue

        Show
        Devaraj Das added a comment - HADOOP-6573 & HDFS-984 seems to have broken other testcases in MR not just the compilation of the TestDelegationTokenRenewal. Reopening issue
        Hide
        Rodrigo Schmidt added a comment -

        If it helps, this is the error that I'm getting in TestRaidHar inside contrib/raid:

        [junit] java.io.IOException: java.lang.NullPointerException
        [junit] at org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.createPassword(AbstractDelegationTokenSecretManager.java:189)
        [junit] at org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.createPassword(AbstractDelegationTokenSecretManager.java:45)
        [junit] at org.apache.hadoop.security.token.Token.<init>(Token.java:51)
        [junit] at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getDelegationToken(FSNamesystem.java:4445)
        [junit] at org.apache.hadoop.hdfs.server.namenode.NameNode.getDelegationToken(NameNode.java:563)
        [junit] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        [junit] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        [junit] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        [junit] at java.lang.reflect.Method.invoke(Method.java:597)
        [junit] at org.apache.hadoop.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:342)
        [junit] at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1224)
        [junit] at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1220)
        [junit] at java.security.AccessController.doPrivileged(Native Method)
        [junit] at javax.security.auth.Subject.doAs(Subject.java:396)
        [junit] at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:706)
        [junit] at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1218)
        [junit] 10/02/26 01:37:46 INFO mapreduce.JobSubmitter: Cleaning up the staging area hdfs://localhost:56663/tmp/hadoop-rschmidt/mapred/staging/rschmidt/.staging/job_20100226013723998_0002
        [junit] 10/02/26 01:37:46 INFO security.Groups: Returning cached groups for 'rschmidt'
        [junit] java.io.IOException: java.lang.NullPointerException
        [junit] at org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.createPassword(AbstractDelegationTokenSecretManager.java:189)
        [junit] at org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.createPassword(AbstractDelegationTokenSecretManager.java:45)
        [junit] at org.apache.hadoop.security.token.Token.<init>(Token.java:51)
        [junit] at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getDelegationToken(FSNamesystem.java:4445)
        [junit] at org.apache.hadoop.hdfs.server.namenode.NameNode.getDelegationToken(NameNode.java:563)
        [junit] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        [junit] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        [junit] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        [junit] at java.lang.reflect.Method.invoke(Method.java:597)
        [junit] at org.apache.hadoop.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:342)
        [junit] at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1224)
        [junit] at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1220)
        [junit] at java.security.AccessController.doPrivileged(Native Method)
        [junit] at javax.security.auth.Subject.doAs(Subject.java:396)
        [junit] at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:706)
        [junit] at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1218)

        Show
        Rodrigo Schmidt added a comment - If it helps, this is the error that I'm getting in TestRaidHar inside contrib/raid: [junit] java.io.IOException: java.lang.NullPointerException [junit] at org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.createPassword(AbstractDelegationTokenSecretManager.java:189) [junit] at org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.createPassword(AbstractDelegationTokenSecretManager.java:45) [junit] at org.apache.hadoop.security.token.Token.<init>(Token.java:51) [junit] at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getDelegationToken(FSNamesystem.java:4445) [junit] at org.apache.hadoop.hdfs.server.namenode.NameNode.getDelegationToken(NameNode.java:563) [junit] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [junit] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) [junit] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [junit] at java.lang.reflect.Method.invoke(Method.java:597) [junit] at org.apache.hadoop.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:342) [junit] at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1224) [junit] at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1220) [junit] at java.security.AccessController.doPrivileged(Native Method) [junit] at javax.security.auth.Subject.doAs(Subject.java:396) [junit] at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:706) [junit] at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1218) [junit] 10/02/26 01:37:46 INFO mapreduce.JobSubmitter: Cleaning up the staging area hdfs://localhost:56663/tmp/hadoop-rschmidt/mapred/staging/rschmidt/.staging/job_20100226013723998_0002 [junit] 10/02/26 01:37:46 INFO security.Groups: Returning cached groups for 'rschmidt' [junit] java.io.IOException: java.lang.NullPointerException [junit] at org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.createPassword(AbstractDelegationTokenSecretManager.java:189) [junit] at org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.createPassword(AbstractDelegationTokenSecretManager.java:45) [junit] at org.apache.hadoop.security.token.Token.<init>(Token.java:51) [junit] at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getDelegationToken(FSNamesystem.java:4445) [junit] at org.apache.hadoop.hdfs.server.namenode.NameNode.getDelegationToken(NameNode.java:563) [junit] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [junit] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) [junit] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [junit] at java.lang.reflect.Method.invoke(Method.java:597) [junit] at org.apache.hadoop.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:342) [junit] at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1224) [junit] at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1220) [junit] at java.security.AccessController.doPrivileged(Native Method) [junit] at javax.security.auth.Subject.doAs(Subject.java:396) [junit] at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:706) [junit] at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1218)
        Hide
        Rodrigo Schmidt added a comment -

        Can you please confirm the error I'm seeing is related to this issue? Thanks!

        Show
        Rodrigo Schmidt added a comment - Can you please confirm the error I'm seeing is related to this issue? Thanks!
        Hide
        Ravi Gummadi added a comment -

        Yes. That is the stack trace from the other tests failures also. This JIRA is reopened to address that. As Devaraj pointed out, the actual fix could be in HDFS and/or COMMON.

        Show
        Ravi Gummadi added a comment - Yes. That is the stack trace from the other tests failures also. This JIRA is reopened to address that. As Devaraj pointed out, the actual fix could be in HDFS and/or COMMON.
        Hide
        Jitendra Nath Pandey added a comment -

        A new patch added. It fetches delegation tokens from namenode only if security is enabled.

        Show
        Jitendra Nath Pandey added a comment - A new patch added. It fetches delegation tokens from namenode only if security is enabled.
        Hide
        Jitendra Nath Pandey added a comment -

        test-patch results:

        [exec] +1 overall.
        [exec]
        [exec] +1 @author. The patch does not contain any @author tags.
        [exec]
        [exec] +1 tests included. The patch appears to include 3 new or modified tests.
        [exec]
        [exec] +1 javadoc. The javadoc tool did not generate any warning messages.
        [exec]
        [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
        [exec]
        [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
        [exec]
        [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.

        Show
        Jitendra Nath Pandey added a comment - test-patch results: [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 3 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
        Hide
        Jitendra Nath Pandey added a comment -

        ant test results:
        4 failures due to unrelated reasons

        1. TestJobHistory: succeeds when run from command line

        2. TestMiniMRLocalFS: File QuasiMonteCarlo_TMP_3_141592654/out/reduce-out does not exist.
        java.io.FileNotFoundException: File QuasiMonteCarlo_TMP_3_141592654/out/reduce-out does not exist.
        at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:420)

        3. TestDelegationTokenRenewal: It is a flaky test (MR-1532 addresses it)

        4. TestMiniMRChildTask:
        junit.framework.AssertionFailedError: Exception in testing temp dir
        junit.framework.AssertionFailedError: Exception in testing child env
        junit.framework.AssertionFailedError: Exception in testing child env

        Show
        Jitendra Nath Pandey added a comment - ant test results: 4 failures due to unrelated reasons 1. TestJobHistory: succeeds when run from command line 2. TestMiniMRLocalFS: File QuasiMonteCarlo_TMP_3_141592654/out/reduce-out does not exist. java.io.FileNotFoundException: File QuasiMonteCarlo_TMP_3_141592654/out/reduce-out does not exist. at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:420) 3. TestDelegationTokenRenewal: It is a flaky test (MR-1532 addresses it) 4. TestMiniMRChildTask: junit.framework.AssertionFailedError: Exception in testing temp dir junit.framework.AssertionFailedError: Exception in testing child env junit.framework.AssertionFailedError: Exception in testing child env
        Hide
        Devaraj Das added a comment -

        +1
        I just committed this. Thanks, Jitendra!

        Show
        Devaraj Das added a comment - +1 I just committed this. Thanks, Jitendra!
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #252 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/252/)

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #252 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/252/ )
        Hide
        Amar Kamat added a comment -

        This test failed for me once today with the following message

        renew wasn't called as many times as expected expected:<5> but was:<4>
        junit.framework.AssertionFailedError: renew wasn't called as many times as expected expected:<5> but was:<4>
                at org.apache.hadoop.mapreduce.security.token.TestDelegationTokenRenewal.testDTRenewal(TestDelegationTokenRenewal.java:303)
        
        Show
        Amar Kamat added a comment - This test failed for me once today with the following message renew wasn't called as many times as expected expected:<5> but was:<4> junit.framework.AssertionFailedError: renew wasn't called as many times as expected expected:<5> but was:<4> at org.apache.hadoop.mapreduce.security.token.TestDelegationTokenRenewal.testDTRenewal(TestDelegationTokenRenewal.java:303)

          People

          • Assignee:
            Jitendra Nath Pandey
            Reporter:
            Jitendra Nath Pandey
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development