Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4313

TestTokenCache doesn't compile due TokenCache.getDelegationToken compilation error

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.0.2-alpha
    • Component/s: build, test
    • Labels:
      None

      Description

      Saw this on the trunk Jenkins job:

      compile-mapred-test:
          [mkdir] Created dir: /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk/trunk/hadoop-mapreduce-project/build/test/mapred/classes
          [mkdir] Created dir: /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk/trunk/hadoop-mapreduce-project/build/test/mapred/testjar
          [mkdir] Created dir: /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk/trunk/hadoop-mapreduce-project/build/test/mapred/testshell
          [javac] Compiling 95 source files to /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk/trunk/hadoop-mapreduce-project/build/test/mapred/classes
          [javac] /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/security/TestTokenCache.java:292: incompatible types
          [javac] found   : org.apache.hadoop.security.token.Token<capture#315 of ?>
          [javac] required: org.apache.hadoop.security.token.Token<org.apache.hadoop.hdfs.security.token.delegation.DelegationTokenIdentifier>
          [javac]     Token<DelegationTokenIdentifier> nnt = TokenCache.getDelegationToken(
          [javac]                                                                         ^
          [javac] Note: Some input files use or override a deprecated API.
          [javac] Note: Recompile with -Xlint:deprecation for details.
          [javac] 1 error
      

        Activity

        Hide
        eli2 Eli Collins added a comment -

        Perhaps we're not using internal resolvers or publishing the right common artifact?

        Show
        eli2 Eli Collins added a comment - Perhaps we're not using internal resolvers or publishing the right common artifact?
        Hide
        daryn Daryn Sharp added a comment -

        I'll dig into this. I think it may be related to recent change to move the token renewer into common because I seem to recall it changed method signatures.

        Show
        daryn Daryn Sharp added a comment - I'll dig into this. I think it may be related to recent change to move the token renewer into common because I seem to recall it changed method signatures.
        Hide
        revans2 Robert Joseph Evans added a comment -

        OK I was able to reproduce the issue, and resolvers=internal appears to be working.

        mvn clean install -DskipTests
        cd hadoop-mapreduce-project/
        ant -Dresolvers=internal jar
        ant -Dresolvers=internal test
        

        The line that changed inside TokenCache.java appears to have come from MAPREDUCE-4148, Both Daryn and I approved it but it looks like Tom White did the original change, so I am adding him in as a watcher.

        Show
        revans2 Robert Joseph Evans added a comment - OK I was able to reproduce the issue, and resolvers=internal appears to be working. mvn clean install -DskipTests cd hadoop-mapreduce-project/ ant -Dresolvers=internal jar ant -Dresolvers=internal test The line that changed inside TokenCache.java appears to have come from MAPREDUCE-4148 , Both Daryn and I approved it but it looks like Tom White did the original change, so I am adding him in as a watcher.
        Hide
        revans2 Robert Joseph Evans added a comment -

        Well that being said I fixed the test:

        Index: src/test/mapred/org/apache/hadoop/mapreduce/security/TestTokenCache.java
        ===================================================================
        --- src/test/mapred/org/apache/hadoop/mapreduce/security/TestTokenCache.java    (revision 1346388)
        +++ src/test/mapred/org/apache/hadoop/mapreduce/security/TestTokenCache.java    (working copy)
        @@ -289,7 +289,7 @@
             // this token is keyed by hostname:port key.
             String fs_addr = 
               SecurityUtil.buildDTServiceName(p1.toUri(), NameNode.DEFAULT_PORT);
        -    Token<DelegationTokenIdentifier> nnt = TokenCache.getDelegationToken(
        +    Token<DelegationTokenIdentifier> nnt = (Token<DelegationTokenIdentifier>)TokenCache.getDelegationToken(
                 credentials, fs_addr);
             System.out.println("dt for " + p1 + "(" + fs_addr + ")" + " = " +  nnt);
             assertNotNull("Token for nn is null", nnt);
        

        But now I get a different error

        BUILD FAILED
        /home/evans/src/commit/trunk/hadoop-mapreduce-project/build.xml:782: The following error occurred while executing this line:
        /home/evans/src/commit/trunk/hadoop-mapreduce-project/build.xml:744: The following error occurred while executing this line:
        /home/evans/src/commit/trunk/hadoop-mapreduce-project/build.xml:639: hadoop-mapreduce-project/src/test/system/test does not exist.
        

        which if I remember correctly was also removed recently. We are almost to the point that I would say we shut off the ant build. Mavinization is almost done, and most if not all of our downstream projects should be off of the MRMiniCluster.

        Show
        revans2 Robert Joseph Evans added a comment - Well that being said I fixed the test: Index: src/test/mapred/org/apache/hadoop/mapreduce/security/TestTokenCache.java =================================================================== --- src/test/mapred/org/apache/hadoop/mapreduce/security/TestTokenCache.java (revision 1346388) +++ src/test/mapred/org/apache/hadoop/mapreduce/security/TestTokenCache.java (working copy) @@ -289,7 +289,7 @@ // this token is keyed by hostname:port key. String fs_addr = SecurityUtil.buildDTServiceName(p1.toUri(), NameNode.DEFAULT_PORT); - Token<DelegationTokenIdentifier> nnt = TokenCache.getDelegationToken( + Token<DelegationTokenIdentifier> nnt = (Token<DelegationTokenIdentifier>)TokenCache.getDelegationToken( credentials, fs_addr); System.out.println("dt for " + p1 + "(" + fs_addr + ")" + " = " + nnt); assertNotNull("Token for nn is null", nnt); But now I get a different error BUILD FAILED /home/evans/src/commit/trunk/hadoop-mapreduce-project/build.xml:782: The following error occurred while executing this line: /home/evans/src/commit/trunk/hadoop-mapreduce-project/build.xml:744: The following error occurred while executing this line: /home/evans/src/commit/trunk/hadoop-mapreduce-project/build.xml:639: hadoop-mapreduce-project/src/test/system/test does not exist. which if I remember correctly was also removed recently. We are almost to the point that I would say we shut off the ant build. Mavinization is almost done, and most if not all of our downstream projects should be off of the MRMiniCluster.
        Hide
        revans2 Robert Joseph Evans added a comment -

        Oh and doing the mkdir for the missing directory fixes the issue and the tests start to run again. If someone wants to +1 the above "patch" I am happy to check in the fix, although I still think that turning off ant is a direction we should be going in.

        Show
        revans2 Robert Joseph Evans added a comment - Oh and doing the mkdir for the missing directory fixes the issue and the tests start to run again. If someone wants to +1 the above "patch" I am happy to check in the fix, although I still think that turning off ant is a direction we should be going in.
        Hide
        tgraves Thomas Graves added a comment -

        +1. lgtm to make it compile. I'm not sure why we have jenkins running the ant part on trunk right now anyway.

        Show
        tgraves Thomas Graves added a comment - +1. lgtm to make it compile. I'm not sure why we have jenkins running the ant part on trunk right now anyway.
        Hide
        revans2 Robert Joseph Evans added a comment -

        I checked the small fix into branch-2, and trunk. These are the two places that the other change went in that broke this.

        Show
        revans2 Robert Joseph Evans added a comment - I checked the small fix into branch-2, and trunk. These are the two places that the other change went in that broke this.
        Hide
        hudson Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #2323 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2323/)
        MAPREDUCE-4313. TestTokenCache doesn't compile due TokenCache.getDelegationToken compilation error (bobby) (Revision 1346406)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1346406
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/security/TestTokenCache.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/src/test/system
        • /hadoop/common/trunk/hadoop-mapreduce-project/src/test/system/test
        Show
        hudson Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2323 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2323/ ) MAPREDUCE-4313 . TestTokenCache doesn't compile due TokenCache.getDelegationToken compilation error (bobby) (Revision 1346406) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1346406 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/security/TestTokenCache.java /hadoop/common/trunk/hadoop-mapreduce-project/src/test/system /hadoop/common/trunk/hadoop-mapreduce-project/src/test/system/test
        Hide
        hudson Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #2396 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2396/)
        MAPREDUCE-4313. TestTokenCache doesn't compile due TokenCache.getDelegationToken compilation error (bobby) (Revision 1346406)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1346406
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/security/TestTokenCache.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/src/test/system
        • /hadoop/common/trunk/hadoop-mapreduce-project/src/test/system/test
        Show
        hudson Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2396 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2396/ ) MAPREDUCE-4313 . TestTokenCache doesn't compile due TokenCache.getDelegationToken compilation error (bobby) (Revision 1346406) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1346406 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/security/TestTokenCache.java /hadoop/common/trunk/hadoop-mapreduce-project/src/test/system /hadoop/common/trunk/hadoop-mapreduce-project/src/test/system/test
        Hide
        hudson Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #2342 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2342/)
        MAPREDUCE-4313. TestTokenCache doesn't compile due TokenCache.getDelegationToken compilation error (bobby) (Revision 1346406)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1346406
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/security/TestTokenCache.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/src/test/system
        • /hadoop/common/trunk/hadoop-mapreduce-project/src/test/system/test
        Show
        hudson Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2342 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2342/ ) MAPREDUCE-4313 . TestTokenCache doesn't compile due TokenCache.getDelegationToken compilation error (bobby) (Revision 1346406) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1346406 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/security/TestTokenCache.java /hadoop/common/trunk/hadoop-mapreduce-project/src/test/system /hadoop/common/trunk/hadoop-mapreduce-project/src/test/system/test
        Hide
        hudson Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #1069 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1069/)
        MAPREDUCE-4313. TestTokenCache doesn't compile due TokenCache.getDelegationToken compilation error (bobby) (Revision 1346406)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1346406
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/security/TestTokenCache.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/src/test/system
        • /hadoop/common/trunk/hadoop-mapreduce-project/src/test/system/test
        Show
        hudson Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1069 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1069/ ) MAPREDUCE-4313 . TestTokenCache doesn't compile due TokenCache.getDelegationToken compilation error (bobby) (Revision 1346406) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1346406 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/security/TestTokenCache.java /hadoop/common/trunk/hadoop-mapreduce-project/src/test/system /hadoop/common/trunk/hadoop-mapreduce-project/src/test/system/test
        Hide
        hudson Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1102 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1102/)
        MAPREDUCE-4313. TestTokenCache doesn't compile due TokenCache.getDelegationToken compilation error (bobby) (Revision 1346406)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1346406
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/security/TestTokenCache.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/src/test/system
        • /hadoop/common/trunk/hadoop-mapreduce-project/src/test/system/test
        Show
        hudson Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1102 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1102/ ) MAPREDUCE-4313 . TestTokenCache doesn't compile due TokenCache.getDelegationToken compilation error (bobby) (Revision 1346406) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1346406 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/security/TestTokenCache.java /hadoop/common/trunk/hadoop-mapreduce-project/src/test/system /hadoop/common/trunk/hadoop-mapreduce-project/src/test/system/test

          People

          • Assignee:
            revans2 Robert Joseph Evans
            Reporter:
            eli2 Eli Collins
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development