Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4313

TestTokenCache doesn't compile due TokenCache.getDelegationToken compilation error

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.0, 2.0.2-alpha
    • Component/s: build, test
    • Labels:
      None

      Description

      Saw this on the trunk Jenkins job:

      compile-mapred-test:
          [mkdir] Created dir: /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk/trunk/hadoop-mapreduce-project/build/test/mapred/classes
          [mkdir] Created dir: /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk/trunk/hadoop-mapreduce-project/build/test/mapred/testjar
          [mkdir] Created dir: /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk/trunk/hadoop-mapreduce-project/build/test/mapred/testshell
          [javac] Compiling 95 source files to /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk/trunk/hadoop-mapreduce-project/build/test/mapred/classes
          [javac] /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/security/TestTokenCache.java:292: incompatible types
          [javac] found   : org.apache.hadoop.security.token.Token<capture#315 of ?>
          [javac] required: org.apache.hadoop.security.token.Token<org.apache.hadoop.hdfs.security.token.delegation.DelegationTokenIdentifier>
          [javac]     Token<DelegationTokenIdentifier> nnt = TokenCache.getDelegationToken(
          [javac]                                                                         ^
          [javac] Note: Some input files use or override a deprecated API.
          [javac] Note: Recompile with -Xlint:deprecation for details.
          [javac] 1 error
      

        Activity

        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1102 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1102/)
        MAPREDUCE-4313. TestTokenCache doesn't compile due TokenCache.getDelegationToken compilation error (bobby) (Revision 1346406)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1346406
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/security/TestTokenCache.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/src/test/system
        • /hadoop/common/trunk/hadoop-mapreduce-project/src/test/system/test
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1102 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1102/ ) MAPREDUCE-4313 . TestTokenCache doesn't compile due TokenCache.getDelegationToken compilation error (bobby) (Revision 1346406) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1346406 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/security/TestTokenCache.java /hadoop/common/trunk/hadoop-mapreduce-project/src/test/system /hadoop/common/trunk/hadoop-mapreduce-project/src/test/system/test
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #1069 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1069/)
        MAPREDUCE-4313. TestTokenCache doesn't compile due TokenCache.getDelegationToken compilation error (bobby) (Revision 1346406)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1346406
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/security/TestTokenCache.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/src/test/system
        • /hadoop/common/trunk/hadoop-mapreduce-project/src/test/system/test
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1069 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1069/ ) MAPREDUCE-4313 . TestTokenCache doesn't compile due TokenCache.getDelegationToken compilation error (bobby) (Revision 1346406) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1346406 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/security/TestTokenCache.java /hadoop/common/trunk/hadoop-mapreduce-project/src/test/system /hadoop/common/trunk/hadoop-mapreduce-project/src/test/system/test
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #2342 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2342/)
        MAPREDUCE-4313. TestTokenCache doesn't compile due TokenCache.getDelegationToken compilation error (bobby) (Revision 1346406)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1346406
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/security/TestTokenCache.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/src/test/system
        • /hadoop/common/trunk/hadoop-mapreduce-project/src/test/system/test
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2342 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2342/ ) MAPREDUCE-4313 . TestTokenCache doesn't compile due TokenCache.getDelegationToken compilation error (bobby) (Revision 1346406) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1346406 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/security/TestTokenCache.java /hadoop/common/trunk/hadoop-mapreduce-project/src/test/system /hadoop/common/trunk/hadoop-mapreduce-project/src/test/system/test
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #2396 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2396/)
        MAPREDUCE-4313. TestTokenCache doesn't compile due TokenCache.getDelegationToken compilation error (bobby) (Revision 1346406)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1346406
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/security/TestTokenCache.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/src/test/system
        • /hadoop/common/trunk/hadoop-mapreduce-project/src/test/system/test
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2396 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2396/ ) MAPREDUCE-4313 . TestTokenCache doesn't compile due TokenCache.getDelegationToken compilation error (bobby) (Revision 1346406) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1346406 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/security/TestTokenCache.java /hadoop/common/trunk/hadoop-mapreduce-project/src/test/system /hadoop/common/trunk/hadoop-mapreduce-project/src/test/system/test
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #2323 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2323/)
        MAPREDUCE-4313. TestTokenCache doesn't compile due TokenCache.getDelegationToken compilation error (bobby) (Revision 1346406)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1346406
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/security/TestTokenCache.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/src/test/system
        • /hadoop/common/trunk/hadoop-mapreduce-project/src/test/system/test
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2323 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2323/ ) MAPREDUCE-4313 . TestTokenCache doesn't compile due TokenCache.getDelegationToken compilation error (bobby) (Revision 1346406) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1346406 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/security/TestTokenCache.java /hadoop/common/trunk/hadoop-mapreduce-project/src/test/system /hadoop/common/trunk/hadoop-mapreduce-project/src/test/system/test
        Hide
        Robert Joseph Evans added a comment -

        I checked the small fix into branch-2, and trunk. These are the two places that the other change went in that broke this.

        Show
        Robert Joseph Evans added a comment - I checked the small fix into branch-2, and trunk. These are the two places that the other change went in that broke this.
        Hide
        Thomas Graves added a comment -

        +1. lgtm to make it compile. I'm not sure why we have jenkins running the ant part on trunk right now anyway.

        Show
        Thomas Graves added a comment - +1. lgtm to make it compile. I'm not sure why we have jenkins running the ant part on trunk right now anyway.
        Hide
        Robert Joseph Evans added a comment -

        Oh and doing the mkdir for the missing directory fixes the issue and the tests start to run again. If someone wants to +1 the above "patch" I am happy to check in the fix, although I still think that turning off ant is a direction we should be going in.

        Show
        Robert Joseph Evans added a comment - Oh and doing the mkdir for the missing directory fixes the issue and the tests start to run again. If someone wants to +1 the above "patch" I am happy to check in the fix, although I still think that turning off ant is a direction we should be going in.
        Hide
        Robert Joseph Evans added a comment -

        Well that being said I fixed the test:

        Index: src/test/mapred/org/apache/hadoop/mapreduce/security/TestTokenCache.java
        ===================================================================
        --- src/test/mapred/org/apache/hadoop/mapreduce/security/TestTokenCache.java    (revision 1346388)
        +++ src/test/mapred/org/apache/hadoop/mapreduce/security/TestTokenCache.java    (working copy)
        @@ -289,7 +289,7 @@
             // this token is keyed by hostname:port key.
             String fs_addr = 
               SecurityUtil.buildDTServiceName(p1.toUri(), NameNode.DEFAULT_PORT);
        -    Token<DelegationTokenIdentifier> nnt = TokenCache.getDelegationToken(
        +    Token<DelegationTokenIdentifier> nnt = (Token<DelegationTokenIdentifier>)TokenCache.getDelegationToken(
                 credentials, fs_addr);
             System.out.println("dt for " + p1 + "(" + fs_addr + ")" + " = " +  nnt);
             assertNotNull("Token for nn is null", nnt);
        

        But now I get a different error

        BUILD FAILED
        /home/evans/src/commit/trunk/hadoop-mapreduce-project/build.xml:782: The following error occurred while executing this line:
        /home/evans/src/commit/trunk/hadoop-mapreduce-project/build.xml:744: The following error occurred while executing this line:
        /home/evans/src/commit/trunk/hadoop-mapreduce-project/build.xml:639: hadoop-mapreduce-project/src/test/system/test does not exist.
        

        which if I remember correctly was also removed recently. We are almost to the point that I would say we shut off the ant build. Mavinization is almost done, and most if not all of our downstream projects should be off of the MRMiniCluster.

        Show
        Robert Joseph Evans added a comment - Well that being said I fixed the test: Index: src/test/mapred/org/apache/hadoop/mapreduce/security/TestTokenCache.java =================================================================== --- src/test/mapred/org/apache/hadoop/mapreduce/security/TestTokenCache.java (revision 1346388) +++ src/test/mapred/org/apache/hadoop/mapreduce/security/TestTokenCache.java (working copy) @@ -289,7 +289,7 @@ // this token is keyed by hostname:port key. String fs_addr = SecurityUtil.buildDTServiceName(p1.toUri(), NameNode.DEFAULT_PORT); - Token<DelegationTokenIdentifier> nnt = TokenCache.getDelegationToken( + Token<DelegationTokenIdentifier> nnt = (Token<DelegationTokenIdentifier>)TokenCache.getDelegationToken( credentials, fs_addr); System.out.println("dt for " + p1 + "(" + fs_addr + ")" + " = " + nnt); assertNotNull("Token for nn is null", nnt); But now I get a different error BUILD FAILED /home/evans/src/commit/trunk/hadoop-mapreduce-project/build.xml:782: The following error occurred while executing this line: /home/evans/src/commit/trunk/hadoop-mapreduce-project/build.xml:744: The following error occurred while executing this line: /home/evans/src/commit/trunk/hadoop-mapreduce-project/build.xml:639: hadoop-mapreduce-project/src/test/system/test does not exist. which if I remember correctly was also removed recently. We are almost to the point that I would say we shut off the ant build. Mavinization is almost done, and most if not all of our downstream projects should be off of the MRMiniCluster.
        Hide
        Robert Joseph Evans added a comment -

        OK I was able to reproduce the issue, and resolvers=internal appears to be working.

        mvn clean install -DskipTests
        cd hadoop-mapreduce-project/
        ant -Dresolvers=internal jar
        ant -Dresolvers=internal test
        

        The line that changed inside TokenCache.java appears to have come from MAPREDUCE-4148, Both Daryn and I approved it but it looks like Tom White did the original change, so I am adding him in as a watcher.

        Show
        Robert Joseph Evans added a comment - OK I was able to reproduce the issue, and resolvers=internal appears to be working. mvn clean install -DskipTests cd hadoop-mapreduce-project/ ant -Dresolvers=internal jar ant -Dresolvers=internal test The line that changed inside TokenCache.java appears to have come from MAPREDUCE-4148 , Both Daryn and I approved it but it looks like Tom White did the original change, so I am adding him in as a watcher.
        Hide
        Daryn Sharp added a comment -

        I'll dig into this. I think it may be related to recent change to move the token renewer into common because I seem to recall it changed method signatures.

        Show
        Daryn Sharp added a comment - I'll dig into this. I think it may be related to recent change to move the token renewer into common because I seem to recall it changed method signatures.
        Hide
        Eli Collins added a comment -

        Perhaps we're not using internal resolvers or publishing the right common artifact?

        Show
        Eli Collins added a comment - Perhaps we're not using internal resolvers or publishing the right common artifact?

          People

          • Assignee:
            Robert Joseph Evans
            Reporter:
            Eli Collins
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development