Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-3012

Exception while renewing delegation token

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 0.23.1
    • Fix Version/s: 0.23.2
    • Component/s: None
    • Labels:
      None
    1. HDFS-3012.txt
      0.8 kB
      Robert Joseph Evans
    2. HDFS-3012.txt
      2 kB
      Robert Joseph Evans

      Activity

      Hide
      Hudson added a comment -

      Integrated in Hadoop-Mapreduce-trunk #1006 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1006/)
      HDFS-3012. Moving to hadoop-0.23.2. (Revision 1295310)
      HDFS-3012. Exception while renewing delegation token. Contributed by Bobby Evans. (Revision 1295232)

      Result = SUCCESS
      acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1295310
      Files :

      • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt

      jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1295232
      Files :

      • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
      • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
      Show
      Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1006 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1006/ ) HDFS-3012 . Moving to hadoop-0.23.2. (Revision 1295310) HDFS-3012 . Exception while renewing delegation token. Contributed by Bobby Evans. (Revision 1295232) Result = SUCCESS acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1295310 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1295232 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
      Hide
      Hudson added a comment -

      Integrated in Hadoop-Hdfs-trunk #971 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/971/)
      HDFS-3012. Moving to hadoop-0.23.2. (Revision 1295310)
      HDFS-3012. Exception while renewing delegation token. Contributed by Bobby Evans. (Revision 1295232)

      Result = SUCCESS
      acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1295310
      Files :

      • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt

      jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1295232
      Files :

      • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
      • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
      Show
      Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #971 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/971/ ) HDFS-3012 . Moving to hadoop-0.23.2. (Revision 1295310) HDFS-3012 . Exception while renewing delegation token. Contributed by Bobby Evans. (Revision 1295232) Result = SUCCESS acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1295310 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1295232 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
      Hide
      Hudson added a comment -

      Integrated in Hadoop-Mapreduce-0.23-Commit #622 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/622/)
      Merge -c 1295232 from trunk to branch-0.23 to fix HDFS-3012. Exception while renewing delegation token. (Revision 1295311)

      Result = ABORTED
      acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1295311
      Files :

      • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
      • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
      Show
      Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Commit #622 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/622/ ) Merge -c 1295232 from trunk to branch-0.23 to fix HDFS-3012 . Exception while renewing delegation token. (Revision 1295311) Result = ABORTED acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1295311 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
      Hide
      Hudson added a comment -

      Integrated in Hadoop-Mapreduce-trunk-Commit #1820 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1820/)
      HDFS-3012. Moving to hadoop-0.23.2. (Revision 1295310)

      Result = ABORTED
      acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1295310
      Files :

      • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
      Show
      Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1820 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1820/ ) HDFS-3012 . Moving to hadoop-0.23.2. (Revision 1295310) Result = ABORTED acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1295310 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
      Hide
      Hudson added a comment -

      Integrated in Hadoop-Hdfs-trunk-Commit #1889 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1889/)
      HDFS-3012. Moving to hadoop-0.23.2. (Revision 1295310)

      Result = SUCCESS
      acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1295310
      Files :

      • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
      Show
      Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1889 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1889/ ) HDFS-3012 . Moving to hadoop-0.23.2. (Revision 1295310) Result = SUCCESS acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1295310 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
      Hide
      Hudson added a comment -

      Integrated in Hadoop-Common-0.23-Commit #623 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/623/)
      Merge -c 1295232 from trunk to branch-0.23 to fix HDFS-3012. Exception while renewing delegation token. (Revision 1295311)

      Result = SUCCESS
      acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1295311
      Files :

      • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
      • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
      Show
      Hudson added a comment - Integrated in Hadoop-Common-0.23-Commit #623 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/623/ ) Merge -c 1295232 from trunk to branch-0.23 to fix HDFS-3012 . Exception while renewing delegation token. (Revision 1295311) Result = SUCCESS acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1295311 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
      Hide
      Hudson added a comment -

      Integrated in Hadoop-Common-trunk-Commit #1814 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1814/)
      HDFS-3012. Moving to hadoop-0.23.2. (Revision 1295310)

      Result = SUCCESS
      acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1295310
      Files :

      • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
      Show
      Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1814 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1814/ ) HDFS-3012 . Moving to hadoop-0.23.2. (Revision 1295310) Result = SUCCESS acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1295310 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
      Hide
      Hudson added a comment -

      Integrated in Hadoop-Hdfs-0.23-Commit #612 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/612/)
      Merge -c 1295232 from trunk to branch-0.23 to fix HDFS-3012. Exception while renewing delegation token. (Revision 1295311)

      Result = SUCCESS
      acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1295311
      Files :

      • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
      • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
      Show
      Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Commit #612 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/612/ ) Merge -c 1295232 from trunk to branch-0.23 to fix HDFS-3012 . Exception while renewing delegation token. (Revision 1295311) Result = SUCCESS acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1295311 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
      Hide
      Arun C Murthy added a comment -

      I just merged this to 0.23.2. Thanks Bobby (and Jitendra for original review/commit).

      Show
      Arun C Murthy added a comment - I just merged this to 0.23.2. Thanks Bobby (and Jitendra for original review/commit).
      Hide
      Hudson added a comment -

      Integrated in Hadoop-Mapreduce-trunk-Commit #1817 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1817/)
      HDFS-3012. Exception while renewing delegation token. Contributed by Bobby Evans. (Revision 1295232)

      Result = ABORTED
      jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1295232
      Files :

      • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
      • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
      Show
      Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1817 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1817/ ) HDFS-3012 . Exception while renewing delegation token. Contributed by Bobby Evans. (Revision 1295232) Result = ABORTED jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1295232 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
      Hide
      Hudson added a comment -

      Integrated in Hadoop-Common-trunk-Commit #1810 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1810/)
      HDFS-3012. Exception while renewing delegation token. Contributed by Bobby Evans. (Revision 1295232)

      Result = SUCCESS
      jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1295232
      Files :

      • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
      • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
      Show
      Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1810 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1810/ ) HDFS-3012 . Exception while renewing delegation token. Contributed by Bobby Evans. (Revision 1295232) Result = SUCCESS jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1295232 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
      Hide
      Hudson added a comment -

      Integrated in Hadoop-Hdfs-trunk-Commit #1884 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1884/)
      HDFS-3012. Exception while renewing delegation token. Contributed by Bobby Evans. (Revision 1295232)

      Result = SUCCESS
      jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1295232
      Files :

      • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
      • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
      Show
      Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1884 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1884/ ) HDFS-3012 . Exception while renewing delegation token. Contributed by Bobby Evans. (Revision 1295232) Result = SUCCESS jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1295232 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
      Hide
      Robert Joseph Evans added a comment -

      Thanks for the feedback Jitendra. Could you please merge this in, as I am not an HDFS comitter, just Map/Reduce and Common.

      Show
      Robert Joseph Evans added a comment - Thanks for the feedback Jitendra. Could you please merge this in, as I am not an HDFS comitter, just Map/Reduce and Common.
      Hide
      Jitendra Nath Pandey added a comment -

      +1

      Show
      Jitendra Nath Pandey added a comment - +1
      Hide
      Hadoop QA added a comment -

      -1 overall. Here are the results of testing the latest attachment
      http://issues.apache.org/jira/secure/attachment/12516184/HDFS-3012.txt
      against trunk revision .

      +1 @author. The patch does not contain any @author tags.

      -1 tests included. The patch doesn't appear to include any new or modified tests.
      Please justify why no new tests are needed for this patch.
      Also please list what manual steps were performed to verify this patch.

      +1 javadoc. The javadoc tool did not generate any warning messages.

      +1 javac. The applied patch does not increase the total number of javac compiler warnings.

      +1 eclipse:eclipse. The patch built with eclipse:eclipse.

      +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

      +1 release audit. The applied patch does not increase the total number of release audit warnings.

      +1 core tests. The patch passed unit tests in .

      +1 contrib tests. The patch passed contrib unit tests.

      Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1913//testReport/
      Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1913//console

      This message is automatically generated.

      Show
      Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12516184/HDFS-3012.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1913//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1913//console This message is automatically generated.
      Hide
      Robert Joseph Evans added a comment -

      I have now also manually verified that this patch fixes the renewal issue as well.

      Show
      Robert Joseph Evans added a comment - I have now also manually verified that this patch fixes the renewal issue as well.
      Hide
      Robert Joseph Evans added a comment -

      Comments have been addressed and the code is much smaller then before. I am still doing manual tests to ensure that this too fixes the issue, but I assume that it will so I am uploading the patch now.

      Show
      Robert Joseph Evans added a comment - Comments have been addressed and the code is much smaller then before. I am still doing manual tests to ensure that this too fixes the issue, but I assume that it will so I am uploading the patch now.
      Hide
      Robert Joseph Evans added a comment -

      Canceling patch to address the issues.

      Show
      Robert Joseph Evans added a comment - Canceling patch to address the issues.
      Hide
      Robert Joseph Evans added a comment -

      You are correct about the init method. I did not see it there and there is no point in creating a new noop method thanks for catching that. I will also change calling that method to be be part of a static initializer.

      Show
      Robert Joseph Evans added a comment - You are correct about the init method. I did not see it there and there is no point in creating a new noop method thanks for catching that. I will also change calling that method to be be part of a static initializer.
      Hide
      Jitendra Nath Pandey added a comment -

      HdfsConfiguration#init is already there which is noop.
      I will recommend just adding a static initializer to Renewer that calls HdfsConfiguration#init.

      Show
      Jitendra Nath Pandey added a comment - HdfsConfiguration#init is already there which is noop. I will recommend just adding a static initializer to Renewer that calls HdfsConfiguration#init.
      Hide
      Robert Joseph Evans added a comment -

      Sorry I misread your comment. The Renewer has to be static or it cannot be loaded through the ServiceLoader.

      Show
      Robert Joseph Evans added a comment - Sorry I misread your comment. The Renewer has to be static or it cannot be loaded through the ServiceLoader.
      Hide
      Robert Joseph Evans added a comment -

      @Mahadev,

      The Renewer is already static. The issue is that the Renewer code has no direct interaction with HdfsConfiguration. Just an include is not enough to ensure that java will load the class and run the static block. In the past, and in most other locations the HDFS Client would create an instance of HDFSConfiguration to access any configuration values. This would force it to load, but because the RM does not interact with HDFS at all, except to renew and cancel tokens that interaction with HdfsConfiguration must be on the execution path of the Renewer.

      Show
      Robert Joseph Evans added a comment - @Mahadev, The Renewer is already static. The issue is that the Renewer code has no direct interaction with HdfsConfiguration. Just an include is not enough to ensure that java will load the class and run the static block. In the past, and in most other locations the HDFS Client would create an instance of HDFSConfiguration to access any configuration values. This would force it to load, but because the RM does not interact with HDFS at all, except to renew and cancel tokens that interaction with HdfsConfiguration must be on the execution path of the Renewer.
      Hide
      Robert Joseph Evans added a comment -

      Just fair warning that I think the patch is kind of ugly. I don't like calling an empty method to fix the issue, but I really don't want to change how these resources are loaded, because it would probably result in a much larger patch.

      Show
      Robert Joseph Evans added a comment - Just fair warning that I think the patch is kind of ugly. I don't like calling an empty method to fix the issue, but I really don't want to change how these resources are loaded, because it would probably result in a much larger patch.
      Hide
      Hadoop QA added a comment -

      -1 overall. Here are the results of testing the latest attachment
      http://issues.apache.org/jira/secure/attachment/12515974/HDFS-3012.txt
      against trunk revision .

      +1 @author. The patch does not contain any @author tags.

      -1 tests included. The patch doesn't appear to include any new or modified tests.
      Please justify why no new tests are needed for this patch.
      Also please list what manual steps were performed to verify this patch.

      +1 javadoc. The javadoc tool did not generate any warning messages.

      +1 javac. The applied patch does not increase the total number of javac compiler warnings.

      +1 eclipse:eclipse. The patch built with eclipse:eclipse.

      +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

      +1 release audit. The applied patch does not increase the total number of release audit warnings.

      +1 core tests. The patch passed unit tests in .

      +1 contrib tests. The patch passed contrib unit tests.

      Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1907//testReport/
      Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1907//console

      This message is automatically generated.

      Show
      Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12515974/HDFS-3012.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1907//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1907//console This message is automatically generated.
      Hide
      Robert Joseph Evans added a comment -

      This patch was manually tested on a secure cluster and it does fix the issue. Unit tests would be very hard to write as it has to deal with configuration and the classpath to make it work.

      Show
      Robert Joseph Evans added a comment - This patch was manually tested on a secure cluster and it does fix the issue. Unit tests would be very hard to write as it has to deal with configuration and the classpath to make it work.
      Hide
      Mahadev konar added a comment -

      Hmmm,
      This is very interesting. Would making the Renewer non static work?

      Show
      Mahadev konar added a comment - Hmmm, This is very interesting. Would making the Renewer non static work?
      Hide
      Robert Joseph Evans added a comment -

      OK So the issue is that Tokens and Renewal of the tokens has been abstracted away so much that the RM has no direct reference to HDFS at all. The DFSClient.Renewer is loaded using a ServiceLoader and there is no reference to HdfsConfiguration from DFSClient.Renewer. The best I can think of is to create a static method in HdfsConfiguration that is a noop, but documents what is happening, and have it called from DFSClient.Renewer. Without the method we run the risk of having someone clean up code that is not needed and have these things go away.

      Show
      Robert Joseph Evans added a comment - OK So the issue is that Tokens and Renewal of the tokens has been abstracted away so much that the RM has no direct reference to HDFS at all. The DFSClient.Renewer is loaded using a ServiceLoader and there is no reference to HdfsConfiguration from DFSClient.Renewer. The best I can think of is to create a static method in HdfsConfiguration that is a noop, but documents what is happening, and have it called from DFSClient.Renewer. Without the method we run the risk of having someone clean up code that is not needed and have these things go away.
      Hide
      Robert Joseph Evans added a comment -

      Yes, the issue is with the default HDFS resources not being loaded by the RM. When I added the following to SecurityUtil.java, just as a hack to test this.

        static {
          // adds the default HDFS resources
          Configuration.addDefaultResource("hdfs-default.xml");
          Configuration.addDefaultResource("hdfs-site.xml");
        }
      

      it worked.

      I will try to track down where the proper place is to add a reference to HdfConfiguration where the confs are loaded.

      Show
      Robert Joseph Evans added a comment - Yes, the issue is with the default HDFS resources not being loaded by the RM. When I added the following to SecurityUtil.java, just as a hack to test this. static { // adds the default HDFS resources Configuration.addDefaultResource( "hdfs- default .xml" ); Configuration.addDefaultResource( "hdfs-site.xml" ); } it worked. I will try to track down where the proper place is to add a reference to HdfConfiguration where the confs are loaded.
      Hide
      Robert Joseph Evans added a comment -

      Sorry I have been dealing with other things that have come up, and only now was I able to get back to this issue.

      I have added some debug statements to the Security Utils, and indeed the config option dfs.namenode.kerberos.principal is coming back as null. This does not mean that it is not set. I grepped through the configs and found it set in hdfs-site.xml. I need to see if there is something about the RM that it is not loading this resource, or what.

      Show
      Robert Joseph Evans added a comment - Sorry I have been dealing with other things that have come up, and only now was I able to get back to this issue. I have added some debug statements to the Security Utils, and indeed the config option dfs.namenode.kerberos.principal is coming back as null. This does not mean that it is not set. I grepped through the configs and found it set in hdfs-site.xml. I need to see if there is something about the RM that it is not loading this resource, or what.
      Hide
      Robert Joseph Evans added a comment -

      OK I am seeing it in our clusters too, and I have also verified the configuration. I will dig into this a bit more and see what I can come up with.

      Show
      Robert Joseph Evans added a comment - OK I am seeing it in our clusters too, and I have also verified the configuration. I will dig into this a bit more and see what I can come up with.
      Hide
      Ramya Sunil added a comment -

      How easy is this to reproduce? Do I just need to start running map/reduce jobs, and it starts to show up eventually? Do these need to be long running jobs?

      You just need to start MR jobs and it starts showing up. They dont need to be long running.

      Could you please verify that the following is set in your resource manager configuration "dfs.namenode.kerberos.principal"

      Verified and it is correctly set in the RM configs.

      Show
      Ramya Sunil added a comment - How easy is this to reproduce? Do I just need to start running map/reduce jobs, and it starts to show up eventually? Do these need to be long running jobs? You just need to start MR jobs and it starts showing up. They dont need to be long running. Could you please verify that the following is set in your resource manager configuration "dfs.namenode.kerberos.principal" Verified and it is correctly set in the RM configs.
      Hide
      Robert Joseph Evans added a comment -

      From looking at the code there appear to be a few different possibilities here.

      1. the protocol class is null.
      2. SecurityUtil.securityInfoProviders is some how not loading the AnnotatedSecurityInfo class.
      3. the protocol class is not annotated correctly.
      4. the key returned by the annotation is not set.

      One and Three seem very unlikely, because that code has worked before. Two also seems very unlikely because the AnnotatedSecurityInfo class and the config file for the ServiceLoader is in the same jar as SecurityUtils itself. Found also seems unlikely because the NodeManagers would in theory use the same config and are able to talk to the NN, but this is the most likely culprit. Especially because the RM really does not interact with HDFS at all except here to renew tokens, so if it were misconfigured then the rest of the RM could keep working just fine.

      @Ramya,

      Could you please verify that the following is set in your resource manager configuration "dfs.namenode.kerberos.principal"

      Show
      Robert Joseph Evans added a comment - From looking at the code there appear to be a few different possibilities here. the protocol class is null. SecurityUtil.securityInfoProviders is some how not loading the AnnotatedSecurityInfo class. the protocol class is not annotated correctly. the key returned by the annotation is not set. One and Three seem very unlikely, because that code has worked before. Two also seems very unlikely because the AnnotatedSecurityInfo class and the config file for the ServiceLoader is in the same jar as SecurityUtils itself. Found also seems unlikely because the NodeManagers would in theory use the same config and are able to talk to the NN, but this is the most likely culprit. Especially because the RM really does not interact with HDFS at all except here to renew tokens, so if it were misconfigured then the rest of the RM could keep working just fine. @Ramya, Could you please verify that the following is set in your resource manager configuration "dfs.namenode.kerberos.principal"
      Hide
      Robert Joseph Evans added a comment -

      I am willing to take a look at this. How easy is this to reproduce? Do I just need to start running map/reduce jobs, and it starts to show up eventually? Do these need to be long running jobs? From what I have seen some of the renewal code has hard coded values to renew once a day, but I could be wrong.

      Show
      Robert Joseph Evans added a comment - I am willing to take a look at this. How easy is this to reproduce? Do I just need to start running map/reduce jobs, and it starts to show up eventually? Do these need to be long running jobs? From what I have seen some of the renewal code has hard coded values to renew once a day, but I could be wrong.
      Hide
      Daryn Sharp added a comment -

      No, Mahadev is right, this is not harmless. The interesting part of the log is:

      Couldn't setup connection for <RM principal>/<RMhostname>@<REALM> to null

      There was a similar issue with the JHS where the "to null" error occurred when the service protocol wasn't correctly defined.

      Show
      Daryn Sharp added a comment - No, Mahadev is right, this is not harmless. The interesting part of the log is: Couldn't setup connection for <RM principal>/<RMhostname>@<REALM> to null There was a similar issue with the JHS where the "to null" error occurred when the service protocol wasn't correctly defined.
      Hide
      Mahadev konar added a comment -

      I dont think this is harmless. We should find out whats the issue here.

      Show
      Mahadev konar added a comment - I dont think this is harmless. We should find out whats the issue here.
      Hide
      Ramya Sunil added a comment -

      Exception seen in the RM logs:

      ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:<RM principal>/<RMhostname>@<REALM>  (auth:KERBEROS) cause:java.io.IOException: Failed to specify server's Kerberos principal name
      WARN org.apache.hadoop.ipc.Client: Couldn't setup connection for <RM principal>/<RMhostname>@<REALM>  to null
      ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:<RM principal>/<RMhostname>@<REALM>  (auth:KERBEROS) cause:java.io.IOException: Couldn't setup connection for <RM principal>/<RMhostname>@<REALM>  to null
      ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:<RM principal>/<RMhostname>@<REALM>  (auth:KERBEROS) cause:java.io.IOException: Failed on local exception: java.io.IOException: Couldn't setup connection for <RM principal>/<RMhostname>@<REALM>  to null; Host Details : local host is: "<RM hostname>/<RM IP>"; destination host is: ""<NN hostname>
      ":8020;
      ERROR org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer: Exception renewing tokenIdent: x y z, Kind: HDFS_DELEGATION_TOKEN, Service: <NN IP>:8020. Not rescheduled
      java.io.IOException: Failed on local exception: java.io.IOException: Couldn't setup connection for <RM principal>/<RMhostname>@<REALM>  to null; Host Details : local host is: "<RM hostname>/<RM IP>";
       destination host is: ""<NN hostname>t":8020;
              at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:724)
              at org.apache.hadoop.ipc.Client.call(Client.java:1090)
              at org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:193)
              at $Proxy24.renewDelegationToken(Unknown Source)
              at org.apache.hadoop.hdfs.DFSClient$Renewer.renew(DFSClient.java:619)
              at org.apache.hadoop.security.token.Token.renew(Token.java:312)
              at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$RenewalTimerTask$1.run(DelegationTokenRenewer.java:317)
              at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$RenewalTimerTask$1.run(DelegationTokenRenewer.java:313)
              at java.security.AccessController.doPrivileged(Native Method)
              at javax.security.auth.Subject.doAs(Subject.java:396)
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
              at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$RenewalTimerTask.run(DelegationTokenRenewer.java:312)
              at java.util.TimerThread.mainLoop(Timer.java:512)
              at java.util.TimerThread.run(Timer.java:462)
      Caused by: java.io.IOException: Couldn't setup connection for <RM principal>/<RMhostname>@<REALM>  to null
              at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:528)
              at java.security.AccessController.doPrivileged(Native Method)
              at javax.security.auth.Subject.doAs(Subject.java:396)
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
              at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:500)
              at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:584)
              at org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:206)
              at org.apache.hadoop.ipc.Client.getConnection(Client.java:1196)
              at org.apache.hadoop.ipc.Client.call(Client.java:1066)
              ... 12 more
      Caused by: java.io.IOException: Failed to specify server's Kerberos principal name
              at org.apache.hadoop.security.SaslRpcClient.<init>(SaslRpcClient.java:88)
              at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:407)
              at org.apache.hadoop.ipc.Client$Connection.access$1200(Client.java:206)
              at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:577)
              at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:574)
              at java.security.AccessController.doPrivileged(Native Method)
              at javax.security.auth.Subject.doAs(Subject.java:396)
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
              at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:573)
              ... 15 more
      

      Seems harmless at the moment but need to debug.

      Show
      Ramya Sunil added a comment - Exception seen in the RM logs: ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:<RM principal>/<RMhostname>@<REALM> (auth:KERBEROS) cause:java.io.IOException: Failed to specify server's Kerberos principal name WARN org.apache.hadoop.ipc.Client: Couldn't setup connection for <RM principal>/<RMhostname>@<REALM> to null ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:<RM principal>/<RMhostname>@<REALM> (auth:KERBEROS) cause:java.io.IOException: Couldn't setup connection for <RM principal>/<RMhostname>@<REALM> to null ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:<RM principal>/<RMhostname>@<REALM> (auth:KERBEROS) cause:java.io.IOException: Failed on local exception: java.io.IOException: Couldn't setup connection for <RM principal>/<RMhostname>@<REALM> to null; Host Details : local host is: "<RM hostname>/<RM IP>"; destination host is: ""<NN hostname> ":8020; ERROR org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer: Exception renewing tokenIdent: x y z, Kind: HDFS_DELEGATION_TOKEN, Service: <NN IP>:8020. Not rescheduled java.io.IOException: Failed on local exception: java.io.IOException: Couldn't setup connection for <RM principal>/<RMhostname>@<REALM> to null; Host Details : local host is: "<RM hostname>/<RM IP>"; destination host is: ""<NN hostname>t":8020; at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:724) at org.apache.hadoop.ipc.Client.call(Client.java:1090) at org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:193) at $Proxy24.renewDelegationToken(Unknown Source) at org.apache.hadoop.hdfs.DFSClient$Renewer.renew(DFSClient.java:619) at org.apache.hadoop.security.token.Token.renew(Token.java:312) at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$RenewalTimerTask$1.run(DelegationTokenRenewer.java:317) at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$RenewalTimerTask$1.run(DelegationTokenRenewer.java:313) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177) at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$RenewalTimerTask.run(DelegationTokenRenewer.java:312) at java.util.TimerThread.mainLoop(Timer.java:512) at java.util.TimerThread.run(Timer.java:462) Caused by: java.io.IOException: Couldn't setup connection for <RM principal>/<RMhostname>@<REALM> to null at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:528) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177) at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:500) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:584) at org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:206) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1196) at org.apache.hadoop.ipc.Client.call(Client.java:1066) ... 12 more Caused by: java.io.IOException: Failed to specify server's Kerberos principal name at org.apache.hadoop.security.SaslRpcClient.<init>(SaslRpcClient.java:88) at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:407) at org.apache.hadoop.ipc.Client$Connection.access$1200(Client.java:206) at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:577) at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:574) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:573) ... 15 more Seems harmless at the moment but need to debug.

        People

        • Assignee:
          Robert Joseph Evans
          Reporter:
          Ramya Sunil
        • Votes:
          0 Vote for this issue
          Watchers:
          3 Start watching this issue

          Dates

          • Created:
            Updated:
            Resolved:

            Development