Hadoop Common
  1. Hadoop Common
  2. HADOOP-7229

Absolute path to kinit in auto-renewal thread

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.21.0, 0.22.0, 0.23.0
    • Fix Version/s: 0.22.0
    • Component/s: security
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Hide
      When Hadoop's Kerberos integration is enabled, it is now required that either {{kinit}} be on the path for user accounts running the Hadoop client, or that the {{hadoop.kerberos.kinit.command}} configuration option be manually set to the absolute path to {{kinit}}.
      Show
      When Hadoop's Kerberos integration is enabled, it is now required that either {{kinit}} be on the path for user accounts running the Hadoop client, or that the {{hadoop.kerberos.kinit.command}} configuration option be manually set to the absolute path to {{kinit}}.

      Description

      In the auto-renewal thread for Kerberos credentials in UserGroupInformation, the path to kinit is defaulted to /usr/kerberos/bin/kinit. This is the default path to kinit on RHEL/CentOS for MIT krb5, but not on Debian/Ubuntu (and perhaps others OSes.)

      1. hadoop-7229.2.patch
        1 kB
        Aaron T. Myers
      2. hadoop-7229.1.patch
        1 kB
        Aaron T. Myers
      3. hadoop-7229.0.patch
        0.7 kB
        Aaron T. Myers

        Activity

        Hide
        Aaron T. Myers added a comment -

        Per an offline suggestion from Eli, I'm amending the release note to make it clear that this is only necessary to configure when security is enabled.

        Show
        Aaron T. Myers added a comment - Per an offline suggestion from Eli, I'm amending the release note to make it clear that this is only necessary to configure when security is enabled.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk #666 (See https://builds.apache.org/hudson/job/Hadoop-Common-trunk/666/)
        HADOOP-7229. Do not default to an absolute path for kinit in Kerberos auto-renewal thread. Contributed by Aaron T. Myers.

        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk #666 (See https://builds.apache.org/hudson/job/Hadoop-Common-trunk/666/ ) HADOOP-7229 . Do not default to an absolute path for kinit in Kerberos auto-renewal thread. Contributed by Aaron T. Myers.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-22-branch #40 (See https://builds.apache.org/hudson/job/Hadoop-Common-22-branch/40/)
        HADOOP-7229. Do not default to an absolute path for kinit in Kerberos auto-renewal thread. Contributed by Aaron T. Myers.

        Show
        Hudson added a comment - Integrated in Hadoop-Common-22-branch #40 (See https://builds.apache.org/hudson/job/Hadoop-Common-22-branch/40/ ) HADOOP-7229 . Do not default to an absolute path for kinit in Kerberos auto-renewal thread. Contributed by Aaron T. Myers.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #560 (See https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/560/)
        HADOOP-7229. Do not default to an absolute path for kinit in Kerberos auto-renewal thread. Contributed by Aaron T. Myers.

        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #560 (See https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/560/ ) HADOOP-7229 . Do not default to an absolute path for kinit in Kerberos auto-renewal thread. Contributed by Aaron T. Myers.
        Hide
        Todd Lipcon added a comment -

        Committed to 0.22 and 0.23

        Show
        Todd Lipcon added a comment - Committed to 0.22 and 0.23
        Hide
        Todd Lipcon added a comment -

        +1, thanks for the updates, Aaron.

        Show
        Todd Lipcon added a comment - +1, thanks for the updates, Aaron.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12476946/hadoop-7229.2.patch
        against trunk revision 1095121.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        +1 system test framework. The patch passed system test framework compile.

        Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/370//testReport/
        Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/370//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/370//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12476946/hadoop-7229.2.patch against trunk revision 1095121. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/370//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/370//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/370//console This message is automatically generated.
        Hide
        Aaron T. Myers added a comment -

        Updated release note to reflect Todd's comments.

        Show
        Aaron T. Myers added a comment - Updated release note to reflect Todd's comments.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12476946/hadoop-7229.2.patch
        against trunk revision 1095121.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        +1 system test framework. The patch passed system test framework compile.

        Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/369//testReport/
        Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/369//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/369//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12476946/hadoop-7229.2.patch against trunk revision 1095121. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/369//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/369//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/369//console This message is automatically generated.
        Hide
        Aaron T. Myers added a comment -

        Good catch, Todd. Updated patch.

        Show
        Aaron T. Myers added a comment - Good catch, Todd. Updated patch.
        Hide
        Todd Lipcon added a comment -

        Hey Aaron. Sorry, one nit – the renewal thread is explicitly not used for keytab-based logins:

              if (user.getAuthenticationMethod() == AuthenticationMethod.KERBEROS &&
                  !isKeytab) {
                Thread t = new Thread(new Runnable() {
        

        So the docs should not reference NN/JT. Typically the use case is for users running long-running processes which need to interact with Hadoop over the course of many hours without the user manually renewing the ticket.

        Show
        Todd Lipcon added a comment - Hey Aaron. Sorry, one nit – the renewal thread is explicitly not used for keytab-based logins: if (user.getAuthenticationMethod() == AuthenticationMethod.KERBEROS && !isKeytab) { Thread t = new Thread ( new Runnable () { So the docs should not reference NN/JT. Typically the use case is for users running long-running processes which need to interact with Hadoop over the course of many hours without the user manually renewing the ticket.
        Hide
        Aaron T. Myers added a comment -

        Thanks for the comments, Todd and Owen. Updated patch to address comments.

        Show
        Aaron T. Myers added a comment - Thanks for the comments, Todd and Owen. Updated patch to address comments.
        Hide
        Owen O'Malley added a comment -

        +1 with the addition of the option to the core-default.xml

        Show
        Owen O'Malley added a comment - +1 with the addition of the option to the core-default.xml
        Hide
        Todd Lipcon added a comment -

        While we're at it, would you mind adding the hadoop.kerberos.kinit.command config option to core-default.xml? Now that it's mentioned by the release notes, I think it's worth putting in there with an explanation that it should be overridden if not on the PATH

        Show
        Todd Lipcon added a comment - While we're at it, would you mind adding the hadoop.kerberos.kinit.command config option to core-default.xml? Now that it's mentioned by the release notes, I think it's worth putting in there with an explanation that it should be overridden if not on the PATH
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12476571/hadoop-7229.0.patch
        against trunk revision 1094750.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        +1 system test framework. The patch passed system test framework compile.

        Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/352//testReport/
        Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/352//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/352//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12476571/hadoop-7229.0.patch against trunk revision 1094750. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/352//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/352//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/352//console This message is automatically generated.
        Hide
        Aaron T. Myers added a comment -

        Thanks a lot for the review/comments, Owen. I've updated the release note and marked this as an incompatible change.

        Show
        Aaron T. Myers added a comment - Thanks a lot for the review/comments, Owen. I've updated the release note and marked this as an incompatible change.
        Hide
        Owen O'Malley added a comment -

        This is at the very least an incompatible change, since any site that doesn't have kinit on the hdfs and mapred account's path will break.

        That said, I think it is the right direction.

        Please update the release note field in the jira to describe that.

        Show
        Owen O'Malley added a comment - This is at the very least an incompatible change, since any site that doesn't have kinit on the hdfs and mapred account's path will break. That said, I think it is the right direction. Please update the release note field in the jira to describe that.
        Hide
        Aaron T. Myers added a comment -

        Trivial patch to remove the absolute path to kinit, and instead rely on kinit being in the PATH of the process.

        Show
        Aaron T. Myers added a comment - Trivial patch to remove the absolute path to kinit , and instead rely on kinit being in the PATH of the process.

          People

          • Assignee:
            Aaron T. Myers
            Reporter:
            Aaron T. Myers
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development