Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-8719

Workaround for kerberos-related log errors upon running any hadoop command on OSX

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Trivial
    • Resolution: Fixed
    • Affects Version/s: 2.0.0-alpha
    • Fix Version/s: 3.0.0-alpha1
    • Component/s: scripts
    • Labels:
      None
    • Environment:

      Mac OS X 10.7, Java 1.6.0_26

    • Hadoop Flags:
      Reviewed

      Description

      When starting Hadoop on OS X 10.7 ("Lion") using start-all.sh, Hadoop logs the following errors:
      2011-07-28 11:45:31.469 java[77427:1a03] Unable to load realm info from SCDynamicStore
      Hadoop does seem to function properly despite this.

      The workaround takes only 10 minutes.

      There are numerous discussions about this:
      google "Unable to load realm mapping info from SCDynamicStore" returns 1770 hits. Each one has many discussions.

      Assume each discussion take only 5 minute, a 10-minute fix can save ~150 hours. This does not count much search of this issue and its solution/workaround, which can easily hit (wasted) thousands of hours!!!

      1. HADOOP-8719.patch
        2 kB
        Jianbin Wei
      2. HADOOP-8719.patch
        2 kB
        Jianbin Wei
      3. HADOOP-8719.patch
        2 kB
        Jianbin Wei
      4. HADOOP-8719.patch
        2 kB
        Jianbin Wei

        Issue Links

          Activity

          Hide
          jianbwei Jianbin Wei added a comment -

          the patch can be ported to other versions.

          Show
          jianbwei Jianbin Wei added a comment - the patch can be ported to other versions.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12541868/HADOOP-8719.patch
          against trunk revision .

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1358//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12541868/HADOOP-8719.patch against trunk revision . -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1358//console This message is automatically generated.
          Hide
          jianbwei Jianbin Wei added a comment -

          After the patch, sbin/[start|stop]-[dfs|yarn].sh do not show the annoying messages any more.

          Show
          jianbwei Jianbin Wei added a comment - After the patch, sbin/ [start|stop] - [dfs|yarn] .sh do not show the annoying messages any more.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12542443/HADOOP-8719.patch
          against trunk revision .

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1359//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12542443/HADOOP-8719.patch against trunk revision . -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1359//console This message is automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12542635/HADOOP-8719.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1365//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1365//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12542635/HADOOP-8719.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1365//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1365//console This message is automatically generated.
          Hide
          jianbwei Jianbin Wei added a comment -

          This changes the options used by hadoop-env.sh and yarn-env.sh. After the patch, I did manual test and verified that sbin/[start|stop]-[dfs|yarn].sh do not show the annoying messages any more.

          Show
          jianbwei Jianbin Wei added a comment - This changes the options used by hadoop-env.sh and yarn-env.sh. After the patch, I did manual test and verified that sbin/ [start|stop] - [dfs|yarn] .sh do not show the annoying messages any more.
          Hide
          qwertymaniac Harsh J added a comment -

          Looks good. One nit: can "macos" be renamed to "MAC_OSX", just for readability/consistency sake?

          +1 otherwise, thanks Jianbin.

          Show
          qwertymaniac Harsh J added a comment - Looks good. One nit: can "macos" be renamed to "MAC_OSX", just for readability/consistency sake? +1 otherwise, thanks Jianbin.
          Hide
          jianbwei Jianbin Wei added a comment -

          rename 'macos' to 'MAC_OSX'

          Show
          jianbwei Jianbin Wei added a comment - rename 'macos' to 'MAC_OSX'
          Hide
          jianbwei Jianbin Wei added a comment -

          Used the style from hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh, the check for cygwin. The one used over there should also be changed for consistency I think.

          Show
          jianbwei Jianbin Wei added a comment - Used the style from hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh, the check for cygwin. The one used over there should also be changed for consistency I think.
          Hide
          jianbwei Jianbin Wei added a comment -

          resubmit the patch with renamed 'macos' to 'MAC_OSX'. Thanks Harsh.

          Show
          jianbwei Jianbin Wei added a comment - resubmit the patch with renamed 'macos' to 'MAC_OSX'. Thanks Harsh.
          Hide
          qwertymaniac Harsh J added a comment -

          Committed revision 1377821 to trunk.

          Thanks very much for your contribution Jianbin!

          Show
          qwertymaniac Harsh J added a comment - Committed revision 1377821 to trunk. Thanks very much for your contribution Jianbin!
          Hide
          hudson Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #2644 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2644/)
          HADOOP-8719. Workaround for kerberos-related log errors upon running any hadoop command on OSX. Contributed by Jianbin Wei. (harsh) (Revision 1377821)

          Result = SUCCESS
          harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1377821
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/hadoop-env.sh
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/conf/yarn-env.sh
          Show
          hudson Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2644 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2644/ ) HADOOP-8719 . Workaround for kerberos-related log errors upon running any hadoop command on OSX. Contributed by Jianbin Wei. (harsh) (Revision 1377821) Result = SUCCESS harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1377821 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/hadoop-env.sh /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/conf/yarn-env.sh
          Hide
          hudson Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #2708 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2708/)
          HADOOP-8719. Workaround for kerberos-related log errors upon running any hadoop command on OSX. Contributed by Jianbin Wei. (harsh) (Revision 1377821)

          Result = SUCCESS
          harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1377821
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/hadoop-env.sh
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/conf/yarn-env.sh
          Show
          hudson Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2708 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2708/ ) HADOOP-8719 . Workaround for kerberos-related log errors upon running any hadoop command on OSX. Contributed by Jianbin Wei. (harsh) (Revision 1377821) Result = SUCCESS harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1377821 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/hadoop-env.sh /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/conf/yarn-env.sh
          Hide
          hudson Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #2673 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2673/)
          HADOOP-8719. Workaround for kerberos-related log errors upon running any hadoop command on OSX. Contributed by Jianbin Wei. (harsh) (Revision 1377821)

          Result = FAILURE
          harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1377821
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/hadoop-env.sh
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/conf/yarn-env.sh
          Show
          hudson Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2673 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2673/ ) HADOOP-8719 . Workaround for kerberos-related log errors upon running any hadoop command on OSX. Contributed by Jianbin Wei. (harsh) (Revision 1377821) Result = FAILURE harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1377821 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/hadoop-env.sh /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/conf/yarn-env.sh
          Hide
          hudson Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1148 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1148/)
          HADOOP-8719. Workaround for kerberos-related log errors upon running any hadoop command on OSX. Contributed by Jianbin Wei. (harsh) (Revision 1377821)

          Result = FAILURE
          harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1377821
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/hadoop-env.sh
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/conf/yarn-env.sh
          Show
          hudson Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1148 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1148/ ) HADOOP-8719 . Workaround for kerberos-related log errors upon running any hadoop command on OSX. Contributed by Jianbin Wei. (harsh) (Revision 1377821) Result = FAILURE harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1377821 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/hadoop-env.sh /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/conf/yarn-env.sh
          Hide
          hudson Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1179 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1179/)
          HADOOP-8719. Workaround for kerberos-related log errors upon running any hadoop command on OSX. Contributed by Jianbin Wei. (harsh) (Revision 1377821)

          Result = FAILURE
          harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1377821
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/hadoop-env.sh
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/conf/yarn-env.sh
          Show
          hudson Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1179 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1179/ ) HADOOP-8719 . Workaround for kerberos-related log errors upon running any hadoop command on OSX. Contributed by Jianbin Wei. (harsh) (Revision 1377821) Result = FAILURE harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1377821 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/hadoop-env.sh /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/conf/yarn-env.sh
          Hide
          wheat9 Haohui Mai added a comment -

          Reopen this issue since it breaks secure set ups running under Mac OS X. Secure cluster won't be able to start with null realm.

          I propose to revert this patch.

          Show
          wheat9 Haohui Mai added a comment - Reopen this issue since it breaks secure set ups running under Mac OS X. Secure cluster won't be able to start with null realm. I propose to revert this patch.
          Hide
          jingzhao Jing Zhao added a comment -

          Looks like this patch makes it impossible to debug a secured cluster in Mac OS X. Harsh J and [~jianbin_wei], could you please comment on this?

          Show
          jingzhao Jing Zhao added a comment - Looks like this patch makes it impossible to debug a secured cluster in Mac OS X. Harsh J and [~jianbin_wei] , could you please comment on this?
          Hide
          qwertymaniac Harsh J added a comment -

          When this was committed, OSX was not a targeted platform for security or native support. If that has changed recently, lets revert this fix over a new JIRA - I see no issues with doing that. The fix here merely got rid of a verbose warning appearing unnecessarily over unsecured pseudo-distributed clusters running on OSX.

          Re-resolving. Thanks!

          Show
          qwertymaniac Harsh J added a comment - When this was committed, OSX was not a targeted platform for security or native support. If that has changed recently, lets revert this fix over a new JIRA - I see no issues with doing that. The fix here merely got rid of a verbose warning appearing unnecessarily over unsecured pseudo-distributed clusters running on OSX. Re-resolving. Thanks!
          Hide
          wheat9 Haohui Mai added a comment -

          Thanks Harsh J for taking care of this.

          Show
          wheat9 Haohui Mai added a comment - Thanks Harsh J for taking care of this.
          Hide
          aw Allen Wittenauer added a comment -

          Howdy. Verifying that what is said to be fixed in 3.x is actually fixed in 3.x and not in 2.x . In that process, I read this comment:

          Looks like this patch makes it impossible to debug a secured cluster in Mac OS X.

          This is incorrect. *-env.sh is meant to be edited by the user. Therefore, a user running security on OS X is expected to remove those bits from the *-env.sh files. Since the time that this patch was committed, HADOOP-9902 has reworked this code:

          a) Fixed support for Mavericks
          b) Made the change in one place and not two
          c) Put a comment above the code that specifically says that users running security should remove it.

          Show
          aw Allen Wittenauer added a comment - Howdy. Verifying that what is said to be fixed in 3.x is actually fixed in 3.x and not in 2.x . In that process, I read this comment: Looks like this patch makes it impossible to debug a secured cluster in Mac OS X. This is incorrect. *-env.sh is meant to be edited by the user. Therefore, a user running security on OS X is expected to remove those bits from the *-env.sh files. Since the time that this patch was committed, HADOOP-9902 has reworked this code: a) Fixed support for Mavericks b) Made the change in one place and not two c) Put a comment above the code that specifically says that users running security should remove it.
          Hide
          lars_francke Lars Francke added a comment -

          I know this is old and as Allen Wittenauer said the exact code has been revamped. That said I think it'd be better to warn Mac users that an error message might occur instead of silently overwriting a property.

          This way at least I spent a day trying to figure out why my property was ignored and others worked.

          Show
          lars_francke Lars Francke added a comment - I know this is old and as Allen Wittenauer said the exact code has been revamped. That said I think it'd be better to warn Mac users that an error message might occur instead of silently overwriting a property. This way at least I spent a day trying to figure out why my property was ignored and others worked.

            People

            • Assignee:
              jianbwei Jianbin Wei
              Reporter:
              jianbwei Jianbin Wei
            • Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development