Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-4427

start-dfs.sh generates malformed ssh command when not running with native libs

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: scripts
    • Labels:
      None

      Description

      After HADOOP-8712 the start-dfs.sh script is generating malformed ssh commands when the native hadoop libraries are not present. This is because hdfs getconf is printing a warning, and that warning is accidentally interpreted as one of the machines to target for ssh.

      Here's an example output of hdfs getconf:

      $ hdfs getconf -namenodes 2>/dev/null
      2013-01-22 21:03:59,543 WARN  util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
      localhost
      

        Issue Links

          Activity

          Hide
          Jason Lowe added a comment -

          Native libs are preferable, but if native libs aren't an option another workaround is to explicitly set hadoop.security.group.mapping to org.apache.hadoop.security.ShellBasedUnixGroupsMapping in core-site.xml.

          Show
          Jason Lowe added a comment - Native libs are preferable, but if native libs aren't an option another workaround is to explicitly set hadoop.security.group.mapping to org.apache.hadoop.security.ShellBasedUnixGroupsMapping in core-site.xml.
          Hide
          Forrest Running added a comment -

          Declaring these variables to force correct version of native libraries seems to fix the issue (yet, a workaround):

          export HADOOP_COMMON_LIB_NATIVE_DIR=$

          {HADOOP_PREFIX}

          /lib/native
          export HADOOP_OPTS="-Djava.library.path=$HADOOP_PREFIX/lib"

          This thread has details.

          Show
          Forrest Running added a comment - Declaring these variables to force correct version of native libraries seems to fix the issue (yet, a workaround): export HADOOP_COMMON_LIB_NATIVE_DIR=$ {HADOOP_PREFIX} /lib/native export HADOOP_OPTS="-Djava.library.path=$HADOOP_PREFIX/lib" This thread has details.
          Hide
          Kevin Mader added a comment -

          I have tried changing hadoop.root.logger to all sorts of different values from FATAL,console to OFF,console and the problem persists. The only, very ugly solution that seems to work now is inserting a " | sed "1 d" " command after the hdfs getconf command, but that is clearly not a very viable long term solution.

          Show
          Kevin Mader added a comment - I have tried changing hadoop.root.logger to all sorts of different values from FATAL,console to OFF,console and the problem persists. The only, very ugly solution that seems to work now is inserting a " | sed "1 d" " command after the hdfs getconf command, but that is clearly not a very viable long term solution.
          Hide
          Todd Lipcon added a comment -

          Maybe we need to set up log4j for this command such that the log4j output goes to stderr instead of stdout?

          Another more general solution might be to get rid of this WARN entirely (change to DEBUG) and instead add an equivalent WARN in the various daemon start messages, as well as an API so dependent projects like HBase and MR can easily issue warnings when native isn't available? I imagine that users will find the "WARN" on every "hadoop fs -ls" type command annoying as well.

          Show
          Todd Lipcon added a comment - Maybe we need to set up log4j for this command such that the log4j output goes to stderr instead of stdout? Another more general solution might be to get rid of this WARN entirely (change to DEBUG) and instead add an equivalent WARN in the various daemon start messages, as well as an API so dependent projects like HBase and MR can easily issue warnings when native isn't available? I imagine that users will find the "WARN" on every "hadoop fs -ls" type command annoying as well.

            People

            • Assignee:
              Robert Parker
              Reporter:
              Jason Lowe
            • Votes:
              1 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:

                Development