Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-11245

HDFS ignores HADOOP_CONF_DIR

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Invalid
    • Affects Version/s: 3.0.0-alpha1
    • Fix Version/s: None
    • Component/s: hdfs
    • Labels:
      None
    • Environment:

      Linux

      Description

      It seems that HDFS on trunk is ignoring HADOOP_CONF_DIR. On branch-2 I could export HADOOP_CONF_DIR and use that to store my hdfs-site.xml and log4j.properties. But on trunk it appears to ignore the environment variable.

      Also, even if hdfs can find the log4j.properties, it doesn't seem interested in opening and loading it.

      On Ubuntu 16.10:

      $ source env.sh
      $ cat env.sh 
      #!/bin/bash
      export JAVA_HOME=/usr/lib/jvm/java-8-oracle
      export HADOOP_HOME="$HOME"/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT
      export HADOOP_LOG_DIR="$(pwd)/log"
      PATH="$HADOOP_HOME"/bin:$PATH
      export HADOOP_CLASSPATH=$(hadoop classpath):"$HADOOP_HOME"/share/hadoop/tools/lib/*
      export HADOOP_USER_CLASSPATH_FIRST=true
      

      Then I set the HADOOP_CONF_DIR:

      $ export HADOOP_CONF_DIR="$(pwd)/conf/nn"
      $ ls $HADOOP_CONF_DIR
      hadoop-env.sh  hdfs-site.xml  log4j.properties
      

      Now, we try to run a namenode:

      $ hdfs namenode
      2016-12-14 14:04:51,193 ERROR [main] namenode.NameNode: Failed to start namenode.
      java.lang.IllegalArgumentException: Invalid URI for NameNode address (check fs.defaultFS): file:/// has no authority.
              at org.apache.hadoop.hdfs.DFSUtilClient.getNNAddress(DFSUtilClient.java:648)
              at org.apache.hadoop.hdfs.DFSUtilClient.getNNAddressCheckLogical(DFSUtilClient.java:677)
              at org.apache.hadoop.hdfs.DFSUtilClient.getNNAddress(DFSUtilClient.java:639)
              at org.apache.hadoop.hdfs.server.namenode.NameNode.getRpcServerAddress(NameNode.java:556)
              at org.apache.hadoop.hdfs.server.namenode.NameNode.loginAsNameNodeUser(NameNode.java:687)
              at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:707)
              at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:937)
              at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:916)
              at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1633)
              at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1701)
      

      This is weird. We have the fs.defaultFS set:

      $ grep -n2 fs.defaultFS $HADOOP_CONF_DIR/hdfs-site.xml
      3-<configuration>
      4-    <property>
      5:        <name>fs.defaultFS</name>
      6-        <value>hdfs://localhost:60010</value>
      7-    </property>
      

      So if isn't finding this config. Where is is looking and finding file:///?

      $ strace -f -eopen,stat hdfs namenode 2>&1 | grep hdfs-site.xml
      [pid 16271] stat("/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/common/jdiff/hdfs-site.xml", 0x7f05eb6d21d0) = -1 ENOENT (No such file or directory)
      [pid 16271] stat("/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/common/lib/hdfs-site.xml", 0x7f05eb6d21d0) = -1 ENOENT (No such file or directory)
      [pid 16271] stat("/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/common/sources/hdfs-site.xml", 0x7f05eb6d21d0) = -1 ENOENT (No such file or directory)
      [pid 16271] stat("/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/common/templates/hdfs-site.xml", 0x7f05eb6d21d0) = -1 ENOENT (No such file or directory)
      [pid 16271] stat("/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/common/webapps/hdfs-site.xml", 0x7f05eb6d21d0) = -1 ENOENT (No such file or directory)
      [pid 16271] stat("/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/hdfs/jdiff/hdfs-site.xml", 0x7f05eb6d21d0) = -1 ENOENT (No such file or directory)
      [pid 16271] stat("/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/hdfs/lib/hdfs-site.xml", 0x7f05eb6d21d0) = -1 ENOENT (No such file or directory)
      [pid 16271] stat("/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/hdfs/sources/hdfs-site.xml", 0x7f05eb6d21d0) = -1 ENOENT (No such file or directory)
      [pid 16271] stat("/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/hdfs/templates/hdfs-site.xml", {st_mode=S_IFREG|0664, st_size=775, ...}) = 0
      [pid 16271] stat("/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/hdfs/templates/hdfs-site.xml", {st_mode=S_IFREG|0664, st_size=775, ...}) = 0
      [pid 16271] open("/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/hdfs/templates/hdfs-site.xml", O_RDONLY) = 218
      [pid 16271] stat("/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/common/jdiff/hdfs-site.xml", 0x7f05eb6d17e0) = -1 ENOENT (No such file or directory)
      [pid 16271] stat("/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/common/lib/hdfs-site.xml", 0x7f05eb6d17e0) = -1 ENOENT (No such file or directory)
      [pid 16271] stat("/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/common/sources/hdfs-site.xml", 0x7f05eb6d17e0) = -1 ENOENT (No such file or directory)
      [pid 16271] stat("/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/common/templates/hdfs-site.xml", 0x7f05eb6d17e0) = -1 ENOENT (No such file or directory)
      [pid 16271] stat("/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/common/webapps/hdfs-site.xml", 0x7f05eb6d17e0) = -1 ENOENT (No such file or directory)
      [pid 16271] stat("/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/hdfs/jdiff/hdfs-site.xml", 0x7f05eb6d17e0) = -1 ENOENT (No such file or directory)
      [pid 16271] stat("/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/hdfs/lib/hdfs-site.xml", 0x7f05eb6d17e0) = -1 ENOENT (No such file or directory)
      [pid 16271] stat("/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/hdfs/sources/hdfs-site.xml", 0x7f05eb6d17e0) = -1 ENOENT (No such file or directory)
      [pid 16271] stat("/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/hdfs/templates/hdfs-site.xml", {st_mode=S_IFREG|0664, st_size=775, ...}) = 0
      [pid 16271] stat("/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/hdfs/templates/hdfs-site.xml", {st_mode=S_IFREG|0664, st_size=775, ...}) = 0
      [pid 16271] open("/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/hdfs/templates/hdfs-site.xml", O_RDONLY) = 218
      [pid 16271] stat("/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/common/jdiff/hdfs-site.xml", 0x7f05eb6d2070) = -1 ENOENT (No such file or directory)
      [pid 16271] stat("/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/common/lib/hdfs-site.xml", 0x7f05eb6d2070) = -1 ENOENT (No such file or directory)
      [pid 16271] stat("/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/common/sources/hdfs-site.xml", 0x7f05eb6d2070) = -1 ENOENT (No such file or directory)
      [pid 16271] stat("/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/common/templates/hdfs-site.xml", 0x7f05eb6d2070) = -1 ENOENT (No such file or directory)
      [pid 16271] stat("/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/common/webapps/hdfs-site.xml", 0x7f05eb6d2070) = -1 ENOENT (No such file or directory)
      [pid 16271] stat("/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/hdfs/jdiff/hdfs-site.xml", 0x7f05eb6d2070) = -1 ENOENT (No such file or directory)
      [pid 16271] stat("/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/hdfs/lib/hdfs-site.xml", 0x7f05eb6d2070) = -1 ENOENT (No such file or directory)
      [pid 16271] stat("/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/hdfs/sources/hdfs-site.xml", 0x7f05eb6d2070) = -1 ENOENT (No such file or directory)
      [pid 16271] stat("/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/hdfs/templates/hdfs-site.xml", {st_mode=S_IFREG|0664, st_size=775, ...}) = 0
      [pid 16271] stat("/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/hdfs/templates/hdfs-site.xml", {st_mode=S_IFREG|0664, st_size=775, ...}) = 0
      [pid 16271] open("/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/hdfs/templates/hdfs-site.xml", O_RDONLY) = 218
      

      So it's ignoring HADOOP_CONF_DIR. We can work around it using -conf $(pwd)/conf/nn/hdfs-site.xml:

      $ strace -f -eopen,stat hdfs namenode -conf $(pwd)/conf/nn/hdfs-site.xml 2>&1 | grep hdfs-site.xml
      [pid 16493] stat("/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/common/jdiff/hdfs-site.xml", 0x7f9f60afb1d0) = -1 ENOENT (No such file or directory)
      [pid 16493] stat("/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/common/lib/hdfs-site.xml", 0x7f9f60afb1d0) = -1 ENOENT (No such file or directory)
      [pid 16493] stat("/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/common/sources/hdfs-site.xml", 0x7f9f60afb1d0) = -1 ENOENT (No such file or directory)
      [pid 16493] stat("/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/common/templates/hdfs-site.xml", 0x7f9f60afb1d0) = -1 ENOENT (No such file or directory)
      [pid 16493] stat("/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/common/webapps/hdfs-site.xml", 0x7f9f60afb1d0) = -1 ENOENT (No such file or directory)
      [pid 16493] stat("/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/hdfs/jdiff/hdfs-site.xml", 0x7f9f60afb1d0) = -1 ENOENT (No such file or directory)
      [pid 16493] stat("/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/hdfs/lib/hdfs-site.xml", 0x7f9f60afb1d0) = -1 ENOENT (No such file or directory)
      [pid 16493] stat("/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/hdfs/sources/hdfs-site.xml", 0x7f9f60afb1d0) = -1 ENOENT (No such file or directory)
      [pid 16493] stat("/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/hdfs/templates/hdfs-site.xml", {st_mode=S_IFREG|0664, st_size=775, ...}) = 0
      [pid 16493] stat("/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/hdfs/templates/hdfs-site.xml", {st_mode=S_IFREG|0664, st_size=775, ...}) = 0
      [pid 16493] open("/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/hdfs/templates/hdfs-site.xml", O_RDONLY) = 218
      [pid 16493] stat("/home/ehigg90120/src/hadoop-run/tutorial/conf/nn/hdfs-site.xml", {st_mode=S_IFREG|0644, st_size=2107, ...}) = 0
      [pid 16493] open("/home/ehigg90120/src/hadoop-run/tutorial/conf/nn/hdfs-site.xml", O_RDONLY) = 218
      
      ------8<------
      

      Great! However, it's not finding my log4j.properties for some reason. This is annoying because hdfs isn't printing anything or logging anywhere. Where is it looking?

      $ strace -f hdfs namenode -conf $(pwd)/conf/nn/hdfs-site.xml 2>&1 | grep log4j.properties
      stat("/home/ehigg90120/src/hadoop-run/tutorial/conf/nn/log4j.properties", {st_mode=S_IFREG|0644, st_size=13641, ...}) = 0
      

      It found it, but it only statted it. It never opened it! So it seems there's at least one bug here where log4j.properties is being ignored. But shouldn't HADOOP_OPTS be set and configuring it to print to the console and to my log dir?

      $ hdfs --debug namenode -conf $(pwd)/conf/nn/hdfs-site.xml 2>&1 | grep HADOOP_OPTS
      DEBUG: HADOOP_OPTS accepted -Dhdfs.audit.logger=INFO,NullAppender
      DEBUG: Appending HDFS_NAMENODE_OPTS onto HADOOP_OPTS
      DEBUG: HADOOP_OPTS accepted -Dyarn.log.dir=/home/ehigg90120/src/hadoop-run/tutorial/log
      DEBUG: HADOOP_OPTS accepted -Dyarn.log.file=hadoop.log
      DEBUG: HADOOP_OPTS accepted -Dyarn.home.dir=/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT
      DEBUG: HADOOP_OPTS accepted -Dyarn.root.logger=INFO,console
      DEBUG: HADOOP_OPTS accepted -Djava.library.path=/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/lib/native
      DEBUG: HADOOP_OPTS accepted -Dhadoop.log.dir=/home/ehigg90120/src/hadoop-run/tutorial/log
      DEBUG: HADOOP_OPTS accepted -Dhadoop.log.file=hadoop.log
      DEBUG: HADOOP_OPTS accepted -Dhadoop.home.dir=/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT
      DEBUG: HADOOP_OPTS accepted -Dhadoop.id.str=ehigg90120
      DEBUG: HADOOP_OPTS accepted -Dhadoop.root.logger=INFO,console
      DEBUG: HADOOP_OPTS accepted -Dhadoop.policy.file=hadoop-policy.xml
      DEBUG: HADOOP_OPTS declined -Dhadoop.security.logger=INFO,NullAppender
      DEBUG: Final HADOOP_OPTS: -Djava.net.preferIPv4Stack=true -Dhdfs.audit.logger=INFO,NullAppender -Dhadoop.security.logger=INFO,RFAS -Dyarn.log.dir=/home/ehigg90120/src/hadoop-run/tutorial/log -Dyarn.log.file=hadoop.log -Dyarn.home.dir=/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT -Dyarn.root.logger=INFO,console -Djava.library.path=/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/lib/native -Dhadoop.log.dir=/home/ehigg90120/src/hadoop-run/tutorial/log -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT -Dhadoop.id.str=ehigg90120 -Dhadoop.root.logger=INFO,console -Dhadoop.policy.file=hadoop-policy.xml
      

      So it seems it is being configured and passed to the namenode. It's just not obeying it as far as I can see.

      So maybe there are two possibly related bugs:

      1. HADOOP_CONF_DIR is ignored
      2. The logger is not using log4j.properties or the command line. I would expect it to use the log4j.properties in the HADOOP_CONF_DIR.

      I feel like I must be misunderstanding something since this seems like a pretty big issue but I didn't find any open tickets about it or any tickets describing a new way of configuring clusters.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                ehiggs Ewan Higgs
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: