Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-11245

HDFS ignores HADOOP_CONF_DIR

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Invalid
    • 3.0.0-alpha1
    • None
    • hdfs
    • None

    • Linux

    Description

      It seems that HDFS on trunk is ignoring HADOOP_CONF_DIR. On branch-2 I could export HADOOP_CONF_DIR and use that to store my hdfs-site.xml and log4j.properties. But on trunk it appears to ignore the environment variable.

      Also, even if hdfs can find the log4j.properties, it doesn't seem interested in opening and loading it.

      On Ubuntu 16.10:

      $ source env.sh
$ cat env.sh 
#!/bin/bash
export JAVA_HOME=/usr/lib/jvm/java-8-oracle
export HADOOP_HOME="$HOME"/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT
export HADOOP_LOG_DIR="$(pwd)/log"
PATH="$HADOOP_HOME"/bin:$PATH
export HADOOP_CLASSPATH=$(hadoop classpath):"$HADOOP_HOME"/share/hadoop/tools/lib/*
export HADOOP_USER_CLASSPATH_FIRST=true

      Then I set the HADOOP_CONF_DIR:

      $ export HADOOP_CONF_DIR="$(pwd)/conf/nn"
$ ls $HADOOP_CONF_DIR
hadoop-env.sh  hdfs-site.xml  log4j.properties

      Now, we try to run a namenode:

      $ hdfs namenode
2016-12-14 14:04:51,193 ERROR [main] namenode.NameNode: Failed to start namenode.
java.lang.IllegalArgumentException: Invalid URI for NameNode address (check fs.defaultFS): file:/// has no authority.
        at org.apache.hadoop.hdfs.DFSUtilClient.getNNAddress(DFSUtilClient.java:648)
        at org.apache.hadoop.hdfs.DFSUtilClient.getNNAddressCheckLogical(DFSUtilClient.java:677)
        at org.apache.hadoop.hdfs.DFSUtilClient.getNNAddress(DFSUtilClient.java:639)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.getRpcServerAddress(NameNode.java:556)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.loginAsNameNodeUser(NameNode.java:687)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:707)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:937)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:916)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1633)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1701)

      This is weird. We have the fs.defaultFS set:

      $ grep -n2 fs.defaultFS $HADOOP_CONF_DIR/hdfs-site.xml
3-<configuration>
4-    <property>
5:        <name>fs.defaultFS</name>
6-        <value>hdfs://localhost:60010</value>
7-    </property>

      So if isn't finding this config. Where is is looking and finding file:///?

      $ strace -f -eopen,stat hdfs namenode 2>&1 | grep hdfs-site.xml
[pid 16271] stat("/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/common/jdiff/hdfs-site.xml", 0x7f05eb6d21d0) = -1 ENOENT (No such file or directory)
[pid 16271] stat("/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/common/lib/hdfs-site.xml", 0x7f05eb6d21d0) = -1 ENOENT (No such file or directory)
[pid 16271] stat("/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/common/sources/hdfs-site.xml", 0x7f05eb6d21d0) = -1 ENOENT (No such file or directory)
[pid 16271] stat("/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/common/templates/hdfs-site.xml", 0x7f05eb6d21d0) = -1 ENOENT (No such file or directory)
[pid 16271] stat("/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/common/webapps/hdfs-site.xml", 0x7f05eb6d21d0) = -1 ENOENT (No such file or directory)
[pid 16271] stat("/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/hdfs/jdiff/hdfs-site.xml", 0x7f05eb6d21d0) = -1 ENOENT (No such file or directory)
[pid 16271] stat("/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/hdfs/lib/hdfs-site.xml", 0x7f05eb6d21d0) = -1 ENOENT (No such file or directory)
[pid 16271] stat("/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/hdfs/sources/hdfs-site.xml", 0x7f05eb6d21d0) = -1 ENOENT (No such file or directory)
[pid 16271] stat("/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/hdfs/templates/hdfs-site.xml", {st_mode=S_IFREG|0664, st_size=775, ...}) = 0
[pid 16271] stat("/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/hdfs/templates/hdfs-site.xml", {st_mode=S_IFREG|0664, st_size=775, ...}) = 0
[pid 16271] open("/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/hdfs/templates/hdfs-site.xml", O_RDONLY) = 218
[pid 16271] stat("/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/common/jdiff/hdfs-site.xml", 0x7f05eb6d17e0) = -1 ENOENT (No such file or directory)
[pid 16271] stat("/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/common/lib/hdfs-site.xml", 0x7f05eb6d17e0) = -1 ENOENT (No such file or directory)
[pid 16271] stat("/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/common/sources/hdfs-site.xml", 0x7f05eb6d17e0) = -1 ENOENT (No such file or directory)
[pid 16271] stat("/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/common/templates/hdfs-site.xml", 0x7f05eb6d17e0) = -1 ENOENT (No such file or directory)
[pid 16271] stat("/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/common/webapps/hdfs-site.xml", 0x7f05eb6d17e0) = -1 ENOENT (No such file or directory)
[pid 16271] stat("/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/hdfs/jdiff/hdfs-site.xml", 0x7f05eb6d17e0) = -1 ENOENT (No such file or directory)
[pid 16271] stat("/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/hdfs/lib/hdfs-site.xml", 0x7f05eb6d17e0) = -1 ENOENT (No such file or directory)
[pid 16271] stat("/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/hdfs/sources/hdfs-site.xml", 0x7f05eb6d17e0) = -1 ENOENT (No such file or directory)
[pid 16271] stat("/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/hdfs/templates/hdfs-site.xml", {st_mode=S_IFREG|0664, st_size=775, ...}) = 0
[pid 16271] stat("/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/hdfs/templates/hdfs-site.xml", {st_mode=S_IFREG|0664, st_size=775, ...}) = 0
[pid 16271] open("/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/hdfs/templates/hdfs-site.xml", O_RDONLY) = 218
[pid 16271] stat("/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/common/jdiff/hdfs-site.xml", 0x7f05eb6d2070) = -1 ENOENT (No such file or directory)
[pid 16271] stat("/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/common/lib/hdfs-site.xml", 0x7f05eb6d2070) = -1 ENOENT (No such file or directory)
[pid 16271] stat("/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/common/sources/hdfs-site.xml", 0x7f05eb6d2070) = -1 ENOENT (No such file or directory)
[pid 16271] stat("/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/common/templates/hdfs-site.xml", 0x7f05eb6d2070) = -1 ENOENT (No such file or directory)
[pid 16271] stat("/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/common/webapps/hdfs-site.xml", 0x7f05eb6d2070) = -1 ENOENT (No such file or directory)
[pid 16271] stat("/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/hdfs/jdiff/hdfs-site.xml", 0x7f05eb6d2070) = -1 ENOENT (No such file or directory)
[pid 16271] stat("/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/hdfs/lib/hdfs-site.xml", 0x7f05eb6d2070) = -1 ENOENT (No such file or directory)
[pid 16271] stat("/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/hdfs/sources/hdfs-site.xml", 0x7f05eb6d2070) = -1 ENOENT (No such file or directory)
[pid 16271] stat("/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/hdfs/templates/hdfs-site.xml", {st_mode=S_IFREG|0664, st_size=775, ...}) = 0
[pid 16271] stat("/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/hdfs/templates/hdfs-site.xml", {st_mode=S_IFREG|0664, st_size=775, ...}) = 0
[pid 16271] open("/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/hdfs/templates/hdfs-site.xml", O_RDONLY) = 218

      So it's ignoring HADOOP_CONF_DIR. We can work around it using -conf $(pwd)/conf/nn/hdfs-site.xml:

      $ strace -f -eopen,stat hdfs namenode -conf $(pwd)/conf/nn/hdfs-site.xml 2>&1 | grep hdfs-site.xml
[pid 16493] stat("/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/common/jdiff/hdfs-site.xml", 0x7f9f60afb1d0) = -1 ENOENT (No such file or directory)
[pid 16493] stat("/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/common/lib/hdfs-site.xml", 0x7f9f60afb1d0) = -1 ENOENT (No such file or directory)
[pid 16493] stat("/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/common/sources/hdfs-site.xml", 0x7f9f60afb1d0) = -1 ENOENT (No such file or directory)
[pid 16493] stat("/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/common/templates/hdfs-site.xml", 0x7f9f60afb1d0) = -1 ENOENT (No such file or directory)
[pid 16493] stat("/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/common/webapps/hdfs-site.xml", 0x7f9f60afb1d0) = -1 ENOENT (No such file or directory)
[pid 16493] stat("/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/hdfs/jdiff/hdfs-site.xml", 0x7f9f60afb1d0) = -1 ENOENT (No such file or directory)
[pid 16493] stat("/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/hdfs/lib/hdfs-site.xml", 0x7f9f60afb1d0) = -1 ENOENT (No such file or directory)
[pid 16493] stat("/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/hdfs/sources/hdfs-site.xml", 0x7f9f60afb1d0) = -1 ENOENT (No such file or directory)
[pid 16493] stat("/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/hdfs/templates/hdfs-site.xml", {st_mode=S_IFREG|0664, st_size=775, ...}) = 0
[pid 16493] stat("/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/hdfs/templates/hdfs-site.xml", {st_mode=S_IFREG|0664, st_size=775, ...}) = 0
[pid 16493] open("/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/share/hadoop/hdfs/templates/hdfs-site.xml", O_RDONLY) = 218
[pid 16493] stat("/home/ehigg90120/src/hadoop-run/tutorial/conf/nn/hdfs-site.xml", {st_mode=S_IFREG|0644, st_size=2107, ...}) = 0
[pid 16493] open("/home/ehigg90120/src/hadoop-run/tutorial/conf/nn/hdfs-site.xml", O_RDONLY) = 218

------8<------

      Great! However, it's not finding my log4j.properties for some reason. This is annoying because hdfs isn't printing anything or logging anywhere. Where is it looking?

      $ strace -f hdfs namenode -conf $(pwd)/conf/nn/hdfs-site.xml 2>&1 | grep log4j.properties
stat("/home/ehigg90120/src/hadoop-run/tutorial/conf/nn/log4j.properties", {st_mode=S_IFREG|0644, st_size=13641, ...}) = 0

      It found it, but it only statted it. It never opened it! So it seems there's at least one bug here where log4j.properties is being ignored. But shouldn't HADOOP_OPTS be set and configuring it to print to the console and to my log dir?

      $ hdfs --debug namenode -conf $(pwd)/conf/nn/hdfs-site.xml 2>&1 | grep HADOOP_OPTS
DEBUG: HADOOP_OPTS accepted -Dhdfs.audit.logger=INFO,NullAppender
DEBUG: Appending HDFS_NAMENODE_OPTS onto HADOOP_OPTS
DEBUG: HADOOP_OPTS accepted -Dyarn.log.dir=/home/ehigg90120/src/hadoop-run/tutorial/log
DEBUG: HADOOP_OPTS accepted -Dyarn.log.file=hadoop.log
DEBUG: HADOOP_OPTS accepted -Dyarn.home.dir=/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT
DEBUG: HADOOP_OPTS accepted -Dyarn.root.logger=INFO,console
DEBUG: HADOOP_OPTS accepted -Djava.library.path=/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/lib/native
DEBUG: HADOOP_OPTS accepted -Dhadoop.log.dir=/home/ehigg90120/src/hadoop-run/tutorial/log
DEBUG: HADOOP_OPTS accepted -Dhadoop.log.file=hadoop.log
DEBUG: HADOOP_OPTS accepted -Dhadoop.home.dir=/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT
DEBUG: HADOOP_OPTS accepted -Dhadoop.id.str=ehigg90120
DEBUG: HADOOP_OPTS accepted -Dhadoop.root.logger=INFO,console
DEBUG: HADOOP_OPTS accepted -Dhadoop.policy.file=hadoop-policy.xml
DEBUG: HADOOP_OPTS declined -Dhadoop.security.logger=INFO,NullAppender
DEBUG: Final HADOOP_OPTS: -Djava.net.preferIPv4Stack=true -Dhdfs.audit.logger=INFO,NullAppender -Dhadoop.security.logger=INFO,RFAS -Dyarn.log.dir=/home/ehigg90120/src/hadoop-run/tutorial/log -Dyarn.log.file=hadoop.log -Dyarn.home.dir=/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT -Dyarn.root.logger=INFO,console -Djava.library.path=/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT/lib/native -Dhadoop.log.dir=/home/ehigg90120/src/hadoop-run/tutorial/log -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/home/ehigg90120/src/hadoop/hadoop-dist/target/hadoop-3.0.0-alpha2-SNAPSHOT -Dhadoop.id.str=ehigg90120 -Dhadoop.root.logger=INFO,console -Dhadoop.policy.file=hadoop-policy.xml

      So it seems it is being configured and passed to the namenode. It's just not obeying it as far as I can see.

      So maybe there are two possibly related bugs:

      1. HADOOP_CONF_DIR is ignored
      2. The logger is not using log4j.properties or the command line. I would expect it to use the log4j.properties in the HADOOP_CONF_DIR.

      I feel like I must be misunderstanding something since this seems like a pretty big issue but I didn't find any open tickets about it or any tickets describing a new way of configuring clusters.

      Attachments

        Issue Links

          is related to

          New Feature - A new feature of the product, which has yet to be developed. HADOOP-9902 Shell script rewrite

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              Unassigned Unassigned
              ehiggs Ewan Higgs
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: