Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-11058

Missing HADOOP_CONF_DIR generates strange results

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.0-alpha1
    • Component/s: scripts
    • Labels:

      Description

      If HADOOP_CONF_DIR is defined but points to a directory that either doesn't exist or isn't actually a viable configuration directory, all sorts of weird things happen, especially for logging. The shell code should do a better job of verifying the directory is valid and exit if it detects if it is broken in some way.

      1. HADOOP-11058.1.patch
        0.6 kB
        Masatake Iwasaki
      2. HADOOP-11058.2.patch
        1.0 kB
        Masatake Iwasaki
      3. HADOOP-11058.3.patch
        1 kB
        Masatake Iwasaki
      4. HADOOP-11058.4.patch
        1 kB
        Masatake Iwasaki

        Issue Links

          Activity

          Hide
          aw Allen Wittenauer added a comment -

          This is also true for branch-2, so marking this as an improvement.

          In particular, I noticed logging going haywire. As mentioned in HADOOP-10879, log4j settings should almost certainly be found.

          Show
          aw Allen Wittenauer added a comment - This is also true for branch-2, so marking this as an improvement. In particular, I noticed logging going haywire. As mentioned in HADOOP-10879 , log4j settings should almost certainly be found.
          Hide
          iwasakims Masatake Iwasaki added a comment -

          I added warning if log4j.properties is not found on daemon startup because log appender is set to not console but RFA if --daemon is used.

          Show
          iwasakims Masatake Iwasaki added a comment - I added warning if log4j.properties is not found on daemon startup because log appender is set to not console but RFA if --daemon is used.
          Hide
          iwasakims Masatake Iwasaki added a comment -

          Basically commands work by default config values even if there are no files in HADOOP_CONF_DIR (except for FsShell. Only FsShell throws RuntimeException if resources such as core-site.xml and hdfs-site.xml are not found. See HADOOP-11405).

          Show
          iwasakims Masatake Iwasaki added a comment - Basically commands work by default config values even if there are no files in HADOOP_CONF_DIR (except for FsShell. Only FsShell throws RuntimeException if resources such as core-site.xml and hdfs-site.xml are not found. See HADOOP-11405 ).
          Hide
          iwasakims Masatake Iwasaki added a comment -

          We do not need to change the code for finding conf dir by hadoop-env.sh in order to move HADOOP-10879 forward. It is only for backward compatibility.

          Show
          iwasakims Masatake Iwasaki added a comment - We do not need to change the code for finding conf dir by hadoop-env.sh in order to move HADOOP-10879 forward. It is only for backward compatibility.
          Hide
          aw Allen Wittenauer added a comment -

          After thinking about this for a few days, I think this is a fine solution. A warning that log4j bits are missing for daemons is a huge indicator that things are wrong. IMO, we should expand the message to indicate that $HADOOP_CONF_DIR may be incomplete or broken or whatever beyond just that the file is missing though.

          Also, we need this same code in the secure version of the wrapper as well.

          Show
          aw Allen Wittenauer added a comment - After thinking about this for a few days, I think this is a fine solution. A warning that log4j bits are missing for daemons is a huge indicator that things are wrong. IMO, we should expand the message to indicate that $HADOOP_CONF_DIR may be incomplete or broken or whatever beyond just that the file is missing though. Also, we need this same code in the secure version of the wrapper as well.
          Hide
          iwasakims Masatake Iwasaki added a comment -

          Thanks for the comment. I updated the patch.

          Show
          iwasakims Masatake Iwasaki added a comment - Thanks for the comment. I updated the patch.
          Hide
          aw Allen Wittenauer added a comment -

          OK, upon further playing, this doesn't cover one particularly interesting case. If log4j.settings is missing, one can't reset the log level either. So doing something like:

          $ hdfs --loglevel DEBUG namenode
          

          (note the lack of --daemon!) doesn't work as expected.

          This also means that it won't work for interactive commands as well.

          I think this if statement needs to get pushed into hadoop_basic_init so that we catch all the cases.

          Show
          aw Allen Wittenauer added a comment - OK, upon further playing, this doesn't cover one particularly interesting case. If log4j.settings is missing, one can't reset the log level either. So doing something like: $ hdfs --loglevel DEBUG namenode (note the lack of --daemon!) doesn't work as expected. This also means that it won't work for interactive commands as well. I think this if statement needs to get pushed into hadoop_basic_init so that we catch all the cases.
          Hide
          aw Allen Wittenauer added a comment -

          Actually, it should be a separate function that gets called prior to basic_init in hadoop-configh.sh. That will allow for admins to replace it with something more complex if they desire!

          Show
          aw Allen Wittenauer added a comment - Actually, it should be a separate function that gets called prior to basic_init in hadoop-configh.sh. That will allow for admins to replace it with something more complex if they desire!
          Hide
          iwasakims Masatake Iwasaki added a comment -

          I agree and updated the patch.

          Show
          iwasakims Masatake Iwasaki added a comment - I agree and updated the patch.
          Hide
          aw Allen Wittenauer added a comment -

          Users can do a HADOOP_CONF_DIR in .hadooprc, which means that the check code will miss it. So needs to be after hadoop_exec_hadooprc.

          I really need to get the unit tests for shell code built so that others are aware of these gotchas.

          Show
          aw Allen Wittenauer added a comment - Users can do a HADOOP_CONF_DIR in .hadooprc, which means that the check code will miss it. So needs to be after hadoop_exec_hadooprc. I really need to get the unit tests for shell code built so that others are aware of these gotchas.
          Hide
          iwasakims Masatake Iwasaki added a comment -

          Hmm... the fact that hadoop_exec_hadoopenv and hadoop_exec_userfuncs use the value of HADOOP_CONF_DIR before hadoop_exec_hadooprc may be confusing. Anyway, I moved the check after hadoop_exec_hadooprc on v4 patch.

          Show
          iwasakims Masatake Iwasaki added a comment - Hmm... the fact that hadoop_exec_hadoopenv and hadoop_exec_userfuncs use the value of HADOOP_CONF_DIR before hadoop_exec_hadooprc may be confusing. Anyway, I moved the check after hadoop_exec_hadooprc on v4 patch.
          Hide
          aw Allen Wittenauer added a comment -

          Hmm... the fact that hadoop_exec_hadoopenv and hadoop_exec_userfuncs use the value of HADOOP_CONF_DIR before hadoop_exec_hadooprc may be confusing.

          It calls that very, very minimally. Those functions use them to build up the shell environment using any site settings. Now a user can add/modify to the existing site level shell env. After this call, HADOOP_CONF_DIR is added to the classpath for Java.

          Looks like the wrong file was uploaded for -04....

          Show
          aw Allen Wittenauer added a comment - Hmm... the fact that hadoop_exec_hadoopenv and hadoop_exec_userfuncs use the value of HADOOP_CONF_DIR before hadoop_exec_hadooprc may be confusing. It calls that very, very minimally. Those functions use them to build up the shell environment using any site settings. Now a user can add/modify to the existing site level shell env. After this call, HADOOP_CONF_DIR is added to the classpath for Java. Looks like the wrong file was uploaded for -04....
          Hide
          iwasakims Masatake Iwasaki added a comment -

          Thanks for clarifying.

          Looks like the wrong file was uploaded for -04....

          Sorry, I am attaching the file again.

          Show
          iwasakims Masatake Iwasaki added a comment - Thanks for clarifying. Looks like the wrong file was uploaded for -04.... Sorry, I am attaching the file again.
          Hide
          aw Allen Wittenauer added a comment -

          Deleted the broken -04.

          +1 will commit to trunk.

          Thanks!

          Show
          aw Allen Wittenauer added a comment - Deleted the broken -04. +1 will commit to trunk. Thanks!
          Hide
          aw Allen Wittenauer added a comment -

          Committed.

          Show
          aw Allen Wittenauer added a comment - Committed.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #6815 (See https://builds.apache.org/job/Hadoop-trunk-Commit/6815/)
          HADOOP-11058. Missing HADOOP_CONF_DIR generates strange results (Masatake Iwasaki via aw) (aw: rev cd7d78914cd2929e42b5a893769acefdd2ed6451)

          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh
          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #6815 (See https://builds.apache.org/job/Hadoop-trunk-Commit/6815/ ) HADOOP-11058 . Missing HADOOP_CONF_DIR generates strange results (Masatake Iwasaki via aw) (aw: rev cd7d78914cd2929e42b5a893769acefdd2ed6451) hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #66 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/66/)
          HADOOP-11058. Missing HADOOP_CONF_DIR generates strange results (Masatake Iwasaki via aw) (aw: rev cd7d78914cd2929e42b5a893769acefdd2ed6451)

          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #66 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/66/ ) HADOOP-11058 . Missing HADOOP_CONF_DIR generates strange results (Masatake Iwasaki via aw) (aw: rev cd7d78914cd2929e42b5a893769acefdd2ed6451) hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #800 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/800/)
          HADOOP-11058. Missing HADOOP_CONF_DIR generates strange results (Masatake Iwasaki via aw) (aw: rev cd7d78914cd2929e42b5a893769acefdd2ed6451)

          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #800 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/800/ ) HADOOP-11058 . Missing HADOOP_CONF_DIR generates strange results (Masatake Iwasaki via aw) (aw: rev cd7d78914cd2929e42b5a893769acefdd2ed6451) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #1998 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1998/)
          HADOOP-11058. Missing HADOOP_CONF_DIR generates strange results (Masatake Iwasaki via aw) (aw: rev cd7d78914cd2929e42b5a893769acefdd2ed6451)

          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh
          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #1998 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1998/ ) HADOOP-11058 . Missing HADOOP_CONF_DIR generates strange results (Masatake Iwasaki via aw) (aw: rev cd7d78914cd2929e42b5a893769acefdd2ed6451) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Hdfs-trunk-Java8 #63 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/63/)
          HADOOP-11058. Missing HADOOP_CONF_DIR generates strange results (Masatake Iwasaki via aw) (aw: rev cd7d78914cd2929e42b5a893769acefdd2ed6451)

          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk-Java8 #63 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/63/ ) HADOOP-11058 . Missing HADOOP_CONF_DIR generates strange results (Masatake Iwasaki via aw) (aw: rev cd7d78914cd2929e42b5a893769acefdd2ed6451) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #67 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/67/)
          HADOOP-11058. Missing HADOOP_CONF_DIR generates strange results (Masatake Iwasaki via aw) (aw: rev cd7d78914cd2929e42b5a893769acefdd2ed6451)

          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #67 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/67/ ) HADOOP-11058 . Missing HADOOP_CONF_DIR generates strange results (Masatake Iwasaki via aw) (aw: rev cd7d78914cd2929e42b5a893769acefdd2ed6451) hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2017 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2017/)
          HADOOP-11058. Missing HADOOP_CONF_DIR generates strange results (Masatake Iwasaki via aw) (aw: rev cd7d78914cd2929e42b5a893769acefdd2ed6451)

          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh
          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2017 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2017/ ) HADOOP-11058 . Missing HADOOP_CONF_DIR generates strange results (Masatake Iwasaki via aw) (aw: rev cd7d78914cd2929e42b5a893769acefdd2ed6451) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh

            People

            • Assignee:
              iwasakims Masatake Iwasaki
              Reporter:
              aw Allen Wittenauer
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development