Hive
  1. Hive
  2. HIVE-3428

Fix log4j configuration errors when running hive on hadoop23

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.10.0
    • Fix Version/s: 0.11.0
    • Component/s: None
    • Labels:
      None

      Description

      There are log4j configuration errors when running hive on hadoop23, some of them may fail testcases, since the following log4j error message could printed to console, or to output file, which diffs from the expected output:

      [junit] < log4j:ERROR Could not find value for key log4j.appender.NullAppender
      [junit] < log4j:ERROR Could not instantiate appender named "NullAppender".
      [junit] < 12/09/04 11:34:42 WARN conf.HiveConf: hive-site.xml not found on CLASSPATH

      1. HIVE-3428.1.patch.txt
        5 kB
        Zhenxiao Luo
      2. HIVE-3428.2.patch.txt
        9 kB
        Zhenxiao Luo
      3. HIVE-3428.3.patch.txt
        35 kB
        Zhenxiao Luo
      4. HIVE-3428.4.patch.txt
        35 kB
        Zhenxiao Luo
      5. HIVE-3428.5.patch.txt
        36 kB
        Zhenxiao Luo
      6. HIVE-3428.6.patch.txt
        36 kB
        Zhenxiao Luo
      7. HIVE-3428_SHIM_EVENT_COUNTER.patch
        7 kB
        Gunther Hagleitner
      8. HIVE-3428.1.D8805.patch
        7 kB
        Gunther Hagleitner

        Issue Links

          Activity

          Hide
          Zhenxiao Luo added a comment -

          Problems in the current log4j configuration:

          1. When hive is running MapReduce job using hadoop(which locates in testutils/hadoop), hadoop is configured using HADOOP_CONF_DIR. While, currently, HADOOP_CONF_DIR is not set properly(not set at all except for pdk scripts).

          2. hadoop is by default doing log4j initialization by reading log4j.properties in its HADOOP_CONF_DIR. We do have a hive-log4j.properties, which is used to initialize hive log4j, but no such log4j.properties exited for hadoop.

          3. In the current hive-log4j.properties file, there is no log4j.appender.NullAppender defined, which could produce the following log4j error message:

          [junit] < log4j:ERROR Could not find value for key log4j.appender.NullAppender
          [junit] < log4j:ERROR Could not instantiate appender named "NullAppender".

          4. The current log4j Event Counter Appender is initialized with an deprecated value, which would produce the following warning message:

          [junit] < WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.

          Show
          Zhenxiao Luo added a comment - Problems in the current log4j configuration: 1. When hive is running MapReduce job using hadoop(which locates in testutils/hadoop), hadoop is configured using HADOOP_CONF_DIR. While, currently, HADOOP_CONF_DIR is not set properly(not set at all except for pdk scripts). 2. hadoop is by default doing log4j initialization by reading log4j.properties in its HADOOP_CONF_DIR. We do have a hive-log4j.properties, which is used to initialize hive log4j, but no such log4j.properties exited for hadoop. 3. In the current hive-log4j.properties file, there is no log4j.appender.NullAppender defined, which could produce the following log4j error message: [junit] < log4j:ERROR Could not find value for key log4j.appender.NullAppender [junit] < log4j:ERROR Could not instantiate appender named "NullAppender". 4. The current log4j Event Counter Appender is initialized with an deprecated value, which would produce the following warning message: [junit] < WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.
          Hide
          Zhenxiao Luo added a comment -

          Review Request submitted at:
          https://reviews.facebook.net/D5133

          Show
          Zhenxiao Luo added a comment - Review Request submitted at: https://reviews.facebook.net/D5133
          Hide
          Carl Steinbach added a comment -

          +1. Will commit if tests pass.

          Show
          Carl Steinbach added a comment - +1. Will commit if tests pass.
          Hide
          Carl Steinbach added a comment -

          This patch seems to break lots of stuff on 0.21.

          @Zhenxiao: Did you try running any tests on trunk with the standard configuration (i.e. using Hadoop 0.21)?

          Show
          Carl Steinbach added a comment - This patch seems to break lots of stuff on 0.21. @Zhenxiao: Did you try running any tests on trunk with the standard configuration (i.e. using Hadoop 0.21)?
          Hide
          Zhenxiao Luo added a comment -

          Updated patch submitted for review at:
          https://reviews.facebook.net/D5133

          Show
          Zhenxiao Luo added a comment - Updated patch submitted for review at: https://reviews.facebook.net/D5133
          Hide
          Carl Steinbach added a comment -

          More comments on phabricator.

          Show
          Carl Steinbach added a comment - More comments on phabricator.
          Hide
          Zhenxiao Luo added a comment -

          Updated patch, review request submitted at:
          https://reviews.facebook.net/D5133

          Show
          Zhenxiao Luo added a comment - Updated patch, review request submitted at: https://reviews.facebook.net/D5133
          Hide
          Zhenxiao Luo added a comment -

          Updated patch. comments addressed. review request submitted at:
          https://reviews.facebook.net/D5133

          Show
          Zhenxiao Luo added a comment - Updated patch. comments addressed. review request submitted at: https://reviews.facebook.net/D5133
          Hide
          Carl Steinbach added a comment -

          @Zhenxiao: please see my comments on phabricator.

          Show
          Carl Steinbach added a comment - @Zhenxiao: please see my comments on phabricator.
          Hide
          Zhenxiao Luo added a comment -

          @Carl:
          Thanks for the comments. I updated my patch at:
          https://reviews.facebook.net/D5133

          Show
          Zhenxiao Luo added a comment - @Carl: Thanks for the comments. I updated my patch at: https://reviews.facebook.net/D5133
          Hide
          Ashutosh Chauhan added a comment -

          Patch doesn't apply cleanly. Zhenxiao, can you please refresh the patch?

          Show
          Ashutosh Chauhan added a comment - Patch doesn't apply cleanly. Zhenxiao, can you please refresh the patch?
          Hide
          Gunther Hagleitner added a comment -

          Patch doesn't seem to provide configuration for hadoop.mr.rev=20S (hadoop 1-line). More importantly, if the only difference between the files is the Hadoop's deprecated EventCounter, it seems better to create a shim class for that. This way there's only one conf file and hive will pick the right file from the classpath at runtime.

          Show
          Gunther Hagleitner added a comment - Patch doesn't seem to provide configuration for hadoop.mr.rev=20S (hadoop 1-line). More importantly, if the only difference between the files is the Hadoop's deprecated EventCounter, it seems better to create a shim class for that. This way there's only one conf file and hive will pick the right file from the classpath at runtime.
          Hide
          Gunther Hagleitner added a comment -

          Proposed shim for EventCounter class. Does not contain NullAppender or HADOOP_CONF variable.

          Show
          Gunther Hagleitner added a comment - Proposed shim for EventCounter class. Does not contain NullAppender or HADOOP_CONF variable.
          Hide
          Zhenxiao Luo added a comment -

          Hi Gunther, would you please add a review request at:
          https://reviews.facebook.net/
          It is easy to review there.

          Show
          Zhenxiao Luo added a comment - Hi Gunther, would you please add a review request at: https://reviews.facebook.net/ It is easy to review there.
          Hide
          Gunther Hagleitner added a comment -
          Show
          Gunther Hagleitner added a comment - Zhenxiao Luo : Sorry for the delay: https://reviews.facebook.net/D8805
          Hide
          Ashutosh Chauhan added a comment -

          +1 Running tests.

          Show
          Ashutosh Chauhan added a comment - +1 Running tests.
          Hide
          Ashutosh Chauhan added a comment -

          Committed to trunk. Thanks, Gunther!

          Show
          Ashutosh Chauhan added a comment - Committed to trunk. Thanks, Gunther!
          Hide
          Hudson added a comment -

          Integrated in Hive-trunk-h0.21 #1989 (See https://builds.apache.org/job/Hive-trunk-h0.21/1989/)
          HIVE-3428 : Fix log4j configuration errors when running hive on hadoop23 (Gunther Hagleitner via Ashutosh Chauhan) (Revision 1450645)

          Result = SUCCESS
          hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1450645
          Files :

          • /hive/trunk/common/src/java/conf/hive-log4j.properties
          • /hive/trunk/data/conf/hive-log4j.properties
          • /hive/trunk/pdk/scripts/conf/log4j.properties
          • /hive/trunk/ql/src/java/conf/hive-exec-log4j.properties
          • /hive/trunk/shims/ivy.xml
          • /hive/trunk/shims/src/common/java/org/apache/hadoop/hive/shims/HiveEventCounter.java
          • /hive/trunk/shims/src/common/java/org/apache/hadoop/hive/shims/ShimLoader.java
          Show
          Hudson added a comment - Integrated in Hive-trunk-h0.21 #1989 (See https://builds.apache.org/job/Hive-trunk-h0.21/1989/ ) HIVE-3428 : Fix log4j configuration errors when running hive on hadoop23 (Gunther Hagleitner via Ashutosh Chauhan) (Revision 1450645) Result = SUCCESS hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1450645 Files : /hive/trunk/common/src/java/conf/hive-log4j.properties /hive/trunk/data/conf/hive-log4j.properties /hive/trunk/pdk/scripts/conf/log4j.properties /hive/trunk/ql/src/java/conf/hive-exec-log4j.properties /hive/trunk/shims/ivy.xml /hive/trunk/shims/src/common/java/org/apache/hadoop/hive/shims/HiveEventCounter.java /hive/trunk/shims/src/common/java/org/apache/hadoop/hive/shims/ShimLoader.java
          Hide
          Hudson added a comment -

          Integrated in Hive-trunk-hadoop2 #138 (See https://builds.apache.org/job/Hive-trunk-hadoop2/138/)
          HIVE-3428 : Fix log4j configuration errors when running hive on hadoop23 (Gunther Hagleitner via Ashutosh Chauhan) (Revision 1450645)

          Result = FAILURE
          hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1450645
          Files :

          • /hive/trunk/common/src/java/conf/hive-log4j.properties
          • /hive/trunk/data/conf/hive-log4j.properties
          • /hive/trunk/pdk/scripts/conf/log4j.properties
          • /hive/trunk/ql/src/java/conf/hive-exec-log4j.properties
          • /hive/trunk/shims/ivy.xml
          • /hive/trunk/shims/src/common/java/org/apache/hadoop/hive/shims/HiveEventCounter.java
          • /hive/trunk/shims/src/common/java/org/apache/hadoop/hive/shims/ShimLoader.java
          Show
          Hudson added a comment - Integrated in Hive-trunk-hadoop2 #138 (See https://builds.apache.org/job/Hive-trunk-hadoop2/138/ ) HIVE-3428 : Fix log4j configuration errors when running hive on hadoop23 (Gunther Hagleitner via Ashutosh Chauhan) (Revision 1450645) Result = FAILURE hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1450645 Files : /hive/trunk/common/src/java/conf/hive-log4j.properties /hive/trunk/data/conf/hive-log4j.properties /hive/trunk/pdk/scripts/conf/log4j.properties /hive/trunk/ql/src/java/conf/hive-exec-log4j.properties /hive/trunk/shims/ivy.xml /hive/trunk/shims/src/common/java/org/apache/hadoop/hive/shims/HiveEventCounter.java /hive/trunk/shims/src/common/java/org/apache/hadoop/hive/shims/ShimLoader.java

            People

            • Assignee:
              Gunther Hagleitner
              Reporter:
              Zhenxiao Luo
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development