Hadoop Common
  1. Hadoop Common
  2. HADOOP-7055

Update of commons logging libraries causes EventCounter to count logging events incorrectly

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.21.0, 0.21.1, 0.22.0, 0.23.0
    • Fix Version/s: 0.23.0
    • Component/s: metrics
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Hadoop 0.20.2 uses commons logging 1.0.4. EventCounter works correctly with this version of commons logging. Hadoop 0.21.0 uses commons logging 1.1.1 which causes EventCounter to count logging events incorrectly. I have verified it with Hadoop 0.21.0. After start-up of hadoop, I checked jvmmetrics.log after several minutes. In every metrics record, "logError=0, logFatal=0, logInfo=3, logWarn=0" was shown. The following text is an example.

      jvm.metrics: hostName=jingguolin, processName=DataNode, sessionId=, gcCount=3, gcTimeMillis=31, logError=0, logFatal=0, logInfo=3, logWarn=0, maxMemoryM=888.9375, memHeapCommittedM=38.0625, memHeapUsedM=3.6539612, memNonHeapCommittedM=18.25, memNonHeapUsedM=11.335686, threadsBlocked=0, threadsNew=0, threadsRunnable=8, threadsTerminated=0, threadsTimedWaiting=6, threadsWaiting=6

      Then I stopped hadoop and replaced commons logging 1.1.1 with 1.0.4. After the re-start of hadoop, a lot of logging events showed up in jvmmetrics.log.

      I have checked the source code of Log4JLogger for both 1.0.4 (http://svn.apache.org/viewvc/commons/proper/logging/tags/LOGGING_1_0_4/src/java/org/apache/commons/logging/impl/Log4JLogger.java?view=markup) and 1.1.1 (http://svn.apache.org/viewvc/commons/proper/logging/tags/commons-logging-1.1.1/src/java/org/apache/commons/logging/impl/Log4JLogger.java?view=markup). For 1.0.4, Level instances such as Level.INFO are used to construct LoggingEvent. But for 1.1.1, Priority instances such as Priority.INFO are used to construct LoggingEvent. So 1.1.1 version's event.getLevel() always returns Priority instances. EventCounter append method's "==" check always fails between a Level instance and a Priority instance. For "logInfo=3" metrics records produced by commons logging 1.1.1., I think that these 3 logging events are produced by log4j code directly instead of through commons logging API. The following code is EventCounter's append method.

      public void append(LoggingEvent event) {
      Level level = event.getLevel();
      if (level == Level.INFO)

      { counts.incr(INFO); }

      else if (level == Level.WARN)

      { counts.incr(WARN); }

      else if (level == Level.ERROR)

      { counts.incr(ERROR); }

      else if (level == Level.FATAL)

      { counts.incr(FATAL); }

      }

        Issue Links

          Activity

            People

            • Assignee:
              Jingguo Yao
              Reporter:
              Jingguo Yao
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - 0.5h
                0.5h
                Remaining:
                Remaining Estimate - 0.5h
                0.5h
                Logged:
                Time Spent - Not Specified
                Not Specified

                  Development