Hadoop Common
  1. Hadoop Common
  2. HADOOP-3654

Log4J logging of stack trace may deadlock JRockit in TestFileSystem

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Later
    • Affects Version/s: 0.19.0
    • Fix Version/s: None
    • Component/s: conf
    • Labels:
      None
    • Environment:

      Description

      This is being added as a bugrep so that other people can find it, and the workaround

      1. On my machine TestFileSystem will hang, even overnight -even though the build was set with a timeout.
      2. halting the build left a JVM running; it was not being killed.
      3. Under the IDE, the main thread appears hung in the native library call to get a stack trace, somewhere inside Log4J
      4. the IDE could not halt the build, and could not be shut down cleanly either

      The fix for this problem was to edit conf/log4j.properties and switch to a log4J log pattern that did not print the line of the code

      log4j.appender.console.layout.ConversionPattern=%-4r %-5p %c %x - %m%n

      Given that working out a stack trace can be an expensive call, and that it can apparently hang some JVMs, perhaps it should not be the default.

        Activity

        Hide
        steve_l added a comment -

        -this turns off line/file logging, and so stops deadlocks on BEA JRockit(R) (build R27.4.0-90-89592-1.6.0_02-20070928-1715-linux-x86_64, compiled mode). It may make test runs slightly faster too, as every log statement with line numbers requires an exception to be created and its stack extracted.

        -the old appender is there to be switched on in emergency
        -thread and extra context logging have been inserted as thread info is useful for separating different sources of messages.

        Show
        steve_l added a comment - -this turns off line/file logging, and so stops deadlocks on BEA JRockit(R) (build R27.4.0-90-89592-1.6.0_02-20070928-1715-linux-x86_64, compiled mode). It may make test runs slightly faster too, as every log statement with line numbers requires an exception to be created and its stack extracted. -the old appender is there to be switched on in emergency -thread and extra context logging have been inserted as thread info is useful for separating different sources of messages.
        Hide
        steve_l added a comment -

        tuned log4j parameters

        Show
        steve_l added a comment - tuned log4j parameters
        Hide
        steve_l added a comment -

        I should add that the file to patch is not conf/log4.properties but src/test/log4.properties

        Show
        steve_l added a comment - I should add that the file to patch is not conf/log4.properties but src/test/log4.properties
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12385398/hadoop-3654.patch
        against trunk revision 674671.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2807/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2807/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2807/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2807/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12385398/hadoop-3654.patch against trunk revision 674671. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2807/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2807/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2807/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2807/console This message is automatically generated.
        Hide
        Owen O'Malley added a comment -

        I think the line numbers are really useful and would hate to turn them off by default. How about doing the reverse of your patch where there is a configuration that doesn't have line numbers that can be selected?

        Show
        Owen O'Malley added a comment - I think the line numbers are really useful and would hate to turn them off by default. How about doing the reverse of your patch where there is a configuration that doesn't have line numbers that can be selected?
        Hide
        steve_l added a comment -

        -a switch to turn numbers off would work for me; something you could set in build.properties would be ideal. Maybe we could have >1 log4j.properties compatible file in the src/test classpath; and somehow make the choice of which to run tests on configurable. I'll check out the log4j docs next week to see how to do this.

        Show
        steve_l added a comment - -a switch to turn numbers off would work for me; something you could set in build.properties would be ideal. Maybe we could have >1 log4j.properties compatible file in the src/test classpath; and somehow make the choice of which to run tests on configurable. I'll check out the log4j docs next week to see how to do this.
        Hide
        steve_l added a comment -

        With up to date JRockit not being public, and the Oracle/Sun merger going to make for some interesting JDK futures, marking this as a LATER. If the JRockit codebase moves to being the core JDK, it will surface again.

        Show
        steve_l added a comment - With up to date JRockit not being public, and the Oracle/Sun merger going to make for some interesting JDK futures, marking this as a LATER. If the JRockit codebase moves to being the core JDK, it will surface again.

          People

          • Assignee:
            Unassigned
            Reporter:
            Steve Loughran
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development