Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-3654

Log4J logging of stack trace may deadlock JRockit in TestFileSystem

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Later
    • Affects Version/s: 0.19.0
    • Fix Version/s: None
    • Component/s: conf
    • Labels:
      None
    • Environment:

      Description

      This is being added as a bugrep so that other people can find it, and the workaround

      1. On my machine TestFileSystem will hang, even overnight -even though the build was set with a timeout.
      2. halting the build left a JVM running; it was not being killed.
      3. Under the IDE, the main thread appears hung in the native library call to get a stack trace, somewhere inside Log4J
      4. the IDE could not halt the build, and could not be shut down cleanly either

      The fix for this problem was to edit conf/log4j.properties and switch to a log4J log pattern that did not print the line of the code

      log4j.appender.console.layout.ConversionPattern=%-4r %-5p %c %x - %m%n

      Given that working out a stack trace can be an expensive call, and that it can apparently hang some JVMs, perhaps it should not be the default.

      1. hadoop-3654.patch
        0.8 kB
        Steve Loughran

        Activity

        Hide
        steve_l Steve Loughran added a comment -

        With up to date JRockit not being public, and the Oracle/Sun merger going to make for some interesting JDK futures, marking this as a LATER. If the JRockit codebase moves to being the core JDK, it will surface again.

        Show
        steve_l Steve Loughran added a comment - With up to date JRockit not being public, and the Oracle/Sun merger going to make for some interesting JDK futures, marking this as a LATER. If the JRockit codebase moves to being the core JDK, it will surface again.
        Hide
        steve_l Steve Loughran added a comment -

        -a switch to turn numbers off would work for me; something you could set in build.properties would be ideal. Maybe we could have >1 log4j.properties compatible file in the src/test classpath; and somehow make the choice of which to run tests on configurable. I'll check out the log4j docs next week to see how to do this.

        Show
        steve_l Steve Loughran added a comment - -a switch to turn numbers off would work for me; something you could set in build.properties would be ideal. Maybe we could have >1 log4j.properties compatible file in the src/test classpath; and somehow make the choice of which to run tests on configurable. I'll check out the log4j docs next week to see how to do this.
        Hide
        owen.omalley Owen O'Malley added a comment -

        I think the line numbers are really useful and would hate to turn them off by default. How about doing the reverse of your patch where there is a configuration that doesn't have line numbers that can be selected?

        Show
        owen.omalley Owen O'Malley added a comment - I think the line numbers are really useful and would hate to turn them off by default. How about doing the reverse of your patch where there is a configuration that doesn't have line numbers that can be selected?
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12385398/hadoop-3654.patch
        against trunk revision 674671.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2807/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2807/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2807/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2807/console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12385398/hadoop-3654.patch against trunk revision 674671. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2807/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2807/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2807/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2807/console This message is automatically generated.
        Hide
        steve_l Steve Loughran added a comment -

        I should add that the file to patch is not conf/log4.properties but src/test/log4.properties

        Show
        steve_l Steve Loughran added a comment - I should add that the file to patch is not conf/log4.properties but src/test/log4.properties
        Hide
        steve_l Steve Loughran added a comment -

        tuned log4j parameters

        Show
        steve_l Steve Loughran added a comment - tuned log4j parameters
        Hide
        steve_l Steve Loughran added a comment -

        -this turns off line/file logging, and so stops deadlocks on BEA JRockit(R) (build R27.4.0-90-89592-1.6.0_02-20070928-1715-linux-x86_64, compiled mode). It may make test runs slightly faster too, as every log statement with line numbers requires an exception to be created and its stack extracted.

        -the old appender is there to be switched on in emergency
        -thread and extra context logging have been inserted as thread info is useful for separating different sources of messages.

        Show
        steve_l Steve Loughran added a comment - -this turns off line/file logging, and so stops deadlocks on BEA JRockit(R) (build R27.4.0-90-89592-1.6.0_02-20070928-1715-linux-x86_64, compiled mode). It may make test runs slightly faster too, as every log statement with line numbers requires an exception to be created and its stack extracted. -the old appender is there to be switched on in emergency -thread and extra context logging have been inserted as thread info is useful for separating different sources of messages.

          People

          • Assignee:
            Unassigned
            Reporter:
            stevel@apache.org Steve Loughran
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development