Log4j 2
  1. Log4j 2
  2. LOG4J2-196

log4j2 blocks on logging calls if the FlumeAppender loses its connection to the remote flume agent

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.0-beta5
    • Fix Version/s: 2.0-beta5
    • Component/s: Flume Appender
    • Labels:
      None

      Description

      Steps to reproduce (use attached Maven project):

      1. Start your remote flume agent
      2. Run the Maven project
      3. Kill your remote flume agent
      4. Press enter

      Each time you press Enter, a bunch of stacktraces should show up in your console, along with a line that says log message tookXXXX (where XXXX is the number of milliseconds it took for the log message to return). Notice how the call to log something blocks for 2 seconds in the sample output below:

      sent initial error message.
      kill your flume agent now and enter any key to log another message.


      sending msg

      ERROR StatusLogger Unable to create transceiver java.io.IOException: Error connecting to /127.0.0.1:36892
      at org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:261)
      at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:203)
      at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:152)
      at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:120)
      at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:107)
      at org.apache.logging.log4j.flume.appender.FlumeAvroManager.connect(FlumeAvroManager.java:301)
      at org.apache.logging.log4j.flume.appender.FlumeAvroManager.send(FlumeAvroManager.java:241)
      at org.apache.logging.log4j.flume.appender.FlumeAppender.append(FlumeAppender.java:90)
      at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:102)
      at org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:402)
      at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:383)
      at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:386)
      at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:344)
      at org.apache.logging.log4j.core.Logger.log(Logger.java:108)
      at org.apache.logging.log4j.spi.AbstractLogger.error(AbstractLogger.java:559)
      at com.globalrelay.apps.zzzzzzzzzzzzzzzzztest.App.main(App.java:26)
      Caused by: java.net.ConnectException: Connection refused
      at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:692)
      at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:396)
      at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:358)
      at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:274)
      at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at java.lang.Thread.run(Thread.java:722)

      ERROR StatusLogger Unable to create transceiver java.io.IOException: Error connecting to /127.0.0.1:36892
      at org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:261)
      at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:203)
      at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:152)
      at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:120)
      at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:107)
      at org.apache.logging.log4j.flume.appender.FlumeAvroManager.connect(FlumeAvroManager.java:301)
      at org.apache.logging.log4j.flume.appender.FlumeAvroManager.send(FlumeAvroManager.java:241)
      at org.apache.logging.log4j.flume.appender.FlumeAppender.append(FlumeAppender.java:90)
      at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:102)
      at org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:402)
      at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:383)
      at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:386)
      at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:344)
      at org.apache.logging.log4j.core.Logger.log(Logger.java:108)
      at org.apache.logging.log4j.spi.AbstractLogger.error(AbstractLogger.java:559)
      at com.globalrelay.apps.zzzzzzzzzzzzzzzzztest.App.main(App.java:26)
      Caused by: java.net.ConnectException: Connection refused
      at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:692)
      at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:396)
      at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:358)
      at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:274)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at java.lang.Thread.run(Thread.java:722)

      ERROR StatusLogger Unable to create transceiver java.io.IOException: Error connecting to /127.0.0.1:36892
      at org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:261)
      at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:203)
      at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:152)
      at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:120)
      at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:107)
      at org.apache.logging.log4j.flume.appender.FlumeAvroManager.connect(FlumeAvroManager.java:301)
      at org.apache.logging.log4j.flume.appender.FlumeAvroManager.send(FlumeAvroManager.java:241)
      at org.apache.logging.log4j.flume.appender.FlumeAppender.append(FlumeAppender.java:90)
      at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:102)
      at org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:402)
      at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:383)
      at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:386)
      at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:344)
      at org.apache.logging.log4j.core.Logger.log(Logger.java:108)
      at org.apache.logging.log4j.spi.AbstractLogger.error(AbstractLogger.java:559)
      at com.globalrelay.apps.zzzzzzzzzzzzzzzzztest.App.main(App.java:26)
      Caused by: java.net.ConnectException: Connection refused
      at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:692)
      at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:396)
      at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:358)
      at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:274)
      at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at java.lang.Thread.run(Thread.java:722)

      ERROR StatusLogger An exception occurred processing Appender FlumeAvroAppender org.apache.logging.log4j.core.appender.AppenderRuntimeException: Unable to write to FlumeAvro[127.0.0.1:36892,127.0.0.1:36892] at 127.0.0.1:36892
      at org.apache.logging.log4j.flume.appender.FlumeAvroManager.send(FlumeAvroManager.java:267)
      at org.apache.logging.log4j.flume.appender.FlumeAppender.append(FlumeAppender.java:90)
      at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:102)
      at org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:402)
      at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:383)
      at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:386)
      at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:344)
      at org.apache.logging.log4j.core.Logger.log(Logger.java:108)
      at org.apache.logging.log4j.spi.AbstractLogger.error(AbstractLogger.java:559)
      at com.globalrelay.apps.zzzzzzzzzzzzzzzzztest.App.main(App.java:26)

      log message took2079

        Activity

        Hide
        Edward Sargisson added a comment -

        Verified - can be closed.

        Show
        Edward Sargisson added a comment - Verified - can be closed.
        Hide
        Ralph Goers added a comment - - edited

        Fixed in revision 1468306 by changing FlumeAvroManager to use Flume's RPCClient. Please verify and close.

        Show
        Ralph Goers added a comment - - edited Fixed in revision 1468306 by changing FlumeAvroManager to use Flume's RPCClient. Please verify and close.
        Hide
        offbynull added a comment -

        Attached Maven project required for the reproduction steps.

        Show
        offbynull added a comment - Attached Maven project required for the reproduction steps.

          People

          • Assignee:
            Ralph Goers
            Reporter:
            offbynull
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development