Uploaded image for project: 'Qpid'
  1. Qpid
  2. QPID-6331

[Java Broker] Allow persistent AMQP 1.0 message content to be evicted from the heap

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.30
    • 0.31
    • Broker-J
    • None

    • Red Hat Linux 6.5
      java version "1.7.0_71"
      OpenJDK Runtime Environment (rhel-2.5.3.1.el6-x86_64 u71-b14)
      OpenJDK 64-Bit Server VM (build 24.65-b04, mixed mode)

    Description

      While populating a queue with tens of thousands of messages we ran into an OutOfMemoryError. The test case was to load roughly one million, 5KB, messages on to a queue (using roughly 5GB of storage space). What we didn't expect was the broker to run out of memory since the messages should be persisted to the disk. We intentionally closed the session and connection every 5000 messages with the intent of not having many message in memory at a time. We are not sure what thread caused the error so here are all the traces with an OutOfMemoryError:

      2015-01-21 14:33:59,321 ERROR [IoReceiver - /10.96.250.6:41635] (thread.LoggingUncaughtExceptionHandler) - Uncaught exception in thread "IoReceiver - /10.96.250.6:41635"
      java.lang.OutOfMemoryError: Java heap space
      at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
      at java.nio.ByteBuffer.allocate(ByteBuffer.java:331)
      at org.apache.qpid.server.protocol.v1_0.ProtocolEngine_1_0_0_SASL.<init>(ProtocolEngine_1_0_0_SASL.java:111)
      at org.apache.qpid.server.protocol.v1_0.ProtocolEngineCreator_1_0_0_SASL.newProtocolEngine(ProtocolEngineCreator_1_0_0_SASL.java:67)
      at org.apache.qpid.server.protocol.MultiVersionProtocolEngine$SelfDelegateProtocolEngine.received(MultiVersionProtocolEngine.java:329)
      at org.apache.qpid.server.protocol.MultiVersionProtocolEngine$SelfDelegateProtocolEngine.received(MultiVersionProtocolEngine.java:259)
      at org.apache.qpid.server.protocol.MultiVersionProtocolEngine.received(MultiVersionProtocolEngine.java:133)
      at org.apache.qpid.server.protocol.MultiVersionProtocolEngine.received(MultiVersionProtocolEngine.java:49)
      at org.apache.qpid.transport.network.io.IoReceiver.run(IoReceiver.java:161)
      at java.lang.Thread.run(Thread.java:745)
      2015-01-21 14:34:00,176 ERROR [IoReceiver - /10.96.250.6:41635] (server.Main) - Uncaught exception, continuing.
      java.lang.OutOfMemoryError: Java heap space
      at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
      at java.nio.ByteBuffer.allocate(ByteBuffer.java:331)
      at org.apache.qpid.server.protocol.v1_0.ProtocolEngine_1_0_0_SASL.<init>(ProtocolEngine_1_0_0_SASL.java:111)
      at org.apache.qpid.server.protocol.v1_0.ProtocolEngineCreator_1_0_0_SASL.newProtocolEngine(ProtocolEngineCreator_1_0_0_SASL.java:67)
      at org.apache.qpid.server.protocol.MultiVersionProtocolEngine$SelfDelegateProtocolEngine.received(MultiVersionProtocolEngine.java:329)
      at org.apache.qpid.server.protocol.MultiVersionProtocolEngine$SelfDelegateProtocolEngine.received(MultiVersionProtocolEngine.java:259)
      at org.apache.qpid.server.protocol.MultiVersionProtocolEngine.received(MultiVersionProtocolEngine.java:133)
      at org.apache.qpid.server.protocol.MultiVersionProtocolEngine.received(MultiVersionProtocolEngine.java:49)
      at org.apache.qpid.transport.network.io.IoReceiver.run(IoReceiver.java:161)
      at java.lang.Thread.run(Thread.java:745)
      2015-01-21 14:34:08,799 INFO [IoReceiver - /10.96.250.6:41664] (subscription.create) - [con:168(?@/10.96.250.6:41664/default)/ch:0] [sub:82(vh(/default)/qu(emxmc-canary)] SUB-1001 : Create : Arguments : JMSSelectorFilter[selector=type='emxmc-canary-test']
      2015-01-21 14:34:10,016 INFO [IoReceiver - /10.96.250.6:41664] (subscription.close) - [con:168(?@/10.96.250.6:41664/default)/ch:0] [sub:82(vh(/default)/qu(emxmc-canary)] SUB-1002 : Close
      2015-01-21 14:34:57,300 WARN [IoReceiver - /10.96.250.6:41772] (manager.SimpleLDAPAuthenticationManagerImpl) - SASL Authentication Exception
      2015-01-21 14:35:02,103 ERROR [IoReceiver - /10.96.250.6:41772] (thread.LoggingUncaughtExceptionHandler) - Uncaught exception in thread "IoReceiver - /10.96.250.6:41772"
      java.lang.OutOfMemoryError: Java heap space
      at java.util.Arrays.copyOf(Arrays.java:2367)
      at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:130)
      at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:114)
      at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:415)
      at java.lang.StringBuffer.append(StringBuffer.java:237)
      at java.io.StringWriter.write(StringWriter.java:112)
      at java.io.PrintWriter.write(PrintWriter.java:456)
      at java.io.PrintWriter.write(PrintWriter.java:473)
      at java.io.PrintWriter.print(PrintWriter.java:603)
      at java.io.PrintWriter.println(PrintWriter.java:756)
      at java.lang.Throwable$WrappedPrintWriter.println(Throwable.java:764)
      at java.lang.Throwable.printEnclosedStackTrace(Throwable.java:697)
      at java.lang.Throwable.printStackTrace(Throwable.java:667)
      at java.lang.Throwable.printStackTrace(Throwable.java:721)
      at org.apache.log4j.DefaultThrowableRenderer.render(DefaultThrowableRenderer.java:60)
      at org.apache.log4j.spi.ThrowableInformation.getThrowableStrRep(ThrowableInformation.java:87)
      at org.apache.log4j.spi.LoggingEvent.getThrowableStrRep(LoggingEvent.java:413)
      at org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:313)
      at org.apache.log4j.WriterAppender.append(WriterAppender.java:162)
      at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
      at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
      at org.apache.log4j.Category.callAppenders(Category.java:206)
      at org.apache.log4j.Category.forcedLog(Category.java:391)
      at org.apache.log4j.Category.warn(Category.java:1060)
      at org.apache.qpid.server.security.auth.manager.SimpleLDAPAuthenticationManagerImpl$SimpleLDAPPlainCallbackHandler.handle(SimpleLDAPAuthenticationManagerImpl.java:420)
      at org.apache.qpid.server.security.auth.sasl.plain.PlainSaslServer.evaluateResponse(PlainSaslServer.java:83)
      at org.apache.qpid.amqp_1_0.transport.ConnectionEndpoint.receiveSaslInit(ConnectionEndpoint.java:890)
      at org.apache.qpid.amqp_1_0.type.security.SaslInit.invoke(SaslInit.java:112)
      at org.apache.qpid.amqp_1_0.transport.ConnectionEndpoint.receive(ConnectionEndpoint.java:806)
      at org.apache.qpid.amqp_1_0.framing.SASLFrameHandler.parse(SASLFrameHandler.java:240)
      at org.apache.qpid.server.protocol.v1_0.ProtocolEngine_1_0_0_SASL$4.run(ProtocolEngine_1_0_0_SASL.java:383)
      at org.apache.qpid.server.protocol.v1_0.ProtocolEngine_1_0_0_SASL$4.run(ProtocolEngine_1_0_0_SASL.java:379)
      2015-01-21 14:35:02,807 ERROR [IoReceiver - /10.96.250.6:41772] (server.Main) - Uncaught exception, continuing.
      java.lang.OutOfMemoryError: Java heap space
      at java.util.Arrays.copyOf(Arrays.java:2367)
      at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:130)
      at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:114)
      at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:415)
      at java.lang.StringBuffer.append(StringBuffer.java:237)
      at java.io.StringWriter.write(StringWriter.java:112)
      at java.io.PrintWriter.write(PrintWriter.java:456)
      at java.io.PrintWriter.write(PrintWriter.java:473)
      at java.io.PrintWriter.print(PrintWriter.java:603)
      at java.io.PrintWriter.println(PrintWriter.java:756)
      at java.lang.Throwable$WrappedPrintWriter.println(Throwable.java:764)
      at java.lang.Throwable.printEnclosedStackTrace(Throwable.java:697)
      at java.lang.Throwable.printStackTrace(Throwable.java:667)
      at java.lang.Throwable.printStackTrace(Throwable.java:721)
      at org.apache.log4j.DefaultThrowableRenderer.render(DefaultThrowableRenderer.java:60)
      at org.apache.log4j.spi.ThrowableInformation.getThrowableStrRep(ThrowableInformation.java:87)
      at org.apache.log4j.spi.LoggingEvent.getThrowableStrRep(LoggingEvent.java:413)
      at org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:313)
      at org.apache.log4j.WriterAppender.append(WriterAppender.java:162)
      at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
      at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
      at org.apache.log4j.Category.callAppenders(Category.java:206)
      at org.apache.log4j.Category.forcedLog(Category.java:391)
      at org.apache.log4j.Category.warn(Category.java:1060)
      at org.apache.qpid.server.security.auth.manager.SimpleLDAPAuthenticationManagerImpl$SimpleLDAPPlainCallbackHandler.handle(SimpleLDAPAuthenticationManagerImpl.java:420)
      at org.apache.qpid.server.security.auth.sasl.plain.PlainSaslServer.evaluateResponse(PlainSaslServer.java:83)
      at org.apache.qpid.amqp_1_0.transport.ConnectionEndpoint.receiveSaslInit(ConnectionEndpoint.java:890)
      at org.apache.qpid.amqp_1_0.type.security.SaslInit.invoke(SaslInit.java:112)
      at org.apache.qpid.amqp_1_0.transport.ConnectionEndpoint.receive(ConnectionEndpoint.java:806)
      at org.apache.qpid.amqp_1_0.framing.SASLFrameHandler.parse(SASLFrameHandler.java:240)
      at org.apache.qpid.server.protocol.v1_0.ProtocolEngine_1_0_0_SASL$4.run(ProtocolEngine_1_0_0_SASL.java:383)
      at org.apache.qpid.server.protocol.v1_0.ProtocolEngine_1_0_0_SASL$4.run(ProtocolEngine_1_0_0_SASL.java:379)
      2015-01-21 14:35:12,367 ERROR [Thread-1] (server.Main) - Uncaught exception, continuing.
      java.lang.OutOfMemoryError: Java heap space
      at org.apache.qpid.transport.network.io.IoSender.<init>(IoSender.java:67)
      at org.apache.qpid.transport.network.io.IoNetworkConnection.<init>(IoNetworkConnection.java:61)
      at org.apache.qpid.transport.network.io.IoNetworkTransport$AcceptingThread.run(IoNetworkTransport.java:241)

      Attachments

        Activity

          People

            rgodfrey Robert Godfrey
            mark@soderquist.net Mark Soderquist
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: