Uploaded image for project: 'Qpid'
  1. Qpid
  2. QPID-6331

[Java Broker] Allow persistent AMQP 1.0 message content to be evicted from the heap

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.30
    • Fix Version/s: 0.31
    • Component/s: Broker-J
    • Labels:
      None
    • Environment:

      Red Hat Linux 6.5
      java version "1.7.0_71"
      OpenJDK Runtime Environment (rhel-2.5.3.1.el6-x86_64 u71-b14)
      OpenJDK 64-Bit Server VM (build 24.65-b04, mixed mode)

      Description

      While populating a queue with tens of thousands of messages we ran into an OutOfMemoryError. The test case was to load roughly one million, 5KB, messages on to a queue (using roughly 5GB of storage space). What we didn't expect was the broker to run out of memory since the messages should be persisted to the disk. We intentionally closed the session and connection every 5000 messages with the intent of not having many message in memory at a time. We are not sure what thread caused the error so here are all the traces with an OutOfMemoryError:

      2015-01-21 14:33:59,321 ERROR [IoReceiver - /10.96.250.6:41635] (thread.LoggingUncaughtExceptionHandler) - Uncaught exception in thread "IoReceiver - /10.96.250.6:41635"
      java.lang.OutOfMemoryError: Java heap space
      at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
      at java.nio.ByteBuffer.allocate(ByteBuffer.java:331)
      at org.apache.qpid.server.protocol.v1_0.ProtocolEngine_1_0_0_SASL.<init>(ProtocolEngine_1_0_0_SASL.java:111)
      at org.apache.qpid.server.protocol.v1_0.ProtocolEngineCreator_1_0_0_SASL.newProtocolEngine(ProtocolEngineCreator_1_0_0_SASL.java:67)
      at org.apache.qpid.server.protocol.MultiVersionProtocolEngine$SelfDelegateProtocolEngine.received(MultiVersionProtocolEngine.java:329)
      at org.apache.qpid.server.protocol.MultiVersionProtocolEngine$SelfDelegateProtocolEngine.received(MultiVersionProtocolEngine.java:259)
      at org.apache.qpid.server.protocol.MultiVersionProtocolEngine.received(MultiVersionProtocolEngine.java:133)
      at org.apache.qpid.server.protocol.MultiVersionProtocolEngine.received(MultiVersionProtocolEngine.java:49)
      at org.apache.qpid.transport.network.io.IoReceiver.run(IoReceiver.java:161)
      at java.lang.Thread.run(Thread.java:745)
      2015-01-21 14:34:00,176 ERROR [IoReceiver - /10.96.250.6:41635] (server.Main) - Uncaught exception, continuing.
      java.lang.OutOfMemoryError: Java heap space
      at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
      at java.nio.ByteBuffer.allocate(ByteBuffer.java:331)
      at org.apache.qpid.server.protocol.v1_0.ProtocolEngine_1_0_0_SASL.<init>(ProtocolEngine_1_0_0_SASL.java:111)
      at org.apache.qpid.server.protocol.v1_0.ProtocolEngineCreator_1_0_0_SASL.newProtocolEngine(ProtocolEngineCreator_1_0_0_SASL.java:67)
      at org.apache.qpid.server.protocol.MultiVersionProtocolEngine$SelfDelegateProtocolEngine.received(MultiVersionProtocolEngine.java:329)
      at org.apache.qpid.server.protocol.MultiVersionProtocolEngine$SelfDelegateProtocolEngine.received(MultiVersionProtocolEngine.java:259)
      at org.apache.qpid.server.protocol.MultiVersionProtocolEngine.received(MultiVersionProtocolEngine.java:133)
      at org.apache.qpid.server.protocol.MultiVersionProtocolEngine.received(MultiVersionProtocolEngine.java:49)
      at org.apache.qpid.transport.network.io.IoReceiver.run(IoReceiver.java:161)
      at java.lang.Thread.run(Thread.java:745)
      2015-01-21 14:34:08,799 INFO [IoReceiver - /10.96.250.6:41664] (subscription.create) - [con:168(?@/10.96.250.6:41664/default)/ch:0] [sub:82(vh(/default)/qu(emxmc-canary)] SUB-1001 : Create : Arguments : JMSSelectorFilter[selector=type='emxmc-canary-test']
      2015-01-21 14:34:10,016 INFO [IoReceiver - /10.96.250.6:41664] (subscription.close) - [con:168(?@/10.96.250.6:41664/default)/ch:0] [sub:82(vh(/default)/qu(emxmc-canary)] SUB-1002 : Close
      2015-01-21 14:34:57,300 WARN [IoReceiver - /10.96.250.6:41772] (manager.SimpleLDAPAuthenticationManagerImpl) - SASL Authentication Exception
      2015-01-21 14:35:02,103 ERROR [IoReceiver - /10.96.250.6:41772] (thread.LoggingUncaughtExceptionHandler) - Uncaught exception in thread "IoReceiver - /10.96.250.6:41772"
      java.lang.OutOfMemoryError: Java heap space
      at java.util.Arrays.copyOf(Arrays.java:2367)
      at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:130)
      at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:114)
      at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:415)
      at java.lang.StringBuffer.append(StringBuffer.java:237)
      at java.io.StringWriter.write(StringWriter.java:112)
      at java.io.PrintWriter.write(PrintWriter.java:456)
      at java.io.PrintWriter.write(PrintWriter.java:473)
      at java.io.PrintWriter.print(PrintWriter.java:603)
      at java.io.PrintWriter.println(PrintWriter.java:756)
      at java.lang.Throwable$WrappedPrintWriter.println(Throwable.java:764)
      at java.lang.Throwable.printEnclosedStackTrace(Throwable.java:697)
      at java.lang.Throwable.printStackTrace(Throwable.java:667)
      at java.lang.Throwable.printStackTrace(Throwable.java:721)
      at org.apache.log4j.DefaultThrowableRenderer.render(DefaultThrowableRenderer.java:60)
      at org.apache.log4j.spi.ThrowableInformation.getThrowableStrRep(ThrowableInformation.java:87)
      at org.apache.log4j.spi.LoggingEvent.getThrowableStrRep(LoggingEvent.java:413)
      at org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:313)
      at org.apache.log4j.WriterAppender.append(WriterAppender.java:162)
      at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
      at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
      at org.apache.log4j.Category.callAppenders(Category.java:206)
      at org.apache.log4j.Category.forcedLog(Category.java:391)
      at org.apache.log4j.Category.warn(Category.java:1060)
      at org.apache.qpid.server.security.auth.manager.SimpleLDAPAuthenticationManagerImpl$SimpleLDAPPlainCallbackHandler.handle(SimpleLDAPAuthenticationManagerImpl.java:420)
      at org.apache.qpid.server.security.auth.sasl.plain.PlainSaslServer.evaluateResponse(PlainSaslServer.java:83)
      at org.apache.qpid.amqp_1_0.transport.ConnectionEndpoint.receiveSaslInit(ConnectionEndpoint.java:890)
      at org.apache.qpid.amqp_1_0.type.security.SaslInit.invoke(SaslInit.java:112)
      at org.apache.qpid.amqp_1_0.transport.ConnectionEndpoint.receive(ConnectionEndpoint.java:806)
      at org.apache.qpid.amqp_1_0.framing.SASLFrameHandler.parse(SASLFrameHandler.java:240)
      at org.apache.qpid.server.protocol.v1_0.ProtocolEngine_1_0_0_SASL$4.run(ProtocolEngine_1_0_0_SASL.java:383)
      at org.apache.qpid.server.protocol.v1_0.ProtocolEngine_1_0_0_SASL$4.run(ProtocolEngine_1_0_0_SASL.java:379)
      2015-01-21 14:35:02,807 ERROR [IoReceiver - /10.96.250.6:41772] (server.Main) - Uncaught exception, continuing.
      java.lang.OutOfMemoryError: Java heap space
      at java.util.Arrays.copyOf(Arrays.java:2367)
      at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:130)
      at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:114)
      at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:415)
      at java.lang.StringBuffer.append(StringBuffer.java:237)
      at java.io.StringWriter.write(StringWriter.java:112)
      at java.io.PrintWriter.write(PrintWriter.java:456)
      at java.io.PrintWriter.write(PrintWriter.java:473)
      at java.io.PrintWriter.print(PrintWriter.java:603)
      at java.io.PrintWriter.println(PrintWriter.java:756)
      at java.lang.Throwable$WrappedPrintWriter.println(Throwable.java:764)
      at java.lang.Throwable.printEnclosedStackTrace(Throwable.java:697)
      at java.lang.Throwable.printStackTrace(Throwable.java:667)
      at java.lang.Throwable.printStackTrace(Throwable.java:721)
      at org.apache.log4j.DefaultThrowableRenderer.render(DefaultThrowableRenderer.java:60)
      at org.apache.log4j.spi.ThrowableInformation.getThrowableStrRep(ThrowableInformation.java:87)
      at org.apache.log4j.spi.LoggingEvent.getThrowableStrRep(LoggingEvent.java:413)
      at org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:313)
      at org.apache.log4j.WriterAppender.append(WriterAppender.java:162)
      at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
      at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
      at org.apache.log4j.Category.callAppenders(Category.java:206)
      at org.apache.log4j.Category.forcedLog(Category.java:391)
      at org.apache.log4j.Category.warn(Category.java:1060)
      at org.apache.qpid.server.security.auth.manager.SimpleLDAPAuthenticationManagerImpl$SimpleLDAPPlainCallbackHandler.handle(SimpleLDAPAuthenticationManagerImpl.java:420)
      at org.apache.qpid.server.security.auth.sasl.plain.PlainSaslServer.evaluateResponse(PlainSaslServer.java:83)
      at org.apache.qpid.amqp_1_0.transport.ConnectionEndpoint.receiveSaslInit(ConnectionEndpoint.java:890)
      at org.apache.qpid.amqp_1_0.type.security.SaslInit.invoke(SaslInit.java:112)
      at org.apache.qpid.amqp_1_0.transport.ConnectionEndpoint.receive(ConnectionEndpoint.java:806)
      at org.apache.qpid.amqp_1_0.framing.SASLFrameHandler.parse(SASLFrameHandler.java:240)
      at org.apache.qpid.server.protocol.v1_0.ProtocolEngine_1_0_0_SASL$4.run(ProtocolEngine_1_0_0_SASL.java:383)
      at org.apache.qpid.server.protocol.v1_0.ProtocolEngine_1_0_0_SASL$4.run(ProtocolEngine_1_0_0_SASL.java:379)
      2015-01-21 14:35:12,367 ERROR [Thread-1] (server.Main) - Uncaught exception, continuing.
      java.lang.OutOfMemoryError: Java heap space
      at org.apache.qpid.transport.network.io.IoSender.<init>(IoSender.java:67)
      at org.apache.qpid.transport.network.io.IoNetworkConnection.<init>(IoNetworkConnection.java:61)
      at org.apache.qpid.transport.network.io.IoNetworkTransport$AcceptingThread.run(IoNetworkTransport.java:241)

        Attachments

          Activity
          $i18n.getText('security.level.explanation', $currentSelection) Viewable by All Users
          Cancel

            People

            • Assignee:
              rgodfrey Robert Godfrey
              Reporter:
              mark@soderquist.net Mark Soderquist

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment