[QPID-4319] Perf test VaryingNumberOfParticipants produced broker exception Metadata not found for message with id nnnnn [error code 541: internal error]

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.18
• Fix Version/s: 0.30
• Component/s: Broker-J
• Assignee: Keith Wall
• Reporter: Keith Wall
• Created: 18/Sep/12 14:34
• Updated: 14/Mar/17 20:13
• Resolved: 14/Feb/15 15:21

Description

Running the complete perf testsuite (perftest/etc/testdef), I ran into the following failure. The test had reached VaryingNumberOfParticipants - 1 consumer - 10 producers - PERSISTENT and the Broker was struggling for memory (1gb heap).

The client stack was:

2012-09-18 14:57:56,197 ERROR [IoSender - localhost/127.0.0.1:5672] [IoSender] error in write thread
java.net.SocketException: Broken pipe
        at java.net.SocketOutputStream.socketWrite0(Native Method)
        at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
        at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
        at org.apache.qpid.transport.network.io.IoSender.run(IoSender.java:313)
        at java.lang.Thread.run(Thread.java:662)

whilst the Broker stack was:

2012-09-18 14:57:56,094 ERROR [IoReceiver - /127.0.0.1:49348] (AMQProtocolEngine.java:258) - Unexpected exception when processing datablock
java.lang.RuntimeException: org.apache.qpid.AMQStoreException: Metadata not found for message with id 22902968 [error code 541: internal error]
        at org.apache.qpid.server.store.berkeleydb.AbstractBDBMessageStore$StoredBDBMessage.getMetaData(AbstractBDBMessageStore.java:1615)
        at org.apache.qpid.server.message.AMQMessage.getMessageMetaData(AMQMessage.java:93)
        at org.apache.qpid.server.message.AMQMessage.getArrivalTime(AMQMessage.java:160)
        at org.apache.qpid.server.queue.SimpleAMQQueue.getOldestMessageArrivalTime(SimpleAMQQueue.java:1057)
        at org.apache.qpid.server.queue.NotificationCheck$4.notifyIfNecessary(NotificationCheck.java:104)
        at org.apache.qpid.server.queue.SimpleAMQQueue.checkForNotification(SimpleAMQQueue.java:2327)
        at org.apache.qpid.server.queue.SimpleAMQQueue.enqueue(SimpleAMQQueue.java:766)
        at org.apache.qpid.server.AMQChannel$MessageDeliveryAction.postCommit(AMQChannel.java:1207)
        at org.apache.qpid.server.AMQChannel$AsyncCommand.complete(AMQChannel.java:1674)
        at org.apache.qpid.server.AMQChannel.sync(AMQChannel.java:1644)
        at org.apache.qpid.server.AMQChannel.receivedComplete(AMQChannel.java:221)
        at org.apache.qpid.server.protocol.AMQProtocolEngine.receiveComplete(AMQProtocolEngine.java:271)
        at org.apache.qpid.server.protocol.AMQProtocolEngine.received(AMQProtocolEngine.java:254)
        at org.apache.qpid.server.protocol.AMQProtocolEngine.received(AMQProtocolEngine.java:85)
        at org.apache.qpid.server.protocol.MultiVersionProtocolEngine.received(MultiVersionProtocolEngine.java:118)
        at org.apache.qpid.server.protocol.MultiVersionProtocolEngine.received(MultiVersionProtocolEngine.java:37)
        at org.apache.qpid.transport.network.io.IoReceiver.run(IoReceiver.java:152)
        at java.lang.Thread.run(Thread.java:662)
Caused by: org.apache.qpid.AMQStoreException: Metadata not found for message with id 22902968 [error code 541: internal error]
        at org.apache.qpid.server.store.berkeleydb.AbstractBDBMessageStore.getMessageMetaData(AbstractBDBMessageStore.java:1417)
        at org.apache.qpid.server.store.berkeleydb.AbstractBDBMessageStore$StoredBDBMessage.getMetaData(AbstractBDBMessageStore.java:1611)

Issue Links

• is related to: QPID-4523 Housekeeper ended with Metadata not found for message with id 50746740 [error code 541: internal error] during perf test run
• relates to: QPID-5970 [Java Broker] Expose "age of oldest message on queue" to management apis