STOMP client cannot be stopped and disconnected without loosing messages

• Key: AMQ-1192
• Type: Bug
• Status: Resolved
• Resolution: Fixed
• Priority: Major
• Assignee: Unassigned
• Reporter: Joel Schaubert
• Affects Version/s: 4.1.0
• Fix Version/s: 4.1.1

Environment

C stomp client on AS3 linux
apache-activemq-4.1.0-incubator
apache APR 0.9.7

Description

I have added to the STOMP tester so it can take command line args for number of message to be read/sent. Using this the following tests will show the problems

#1) All messages are lost when a client does STOMP disconnect

1.1 start a subscriber to read 5 messages
./activemq_tester -d sub -m 5
1.2 start a publisher to publish 10 messages
./activemq_tester -d pub -m 10
1.3 run subscriber again to get the remaining 5 messages
./activemq_tester -d sub -m 5

---> No messages are available for the second run of the test subscriber. It seems there is no way to shutdown a stomp client without loosing all the messages on that q.

Output here.

-bash-2.05b$ ./activemq_tester -d pub -m 10
Connecting......OK
Sending connect message.OK
Reading Response.Response: CONNECTED,
OK
Sending Message.Sending Message: This is message number 0
Sending Message: This is message number 1
Sending Message: This is message number 2
Sending Message: This is message number 3
Sending Message: This is message number 4
Sending Message: This is message number 5
Sending Message: This is message number 6
Sending Message: This is message number 7
Sending Message: This is message number 8
Sending Message: This is message number 9
OK
Sending Disconnect.OK
Disconnecting...OK

— first run of subscriber ----------------------
-bash-2.05b$ ./activemq_tester -d sub -m 5
Connecting......OK
Sending connect message.OK
Reading Response.Response: CONNECTED,
OK
Sending Subscribe.OK
Reading Subscribed Messsages.Received: MESSAGE, This is message number 0
Received: MESSAGE, This is message number 1
Received: MESSAGE, This is message number 2
Received: MESSAGE, This is message number 3
Received: MESSAGE, This is message number 4
OK
Sending Disconnect.OK
Disconnecting...OK

----- second run of subscriber -----------------
-bash-2.05b$ ./activemq_tester -d sub -m 5
Connecting......OK
Sending connect message.OK
Reading Response.Response: CONNECTED,
OK
Sending Subscribe.OK
Reading Subscribed Messsages.

=======> None of the remaning message numbers 5...9 are available. They are lost.

2) Now to see the ghost STOMP subscriber problem, hit control-C to stop the subscriber.
Then rerun subscriber and publisher and see the following output.....
(publisher output not shown here, same as before)

-bash-2.05b$ ./activemq_tester -d sub -m 5
Connecting......OK
Sending connect message.OK
Reading Response.Response: CONNECTED,
OK
Sending Subscribe.OK
Reading Subscribed Messsages.Received: MESSAGE, This is message number 1
Received: MESSAGE, This is message number 3
Received: MESSAGE, This is message number 5
Received: MESSAGE, This is message number 7
Received: MESSAGE, This is message number 9
OK
Sending Disconnect.OK
Disconnecting...OK

Comments

[James Strachan]

I wonder could you try the current 4.1.1 build please? We've fixed a number of issues in the Stomp support to do with handling disconnected stomp consumers which could well fix your issue

[Joel Schaubert]

Yes I will try that. Can you show me where to find 4.1.1 ?

[James Strachan]

We'll be releasing 4.1.1. in the next week; until then you could try a snapshot build of it...

http://people.apache.org/repo/m2-snapshot-repository/org/apache/activemq/apache-activemq/4.1-SNAPSHOT/

e.g. grab one from March 2007

[Joel Schaubert]

Got it.

issue #2) Ghost subscriber problem is improved. Note on the output from before that I got only odd numbered messages after control-C of a subscriber. Now only message number 0 is missing.
See output here....this is after starting a subscriber and control-c, then starting another subscriber..

-bash-2.05b$ ./activemq_tester -d sub -m 5
Connecting......OK
Sending connect message.OK
Reading Response.Response: CONNECTED,
OK
Sending Subscribe.OK
Reading Subscribed Messsages.Received: MESSAGE, This is message number 1
Received: MESSAGE, This is message number 2
Received: MESSAGE, This is message number 3
Received: MESSAGE, This is message number 4
Received: MESSAGE, This is message number 5
OK
Sending Disconnect.OK
Disconnecting...OK

Issue #1) Missing messages after STOMP client shutdown ... No change here. See above for test setup and description.

Now I'll speculate based on 3 test cases I've been running.
Say I publish 100 messages to a q.
Then I start a slow consumer (subscriber to the q) via stomp, so it reads little by little.
Now I start a second consumer (subscriber, same q)
That second client will get ZERO messages.

So what I think might be happening is that STOMP connector is not really integrated into the broker, it wakes up when that first client comes in and grabs all 100 messages. Then when the second client comes in there are none left for it to get.

Then if that first client dies or even stops gracefully, somehow those 100 messages are allocated to the stomp connecter that was talking on that specific tcp/ip socket. all 100 of those messages seem to be "consumed" from the brokers point of view and cannot be made available to any other STOMP clients. Even thought the actual STOMP client has only gotten say 7 of them across the TCP/IP connection (3 second sleep loop after each message) the stomp connecter representing it has grabbed them all from the broker and is hoarding them and won't share, and won't put them back if that socket connection is closed.

Keep in mind I know nothing about the internals of JMS brokers, this is speculation based on a combination of 3 test scenarios but it would explain all 3 mis-behaviours that I am seeing.

Joel Schaubert

[Joel Schaubert]

Retested with ack set to client during subscribe operations after talking to Hiram.

#1) Ghost client issues are totally resolved in client ack mode
#2) Missing messages resolved in client ack mode

#3) New problem in client ack mode
A) send 20 messages into a Q
B) startup slow reader, read 5 messages and control-C without allowing client to send DISCONNECT
C) restart a reader (as many as desired)
D) the remaining 15 messages are still present but none will be delivered to the STOMP clients until at least 1 new message is written into the Q.

So no loss of messages under these conditions but Q is effectively stalled until a new message is written in.

[James Strachan]

Here's some background on prefetch (and how that affects message delivery)...

http://activemq.apache.org/i-do-not-receive-messages-in-my-second-consumer.html

[Hiram Chirino]

I think we should close this issue out as "not a bug" and open a new issue for your #3.

but let me comment on #3 while I'm here.. I think this is due to the socket being killed but the server side not noticing. But it will notice once it tries to send it a message, so the it shutdown the dead socket and redelivers all messages queued for the client. On some OSes the TCP timeout interval can be tweaked. Otherwise, keep alive packets would need to be periodically transmitted in Stomp so that the server could detect a dead client sooner. Keep alives have not been added to Stomp yet.. but could be in a future version.

[Joel Schaubert]

Agreed.
Nice work on resolving the ghost client issue.

[Joel Schaubert]

Part #1 is closed since this was not a bug.
Subscribing with ack set to client ack resolved the issue

Part #2 ghost subscriber problem is confirmed fixed in 4.1.1 builds while using client ack subscriptions

Part #3 moved to new ticket