Running follow script in hedwig-server project
We may get assertion failure:
This is a race issue which may cause messages been throttled by mistake, the root cause is ActiveSubscriberState.messageConsumed() and ActiveSubscriberState.deliverNextMessage() may be executed in different threads by AbstractSubscriptionManager and FIFODeliveryManager.
Read the log in attachement around Line 2420 if you want to get more information, here I replay the logs onto the code (Line XX denotes code listed below):
- Messages 1-30 are to be delivered and the message window size in Hub server is 10.
- Messages 1-10 are delivered to subscriber while message 11-30 is throttled by the window size limitation.
- Subscriber calls consume 1-10 asynchronously.
- CONSUME 1 is handled and FIFODeliveryManager continue to deliver message 11.
- Subscriber receive message 11 and quickly ack CONSUME 11 to Hub.
- Now there are two threads operate on a same ActiveSubscriberState:
- Thread in AbstractSubscriptionManager: call ActiveSubscriberState.messageConsumed() for message 2, 3, 11 (4-10 is still on the way since it's asynchronous consume). Let's assume this thread happen to run in Line (14) for message 11.
- Thread in FIFODeliveryManager: Coincidently, it's in Line (36) now (with last delivered 11, last consumed 1 and variable isThrottled is still false).
- If thread in AbstractSubscriptionManager executed before FIFODeliveryManager, then consume operator for 11 does nothing more.
- CONSUME [4-10] will be just ignored by the if statement in Line (2) since lastSeqIdConsumedUtil is now 11.
- Further messages like 12 have no chance to been delivered at this time.
By the way, we should also take care of thread-safe issue in other methods for ActiveSubscriberState, which implements some other callback interface.