[ROCKETMQ-332] MappedFileQueue is not thread safe, which will cause message loss.

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 4.0.0-incubating, 4.1.0-incubating
• Component/s: rocketmq-store
• Assignee: Xinyu Zhou
• Reporter: Jas0n918
• Created: 15/Dec/17 04:39
• Updated: 27/Feb/18 07:42

Description

In RocketMQ V3.5.8, there is a readWriteLock in com.alibaba.rocketmq.store.MapedFileQueue, which guarantee thread safety. But in the new org.apache.rocketmq.store.MappedFileQueue, there is not any concurrent control mechanism.

when consumer is fetching message(no large lag), broker calls
org.apache.rocketmq.broker.processor.PullMessageProcessor#processRequest ==>
org.apache.rocketmq.store.DefaultMessageStore#getMessage ==>
org.apache.rocketmq.store.ConsumeQueue#getIndexBuffer ==>
org.apache.rocketmq.store.MappedFileQueue#findMappedFileByOffset

but findMappedFileByOffset is not thread safe, as
org.apache.rocketmq.store.MappedFileQueue#deleteExpiredFile maybe running concurrently( the size of mappedFiles maybe change) , which will results in ConsumeQueue#getIndexBuffer returns null, causing
nextBeginOffset = nextOffsetCorrection(offset, consumeQueue.rollNextFile(offset));+
which will skip the whole consumeQueue file, any messages left in this ConsumeQueue will not be consumed by client.

Issue Links

• links to: GitHub Pull Request #220
• links to: GitHub Pull Request #227