Uploaded image for project: 'Apache RocketMQ'
  1. Apache RocketMQ
  2. ROCKETMQ-332

MappedFileQueue is not thread safe, which will cause message loss.

Add voteWatch issue
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 4.0.0-incubating, 4.1.0-incubating
    • None
    • rocketmq-store
    • None

    Description

      In RocketMQ V3.5.8, there is a readWriteLock in com.alibaba.rocketmq.store.MapedFileQueue, which guarantee thread safety. But in the new org.apache.rocketmq.store.MappedFileQueue, there is not any concurrent control mechanism.

      when consumer is fetching message(no large lag), broker calls
      org.apache.rocketmq.broker.processor.PullMessageProcessor#processRequest ==>
      org.apache.rocketmq.store.DefaultMessageStore#getMessage ==>
      org.apache.rocketmq.store.ConsumeQueue#getIndexBuffer ==>
      org.apache.rocketmq.store.MappedFileQueue#findMappedFileByOffset

      but findMappedFileByOffset is not thread safe, as
      org.apache.rocketmq.store.MappedFileQueue#deleteExpiredFile maybe running concurrently( the size of mappedFiles maybe change) , which will results in ConsumeQueue#getIndexBuffer returns null, causing
      nextBeginOffset = nextOffsetCorrection(offset, consumeQueue.rollNextFile(offset));+
      which will skip the whole consumeQueue file, any messages left in this ConsumeQueue will not be consumed by client.

      Attachments

        1. rocketmq.log
          3 kB
          Jas0n918

        Activity

          People

            yukon Xinyu Zhou
            Jas0n918 Jas0n918

            Dates

              Created:
              Updated:

              Slack

                Issue deployment