Uploaded image for project: 'Apache RocketMQ'
  1. Apache RocketMQ
  2. ROCKETMQ-145

Hit ConcurrentModificationException in doWaitTransfer which happens very offen

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 4.0.0-incubating
    • Fix Version/s: 4.1.0-incubating
    • Component/s: rocketmq-store
    • Labels:
      None

      Description

      we use master and slave , sync transfer and asyn flush, happens this very offen:
      2017-03-17 20:12:38 WARN GroupTransferService - GroupTransferService service has exception.
      java.util.ConcurrentModificationException: null
      at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901) ~[na:1.8.0_121]
      at java.util.ArrayList$Itr.next(ArrayList.java:851) ~[na:1.8.0_121]
      at org.apache.rocketmq.store.ha.HAService$GroupTransferService.doWaitTransfer(HAService.java:277) ~[rocketmq-store-4.0.0-incubating.jar:4.0.0-incubating]
      at org.apache.rocketmq.store.ha.HAService$GroupTransferService.run(HAService.java:301) ~[rocketmq-store-4.0.0-incubating.jar:4.0.0-incubating]
      at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]

        Activity

        Hide
        easonchen Eason Chen added a comment -

        seems fixed

        Show
        easonchen Eason Chen added a comment - seems fixed
        Hide
        lizhanhui Zhanhui Li added a comment -

        Jaskey Lam You are right that the new model wakes up group-commit thread more frequently, at most every 10 milliseconds without considering OS scheduling, which makes this bug easier to reproduce. The code base, after applying this patch 0b39fcadfa2950f1dd3975e1262ccd544f350750, should be working correctly. The only remaining issue, IMO, is that the group-commit thread wakes up unnecessarily frequently.

        Show
        lizhanhui Zhanhui Li added a comment - Jaskey Lam You are right that the new model wakes up group-commit thread more frequently, at most every 10 milliseconds without considering OS scheduling, which makes this bug easier to reproduce. The code base, after applying this patch 0b39fcadfa2950f1dd3975e1262ccd544f350750, should be working correctly. The only remaining issue, IMO, is that the group-commit thread wakes up unnecessarily frequently.
        Hide
        Jaskey Jaskey Lam added a comment - - edited

        yukon Zhanhui Li
        The code base is nearly not changed but the problem actually is easy to reproduced, I suspect what brings the problem up is that the new version changes

        tihis.wait(0)`
        

        into

        `this.wait(10)` 
        

        , is that right? which makes it possible to have race condition.
        In the old version, it will never wake up unless notified, now every 10ms, it will swap, and this make have race condition with the put message thread

        Show
        Jaskey Jaskey Lam added a comment - - edited yukon Zhanhui Li The code base is nearly not changed but the problem actually is easy to reproduced, I suspect what brings the problem up is that the new version changes tihis.wait(0)` into ` this .wait(10)` , is that right? which makes it possible to have race condition. In the old version, it will never wake up unless notified, now every 10ms, it will swap, and this make have race condition with the put message thread
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 0b39fcadfa2950f1dd3975e1262ccd544f350750 in incubator-rocketmq's branch refs/heads/develop from yukon
        [ https://git-wip-us.apache.org/repos/asf?p=incubator-rocketmq.git;h=0b39fca ]

        ROCKETMQ-145[HOTFIX] Resolve concureent issue in HAService and GroupCommitService

        Show
        jira-bot ASF subversion and git services added a comment - Commit 0b39fcadfa2950f1dd3975e1262ccd544f350750 in incubator-rocketmq's branch refs/heads/develop from yukon [ https://git-wip-us.apache.org/repos/asf?p=incubator-rocketmq.git;h=0b39fca ] ROCKETMQ-145 [HOTFIX] Resolve concureent issue in HAService and GroupCommitService
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 155823ef75314fba328200bd0a1a7aec9f2003d2 in incubator-rocketmq's branch refs/heads/master from yukon
        [ https://git-wip-us.apache.org/repos/asf?p=incubator-rocketmq.git;h=155823e ]

        ROCKETMQ-145[HOTFIX] Resolve concureent issue in HAService and GroupCommitService

        Show
        jira-bot ASF subversion and git services added a comment - Commit 155823ef75314fba328200bd0a1a7aec9f2003d2 in incubator-rocketmq's branch refs/heads/master from yukon [ https://git-wip-us.apache.org/repos/asf?p=incubator-rocketmq.git;h=155823e ] ROCKETMQ-145 [HOTFIX] Resolve concureent issue in HAService and GroupCommitService
        Hide
        Yukon yukon added a comment -

        And this issue also appears in GroupCommitService, which will be fixed together.

        Show
        Yukon yukon added a comment - And this issue also appears in GroupCommitService, which will be fixed together.
        Hide
        lizhanhui Zhanhui Li added a comment -

        We have identified cause of this issue and will fix very soon.

        Show
        lizhanhui Zhanhui Li added a comment - We have identified cause of this issue and will fix very soon.

          People

          • Assignee:
            Yukon yukon
            Reporter:
            easonchen Eason Chen
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development