Uploaded image for project: 'Apache RocketMQ'
  1. Apache RocketMQ
  2. ROCKETMQ-175

Consumer may miss messages because of inconsistent subscription

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 4.0.0-incubating
    • Fix Version/s: 4.1.0-incubating
    • Component/s: rocketmq-broker
    • Labels:
      None

      Description

      When consumers in one group have different subscriptions(with tag), broker will select the one has latest client version. Although when this consumer shutdown, broker would not update the subscription until someone in group restart, so other consumers will miss messages by the dead consumer's tag.

      Broker should refresh the group's subscriptions when someone was off-line.

        Issue Links

          Activity

          Hide
          Jaskey Jaskey Lam added a comment -

          This is not a bug, every client should be have exactly the same subscrioptions including the same topic and the same tags which is clarified in the docs.

          But this should be improved , and the related issue is https://issues.apache.org/jira/browse/ROCKETMQ-94

          Show
          Jaskey Jaskey Lam added a comment - This is not a bug, every client should be have exactly the same subscrioptions including the same topic and the same tags which is clarified in the docs. But this should be improved , and the related issue is https://issues.apache.org/jira/browse/ROCKETMQ-94
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user vsair opened a pull request:

          https://github.com/apache/incubator-rocketmq/pull/92

          ROCKETMQ-175 Consumer may miss messages because of inconsistent sub…

          For push mode, update subscription version when rebalance result changed, then send heart beat to broker immediately.

          Maybe subscriptions management mechanism could be re-designed to meet more complicated scenarios.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/vsair/incubator-rocketmq ROCKETMQ-175

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/incubator-rocketmq/pull/92.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #92


          commit 61901647fb1851bbb36b4734195206a364a0c0bd
          Author: vsair <liuxuedee@gmail.com>
          Date: 2017-04-14T06:47:00Z

          ROCKETMQ-175 Consumer may miss messages because of inconsistent subscription
          1. For push mode, update subscription version when rebalance result changed.


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user vsair opened a pull request: https://github.com/apache/incubator-rocketmq/pull/92 ROCKETMQ-175 Consumer may miss messages because of inconsistent sub… For push mode, update subscription version when rebalance result changed, then send heart beat to broker immediately. Maybe subscriptions management mechanism could be re-designed to meet more complicated scenarios. You can merge this pull request into a Git repository by running: $ git pull https://github.com/vsair/incubator-rocketmq ROCKETMQ-175 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-rocketmq/pull/92.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #92 commit 61901647fb1851bbb36b4734195206a364a0c0bd Author: vsair <liuxuedee@gmail.com> Date: 2017-04-14T06:47:00Z ROCKETMQ-175 Consumer may miss messages because of inconsistent subscription 1. For push mode, update subscription version when rebalance result changed.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user coveralls commented on the issue:

          https://github.com/apache/incubator-rocketmq/pull/92

          [![Coverage Status](https://coveralls.io/builds/11084533/badge)](https://coveralls.io/builds/11084533)

          Coverage decreased (-0.2%) to 34.443% when pulling *61901647fb1851bbb36b4734195206a364a0c0bd on vsair:ROCKETMQ-175* into *7bcb3b3eae1e3c441861f2a3cd79ff54a8e691b9 on apache:develop*.

          Show
          githubbot ASF GitHub Bot added a comment - Github user coveralls commented on the issue: https://github.com/apache/incubator-rocketmq/pull/92 [! [Coverage Status] ( https://coveralls.io/builds/11084533/badge)](https://coveralls.io/builds/11084533 ) Coverage decreased (-0.2%) to 34.443% when pulling * 61901647fb1851bbb36b4734195206a364a0c0bd on vsair: ROCKETMQ-175 * into * 7bcb3b3eae1e3c441861f2a3cd79ff54a8e691b9 on apache:develop *.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user coveralls commented on the issue:

          https://github.com/apache/incubator-rocketmq/pull/92

          [![Coverage Status](https://coveralls.io/builds/11084533/badge)](https://coveralls.io/builds/11084533)

          Coverage decreased (-0.2%) to 34.443% when pulling *61901647fb1851bbb36b4734195206a364a0c0bd on vsair:ROCKETMQ-175* into *7bcb3b3eae1e3c441861f2a3cd79ff54a8e691b9 on apache:develop*.

          Show
          githubbot ASF GitHub Bot added a comment - Github user coveralls commented on the issue: https://github.com/apache/incubator-rocketmq/pull/92 [! [Coverage Status] ( https://coveralls.io/builds/11084533/badge)](https://coveralls.io/builds/11084533 ) Coverage decreased (-0.2%) to 34.443% when pulling * 61901647fb1851bbb36b4734195206a364a0c0bd on vsair: ROCKETMQ-175 * into * 7bcb3b3eae1e3c441861f2a3cd79ff54a8e691b9 on apache:develop *.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user coveralls commented on the issue:

          https://github.com/apache/incubator-rocketmq/pull/92

          [![Coverage Status](https://coveralls.io/builds/11084533/badge)](https://coveralls.io/builds/11084533)

          Coverage decreased (-0.2%) to 34.443% when pulling *61901647fb1851bbb36b4734195206a364a0c0bd on vsair:ROCKETMQ-175* into *7bcb3b3eae1e3c441861f2a3cd79ff54a8e691b9 on apache:develop*.

          Show
          githubbot ASF GitHub Bot added a comment - Github user coveralls commented on the issue: https://github.com/apache/incubator-rocketmq/pull/92 [! [Coverage Status] ( https://coveralls.io/builds/11084533/badge)](https://coveralls.io/builds/11084533 ) Coverage decreased (-0.2%) to 34.443% when pulling * 61901647fb1851bbb36b4734195206a364a0c0bd on vsair: ROCKETMQ-175 * into * 7bcb3b3eae1e3c441861f2a3cd79ff54a8e691b9 on apache:develop *.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user Jaskey commented on the issue:

          https://github.com/apache/incubator-rocketmq/pull/92

          Actually missing message can not be avoided when the diffrerent tags are subscribed in the same consumer groups.

          Say for two queues q1 q2, two consumers c1 c2.
          c1->TAG1
          c2->TAGB

          when allocating, c1->q1, c2->q2 100 messages are sent to this topic with TAG1.

          C2 will recive nothing and c1 will recieve 50 messages. but the consumer offset of q2 is till moving on. Then 50 messages are missing.

          Show
          githubbot ASF GitHub Bot added a comment - Github user Jaskey commented on the issue: https://github.com/apache/incubator-rocketmq/pull/92 Actually missing message can not be avoided when the diffrerent tags are subscribed in the same consumer groups. Say for two queues q1 q2, two consumers c1 c2. c1->TAG1 c2->TAGB when allocating, c1->q1, c2->q2 100 messages are sent to this topic with TAG1. C2 will recive nothing and c1 will recieve 50 messages. but the consumer offset of q2 is till moving on. Then 50 messages are missing.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user vsair commented on the issue:

          https://github.com/apache/incubator-rocketmq/pull/92

          This PR does't break the restrict that all consumers should have the same subscription. It only focus on broker should hold the online consumer's subscription. Miss messages is just one result caused by inconsistency subscriptions.

          Show
          githubbot ASF GitHub Bot added a comment - Github user vsair commented on the issue: https://github.com/apache/incubator-rocketmq/pull/92 This PR does't break the restrict that all consumers should have the same subscription. It only focus on broker should hold the online consumer's subscription. Miss messages is just one result caused by inconsistency subscriptions.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user Jaskey commented on the issue:

          https://github.com/apache/incubator-rocketmq/pull/92

          @vsair
          What I mean is that the scene this pr is going to fix is breaking the restrict that the same consumer group has the same subscriptions.

          Though this may help improve the refresh the subscriptions, but the same problem is still there remains no solution in the existing design.

          IMO, we should solve this problem directly to solve all the related problem. Which is issued in https://issues.apache.org/jira/browse/ROCKETMQ-94

          Show
          githubbot ASF GitHub Bot added a comment - Github user Jaskey commented on the issue: https://github.com/apache/incubator-rocketmq/pull/92 @vsair What I mean is that the scene this pr is going to fix is breaking the restrict that the same consumer group has the same subscriptions. Though this may help improve the refresh the subscriptions, but the same problem is still there remains no solution in the existing design. IMO, we should solve this problem directly to solve all the related problem. Which is issued in https://issues.apache.org/jira/browse/ROCKETMQ-94
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user vsair commented on the issue:

          https://github.com/apache/incubator-rocketmq/pull/92

          Yeah, you're right. There is no way to solve the problem in existing design.
          But the result is :
          when user stop the consumer has diff subscription, broker could hold the correct info.
          I also look forward to the solution in https://issues.apache.org/jira/browse/ROCKETMQ-94.

          Show
          githubbot ASF GitHub Bot added a comment - Github user vsair commented on the issue: https://github.com/apache/incubator-rocketmq/pull/92 Yeah, you're right. There is no way to solve the problem in existing design. But the result is : when user stop the consumer has diff subscription, broker could hold the correct info. I also look forward to the solution in https://issues.apache.org/jira/browse/ROCKETMQ-94 .
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user dongeforever commented on the issue:

          https://github.com/apache/incubator-rocketmq/pull/92

          @Jaskey It may take a long time to fix the issue ROCKETMQ-94.
          And we could improve it step by step.
          Now one consumer startup will refresh the subscriptions, but the consumer shutdown does not.
          This maybe confusing.
          IMO, it is reasonable to refresh when rebalance occurs.

          Show
          githubbot ASF GitHub Bot added a comment - Github user dongeforever commented on the issue: https://github.com/apache/incubator-rocketmq/pull/92 @Jaskey It may take a long time to fix the issue ROCKETMQ-94 . And we could improve it step by step. Now one consumer startup will refresh the subscriptions, but the consumer shutdown does not. This maybe confusing. IMO, it is reasonable to refresh when rebalance occurs.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user dongeforever commented on the issue:

          https://github.com/apache/incubator-rocketmq/pull/92

          @zhouxinyu @vongosling

          Show
          githubbot ASF GitHub Bot added a comment - Github user dongeforever commented on the issue: https://github.com/apache/incubator-rocketmq/pull/92 @zhouxinyu @vongosling
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 8280388917c466d030ffb774a2474ca8e4144811 in incubator-rocketmq's branch refs/heads/develop from Eric Liu
          [ https://git-wip-us.apache.org/repos/asf?p=incubator-rocketmq.git;h=8280388 ]

          ROCKETMQ-175 Consumer may miss messages because of inconsistent sub… closes apache/incubator-rocketmq#92

          Show
          jira-bot ASF subversion and git services added a comment - Commit 8280388917c466d030ffb774a2474ca8e4144811 in incubator-rocketmq's branch refs/heads/develop from Eric Liu [ https://git-wip-us.apache.org/repos/asf?p=incubator-rocketmq.git;h=8280388 ] ROCKETMQ-175 Consumer may miss messages because of inconsistent sub… closes apache/incubator-rocketmq#92
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user dongeforever commented on the issue:

          https://github.com/apache/incubator-rocketmq/pull/92

          This PR is merged @vsair

          Show
          githubbot ASF GitHub Bot added a comment - Github user dongeforever commented on the issue: https://github.com/apache/incubator-rocketmq/pull/92 This PR is merged @vsair
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user vsair commented on the issue:

          https://github.com/apache/incubator-rocketmq/pull/92

          OK

          Show
          githubbot ASF GitHub Bot added a comment - Github user vsair commented on the issue: https://github.com/apache/incubator-rocketmq/pull/92 OK
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user vsair closed the pull request at:

          https://github.com/apache/incubator-rocketmq/pull/92

          Show
          githubbot ASF GitHub Bot added a comment - Github user vsair closed the pull request at: https://github.com/apache/incubator-rocketmq/pull/92

            People

            • Assignee:
              eric.liu Eric Liu
              Reporter:
              eric.liu Eric Liu
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development