Uploaded image for project: 'Samza'
  1. Samza
  2. SAMZA-579

KafkaSystemConsumer drops SSPs on failure

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.9.0
    • Fix Version/s: 0.9.0
    • Component/s: kafka
    • Labels:
      None

      Description

      While running SAMZA-394, I discovered a bug in KafkaSystemConsumer that causes it to stop consuming under failure scenarios. This does not cause data loss, but can wedge a container until it's restarted.

      The trigger appears to be when a BrokerProxy fetches from a broker that's still coming up, and hasn't yet claimed ownership for a TopicAndPartition. When the fetch fails, the BrokerProxy abdicate()s the TopicAndPartition, and KafkaSystemConsumer tries to refresh to get the leader. If there is no leader, the KafkaSystemConsumer drops the SSP. This happens in KafkaSystemConsumer.refreshBrokers.

        Attachments

        1. SAMZA-579-2.patch
          14 kB
          Chris Riccomini
        2. SAMZA-579-1.patch
          11 kB
          Chris Riccomini
        3. SAMZA-579-0.patch
          2 kB
          Chris Riccomini

          Issue Links

            Activity

              People

              • Assignee:
                criccomini Chris Riccomini
                Reporter:
                criccomini Chris Riccomini
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: