Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-4815 Idempotent/transactional Producer (KIP-98)
  3. KAFKA-5376

Transactions: Concurrent transactional consumer reads aborted messages

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.11.0.0
    • Component/s: None
    • Labels:

      Description

      This may be a dup of KAFKA-5355, but the system tests in KAFKA-5366 shows that a concurrent transactional consumer reads aborted messages. For the test in question the clients are bounced 6 times. With a transaction size of 500, we expect 3000 aborted messages. The concurrent consumer regularly over counts by 1000 to 1500 messages, suggesting that some aborted transactions are consumed.

      --------------------------------------------------------------------------------
      test_id:    kafkatest.tests.core.transactions_test.TransactionsTest.test_transactions.failure_mode=clean_bounce.bounce_target=clients
      status:     FAIL
      run time:   1 minute 56.102 seconds
      
      
          Detected 1000 dups in concurrently consumed messages
      Traceback (most recent call last):
        File "/usr/local/lib/python2.7/dist-packages/ducktape/tests/runner_client.py", line 123, in run
          data = self.run_test()
        File "/usr/local/lib/python2.7/dist-packages/ducktape/tests/runner_client.py", line 176, in run_test
          return self.test_context.function(self.test)
        File "/usr/local/lib/python2.7/dist-packages/ducktape/mark/_mark.py", line 321, in wrapper
          return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
        File "/opt/kafka-dev/tests/kafkatest/tests/core/transactions_test.py", line 235, in test_transactions
          assert num_dups_in_concurrent_consumer == 0, "Detected %d dups in concurrently consumed messages" % num_dups_in_concurrent_consumer
      AssertionError: Detected 1000 dups in concurrently consumed messages
      

      This behavior continues even after https://github.com/apache/kafka/pull/3221 was merged.

        Attachments

        1. KAFKA-5376.tar.gz
          4.37 MB
          Apurva Mehta

          Issue Links

            Activity

              People

              • Assignee:
                hachikuji Jason Gustafson
                Reporter:
                apurva Apurva Mehta
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: