Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-8677

Flakey test GroupEndToEndAuthorizationTest#testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Reopened
    • Priority: Critical
    • Resolution: Unresolved
    • Affects Version/s: 2.5.0
    • Fix Version/s: None
    • Component/s: core, security, unit tests
    • Labels:

      Description

      https://builds.apache.org/job/kafka-pr-jdk11-scala2.12/6325/console

       
      18:43:39 kafka.api.GroupEndToEndAuthorizationTest > testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl STARTED*18:44:00* kafka.api.GroupEndToEndAuthorizationTest.testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl failed, log available in /home/jenkins/jenkins-slave/workspace/kafka-pr-jdk11-scala2.12/core/build/reports/testOutput/kafka.api.GroupEndToEndAuthorizationTest.testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl.test.stdout*18:44:00* 18:44:00 kafka.api.GroupEndToEndAuthorizationTest > testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl FAILED*18:44:00* org.scalatest.exceptions.TestFailedException: Consumed 0 records before timeout instead of the expected 1 records

      ---------------------------

      I found this flaky test is actually exposing a real bug in consumer: within KafkaConsumer.poll, we have an optimization to try to send the next fetch request before returning the data in order to pipelining the fetch requests:

                      if (!records.isEmpty()) {
                          // before returning the fetched records, we can send off the next round of fetches
                          // and avoid block waiting for their responses to enable pipelining while the user
                          // is handling the fetched records.
                          //
                          // NOTE: since the consumed position has already been updated, we must not allow
                          // wakeups or any other errors to be triggered prior to returning the fetched records.
                          if (fetcher.sendFetches() > 0 || client.hasPendingRequests()) {
                              client.pollNoWakeup();
                          }
      
                          return this.interceptors.onConsume(new ConsumerRecords<>(records));
                      }
      

      As the NOTE mentioned, this pollNoWakeup should NOT throw any exceptions, since at this point the fetch position has been updated. If an exception is thrown here, and the callers decides to capture and continue, those records would never be returned again, causing data loss.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                guozhang Guozhang Wang
                Reporter:
                bchen225242 Boyang Chen
              • Votes:
                0 Vote for this issue
                Watchers:
                12 Start watching this issue

                Dates

                • Created:
                  Updated: