Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-9645

Records could not find corresponding partition/task

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.6.0
    • Fix Version/s: 2.6.0
    • Component/s: consumer
    • Labels:
      None

      Description

      We could be hitting the illegal state when stream kicks off a rebalance with all tasks closed:

      ```

      [2020-03-03T18:36:09-08:00] (streams-soak-trunk-eos_soak_i-050294ea2392cf355_streamslog) [2020-03-04 02:36:09,105] WARN [stream-soak-test-e639a3a1-bd9d-49e0-896e-df9fe0cd63db-StreamThread-1] stream-thread [stream-soak-test-e639a3a1-bd9d-49e0-896e-df9fe0cd63db-StreamThread-1] Detected that the thread is being fenced. This implies that this thread missed a rebalance and dropped out of the consumer group. Will close out all assigned tasks and rejoin the consumer group. (org.apache.kafka.streams.processor.internals.StreamThread)

      [2020-03-03T18:36:09-08:00] (streams-soak-trunk-eos_soak_i-050294ea2392cf355_streamslog) [2020-03-04 02:36:09,105] INFO [stream-soak-test-e639a3a1-bd9d-49e0-896e-df9fe0cd63db-StreamThread-1] [Consumer clientId=stream-soak-test-e639a3a1-bd9d-49e0-896e-df9fe0cd63db-StreamThread-1-restore-consumer, groupId=null] Subscribed to partition(s): stream-soak-test-KSTREAM-AGGREGATE-STATE-STORE-0000000049-changelog-2 (org.apache.kafka.clients.consumer.KafkaConsumer)

      [2020-03-03T18:36:10-08:00] (streams-soak-trunk-eos_soak_i-050294ea2392cf355_streamslog) [2020-03-04 02:36:09,286] INFO [stream-soak-test-e639a3a1-bd9d-49e0-896e-df9fe0cd63db-StreamThread-1] [Producer clientId=stream-soak-test-e639a3a1-bd9d-49e0-896e-df9fe0cd63db-StreamThread-1-1_1-producer, transactionalId=stream-soak-test-1_1] Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms. (org.apache.kafka.clients.producer.KafkaProducer)

      [2020-03-03T18:36:10-08:00] (streams-soak-trunk-eos_soak_i-050294ea2392cf355_streamslog) [2020-03-04 02:36:09,287] INFO [stream-soak-test-e639a3a1-bd9d-49e0-896e-df9fe0cd63db-StreamThread-1] stream-thread [stream-soak-test-e639a3a1-bd9d-49e0-896e-df9fe0cd63db-StreamThread-1] task [1_1] Closed dirty (org.apache.kafka.streams.processor.internals.StreamTask)

      [2020-03-03T18:36:10-08:00] (streams-soak-trunk-eos_soak_i-050294ea2392cf355_streamslog) [2020-03-04 02:36:09,287] INFO [stream-soak-test-e639a3a1-bd9d-49e0-896e-df9fe0cd63db-StreamThread-1] [Consumer clientId=stream-soak-test-e639a3a1-bd9d-49e0-896e-df9fe0cd63db-StreamThread-1-restore-consumer, groupId=null] Unsubscribed all topics or patterns and assigned partitions (org.apache.kafka.clients.consumer.KafkaConsumer)

      [2020-03-03T18:36:10-08:00] (streams-soak-trunk-eos_soak_i-050294ea2392cf355_streamslog) [2020-03-04 02:36:09,290] INFO [stream-soak-test-e639a3a1-bd9d-49e0-896e-df9fe0cd63db-StreamThread-1] [Producer clientId=stream-soak-test-e639a3a1-bd9d-49e0-896e-df9fe0cd63db-StreamThread-1-3_2-producer, transactionalId=stream-soak-test-3_2] Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms. (org.apache.kafka.clients.producer.KafkaProducer)

      [2020-03-03T18:36:10-08:00] (streams-soak-trunk-eos_soak_i-050294ea2392cf355_streamslog) [2020-03-04 02:36:09,292] INFO [stream-soak-test-e639a3a1-bd9d-49e0-896e-df9fe0cd63db-StreamThread-1] stream-thread [stream-soak-test-e639a3a1-bd9d-49e0-896e-df9fe0cd63db-StreamThread-1] task [3_2] Closed dirty (org.apache.kafka.streams.processor.internals.StreamTask)

      [2020-03-03T18:36:10-08:00] (streams-soak-trunk-eos_soak_i-050294ea2392cf355_streamslog) [2020-03-04 02:36:09,293] ERROR [stream-soak-test-e639a3a1-bd9d-49e0-896e-df9fe0cd63db-StreamThread-1] stream-thread [stream-soak-test-e639a3a1-bd9d-49e0-896e-df9fe0cd63db-StreamThread-1] Unable to locate active task for received-record partition node-name-repartition-1. Current tasks: TaskManager

      [2020-03-03T18:36:10-08:00] (streams-soak-trunk-eos_soak_i-050294ea2392cf355_streamslog) >      MetadataState:

      [2020-03-03T18:36:10-08:00] (streams-soak-trunk-eos_soak_i-050294ea2392cf355_streamslog) >      Tasks:

      (org.apache.kafka.streams.processor.internals.StreamThread)

      [2020-03-03T18:36:10-08:00] (streams-soak-trunk-eos_soak_i-050294ea2392cf355_streamslog) [2020-03-04 02:36:09,293] ERROR [stream-soak-test-e639a3a1-bd9d-49e0-896e-df9fe0cd63db-StreamThread-1] stream-thread [stream-soak-test-e639a3a1-bd9d-49e0-896e-df9fe0cd63db-StreamThread-1] Encountered the following exception during processing and the thread is going to shut down:  (org.apache.kafka.streams.processor.internals.StreamThread)

      [2020-03-03T18:36:10-08:00] (streams-soak-trunk-eos_soak_i-050294ea2392cf355_streamslog) java.lang.NullPointerException: Task was unexpectedly missing for partition node-name-repartition-1

              at org.apache.kafka.streams.processor.internals.StreamThread.addRecordsToTasks(StreamThread.java:984)

              at org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:820)

      ```

      We should be more careful in this case by avoiding processing data when no corresponding task is found.

              at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:751)

              at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:725)

      [2020-03-03T18:36:10-08:00] (streams-soak-trunk-eos_soak_i-050294ea2392cf355_streamslog) [2020-03-04 02:36:09,294] INFO [stream-soak-test-e639a3a1-bd9d-49e0-896e-df9fe0cd63db-StreamThread-1] stream-thread [stream-soak-test-e639a3a1-bd9d-49e0-896e-df9fe0cd63db-StreamThread-1] State transition from RUNNING to PENDING_SHUTDOWN (org.apache.kafka.streams.processor.internals.StreamThread)

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                bchen225242 Boyang Chen
                Reporter:
                bchen225242 Boyang Chen
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: