Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-9674

Task corruption should also close the producer if necessary

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: streams
    • Labels:
      None

      Description

      The task revive call only transits the task to CREATED mode. It should handle the recreation of task producer as well.

      Sequence is like:

      1. Task hits out of range exception and throws CorruptedException
      2. Task producer closed along with the task
      3. Task revived and rebalance triggered
      4. Task was assigned back to the same thread
      5. Trying to use task producer will throw as it has already been closed.

      The full log:

       

      [2020-03-03T21:56:29-08:00] (streams-soak-trunk-eos_soak_i-0eaa3f3a6a197f876_streamslog) [2020-03-04 05:56:29,070] WARN [stream-soak-test-93df69e6-1d85-4b6a-81a1-c6d554693e3f-StreamThread-3] stream-thread [stream-soak-test-93df69e6-1d85-4b6a-81a1-c6d554693e3f-StreamThread-3] Encountered org.apache.kafka.clients.consumer.OffsetOutOfRangeException fetching records from restore consumer for partitions [stream-soak-test-KSTREAM-AGGREGATE-STATE-STORE-0000000019-changelog-0], it is likely that the consumer's position has fallen out of the topic partition offset range because the topic was truncated or compacted on the broker, marking the corresponding tasks as corrupted and re-initializing it later. (org.apache.kafka.streams.processor.internals.StoreChangelogReader)

      [2020-03-03T21:56:29-08:00] (streams-soak-trunk-eos_soak_i-0eaa3f3a6a197f876_streamslog) [2020-03-04 05:56:29,071] WARN [stream-soak-test-93df69e6-1d85-4b6a-81a1-c6d554693e3f-StreamThread-3] stream-thread [stream-soak-test-93df69e6-1d85-4b6a-81a1-c6d554693e3f-StreamThread-3] Detected the states of tasks {1_0=[stream-soak-test-KSTREAM-AGGREGATE-STATE-STORE-0000000019-changelog-0]} are corrupted. Will close the task as dirty and re-create and bootstrap from scratch. (org.apache.kafka.streams.processor.internals.StreamThread)

       

      [2020-03-03T21:56:30-08:00] (streams-soak-trunk-eos_soak_i-0eaa3f3a6a197f876_streamslog) [2020-03-04 05:56:30,010] INFO [stream-soak-test-93df69e6-1d85-4b6a-81a1-c6d554693e3f-StreamThread-3] [Producer clientId=stream-soak-test-93df69e6-1d85-4b6a-81a1-c6d554693e3f-StreamThread-3-1_0-producer, transactionalId=stream-soak-test-1_0] Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms. (org.apache.kafka.clients.producer.KafkaProducer)

       

       

      [2020-03-03T21:56:30-08:00] (streams-soak-trunk-eos_soak_i-0eaa3f3a6a197f876_streamslog) [2020-03-04 05:56:30,017] INFO [stream-soak-test-93df69e6-1d85-4b6a-81a1-c6d554693e3f-StreamThread-3] stream-thread [stream-soak-test-93df69e6-1d85-4b6a-81a1-c6d554693e3f-StreamThread-3] task [1_0] Closed clean (org.apache.kafka.streams.processor.internals.StreamTask)

       

       

      [2020-03-03T21:56:22-08:00] (streams-soak-trunk-eos_soak_i-0eaa3f3a6a197f876_streamslog) [2020-03-04 05:56:22,827] INFO [stream-soak-test-93df69e6-1d85-4b6a-81a1-c6d554693e3f-StreamThread-3] [Producer clientId=stream-soak-test-93df69e6-1d85-4b6a-81a1-c6d554693e3f-StreamThread-3-1_0-producer, transactionalId=stream-soak-test-1_0] Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms. (org.apache.kafka.clients.producer.KafkaProducer)

      [2020-03-03T21:56:22-08:00] (streams-soak-trunk-eos_soak_i-0eaa3f3a6a197f876_streamslog) [2020-03-04 05:56:22,829] INFO [stream-soak-test-93df69e6-1d85-4b6a-81a1-c6d554693e3f-StreamThread-3] stream-thread [stream-soak-test-93df69e6-1d85-4b6a-81a1-c6d554693e3f-StreamThread-3] task [1_0] Closed dirty (org.apache.kafka.streams.processor.internals.StreamTask)

       

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                bchen225242 Boyang Chen
                Reporter:
                bchen225242 Boyang Chen
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: