Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-6977

Kafka Streams - java.lang.IllegalStateException: Unexpected error code 2 while fetching data

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Blocker
    • Resolution: Unresolved
    • Affects Version/s: 0.11.0.1
    • Fix Version/s: None
    • Component/s: streams
    • Labels:

      Description

      We are running Kafka Streams 0.11.0.1 with Kafka Broker 0.10.2.1 and constantly run into the following exception: 

      [visitstats-mongowriter-838cf286-be42-4a49-b3ab-3a291617233e-StreamThread-1] INFO org.apache.kafka.streams.processor.internals.StreamThread - stream-thread [visitstats-mongowriter-838cf286-be42-4a49-b3ab-3a291617233e-StreamThread-1] partition assignment took 40 ms.
      current active tasks: [0_0, 0_1, 0_3, 0_4, 0_5, 0_10, 0_12, 0_13, 0_14, 0_15, 0_17, 0_20, 0_21, 0_23, 0_25, 0_28, 0_2, 0_6, 0_7, 0_8, 0_9, 0_11, 0_16, 0_18, 0_19, 0_22, 0_24, 0_26, 0_27, 0_29, 0_30, 0_31]
      current standby tasks: []
      previous active tasks: [0_0, 0_1, 0_3, 0_4, 0_5, 0_10, 0_12, 0_13, 0_14, 0_15, 0_17, 0_20, 0_21, 0_23, 0_25, 0_28]
      [visitstats-mongowriter-838cf286-be42-4a49-b3ab-3a291617233e-StreamThread-1] INFO org.apache.kafka.streams.processor.internals.StreamThread - stream-thread [visitstats-mongowriter-838cf286-be42-4a49-b3ab-3a291617233e-StreamThread-1] State transition from PARTITIONS_ASSIGNED to RUNNING.
      [visitstats-mongowriter-838cf286-be42-4a49-b3ab-3a291617233e-StreamThread-1] INFO org.apache.kafka.streams.KafkaStreams - stream-client [visitstats-mongowriter-838cf286-be42-4a49-b3ab-3a291617233e] State transition from REBALANCING to RUNNING.
      [visitstats-mongowriter-838cf286-be42-4a49-b3ab-3a291617233e-StreamThread-1] ERROR org.apache.kafka.streams.processor.internals.StreamThread - stream-thread [visitstats-mongowriter-838cf286-be42-4a49-b3ab-3a291617233e-StreamThread-1] Encountered the following error during processing:
      at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:480)
      at org.apache.kafka.clients.consumer.internals.Fetcher.parseCompletedFetch(Fetcher.java:886)
      at org.apache.kafka.clients.consumer.internals.Fetcher.fetchedRecords(Fetcher.java:525)
      at org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1086)
      at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1043)
      at org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:536)
      at org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:490)
      java.lang.IllegalStateException: Unexpected error code 2 while fetching data
      at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:457)
      [visitstats-mongowriter-3727734b-3c2f-4775-87d2-838d72ca3bf4-StreamThread-1] INFO org.apache.kafka.streams.processor.internals.StreamThread - stream-thread [visitstats-mongowriter-3727734b-3c2f-4775-87d2-838d72ca3bf4-StreamThread-1] Shutting down
      [visitstats-mongowriter-3727734b-3c2f-4775-87d2-838d72ca3bf4-StreamThread-1] INFO org.apache.kafka.streams.processor.internals.StreamThread - stream-thread [visitstats-mongowriter-3727734b-3c2f-4775-87d2-838d72ca3bf4-StreamThread-1] State transition from RUNNING to PENDING_SHUTDOWN.
      [visitstats-mongowriter-3727734b-3c2f-4775-87d2-838d72ca3bf4-StreamThread-1] INFO org.apache.kafka.clients.producer.KafkaProducer - Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms.
      [visitstats-mongowriter-3727734b-3c2f-4775-87d2-838d72ca3bf4-StreamThread-1] INFO org.apache.kafka.streams.processor.internals.StreamThread - stream-thread [visitstats-mongowriter-3727734b-3c2f-4775-87d2-838d72ca3bf4-StreamThread-1] Stream thread shutdown complete
      [visitstats-mongowriter-3727734b-3c2f-4775-87d2-838d72ca3bf4-StreamThread-1] INFO org.apache.kafka.streams.processor.internals.StreamThread - stream-thread [visitstats-mongowriter-3727734b-3c2f-4775-87d2-838d72ca3bf4-StreamThread-1] State transition from PENDING_SHUTDOWN to DEAD.
      [visitstats-mongowriter-3727734b-3c2f-4775-87d2-838d72ca3bf4-StreamThread-1] INFO org.apache.kafka.streams.KafkaStreams - stream-client [visitstats-mongowriter-3727734b-3c2f-4775-87d2-838d72ca3bf4] State transition from RUNNING to ERROR.
      [visitstats-mongowriter-3727734b-3c2f-4775-87d2-838d72ca3bf4-StreamThread-1] WARN org.apache.kafka.streams.KafkaStreams - stream-client [visitstats-mongowriter-3727734b-3c2f-4775-87d2-838d72ca3bf4] All stream threads have died. The Kafka Streams instance will be in an error state and should be closed.
      6062195 [visitstats-mongowriter-3727734b-3c2f-4775-87d2-838d72ca3bf4-StreamThread-1] FATAL com.zenreach.data.flows.visitstatsmongoexporter.MongoVisitStatsWriter$ - Exiting main on uncaught exception
      java.lang.IllegalStateException: Unexpected error code 2 while fetching data
      at org.apache.kafka.clients.consumer.internals.Fetcher.parseCompletedFetch(Fetcher.java:886)
      at org.apache.kafka.clients.consumer.internals.Fetcher.fetchedRecords(Fetcher.java:525)
      at org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1086)
      at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1043)
      at org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:536)
      at org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:490)
      at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:480)
      at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:457)
      

      Looks like the error is thrown here https://github.com/apache/kafka/blob/0.11.0.1/clients/src/main/java/org/apache/kafka/clients/consumer/internals/Fetcher.java#L886

      Before giving that exception, our Kafka streams job keeps rebalancing and rebalancing. This is a simple job that reads data of Kafka (log compacted topic) and writes it back to MongoDB. It reads from a topic of 32 partitions and runs on AWS ECS with 32 instances (each with one stream thread). Any idea what could be going wrong? 

      Thanks a lot.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              efeller Eugen Feller
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated: