Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-6349

ConcurrentModificationException during streams state restoration

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 1.1.0
    • Fix Version/s: 1.1.0
    • Component/s: streams
    • Labels:
      None

      Description

      During application startup and state restoration a ConcurrentModificationException was thrown from AbstractStateManager

      [2017-12-12 10:47:09,840] ERROR [streams-saak-test-client-StreamThread-2] stream-thread [streams-saak-test-client-StreamThread-2] Encountered the following error during processing: (org.apache.kafka.streams.processor.internals.StreamThread)
      java.util.ConcurrentModificationException
              at java.util.LinkedHashMap$LinkedHashIterator.nextNode(LinkedHashMap.java:719)
              at java.util.LinkedHashMap$LinkedEntryIterator.next(LinkedHashMap.java:752)
              at java.util.LinkedHashMap$LinkedEntryIterator.next(LinkedHashMap.java:750)
              at org.apache.kafka.streams.processor.internals.AbstractStateManager.reinitializeStateStoresForPartitions(AbstractStateManager.java:74)
              at org.apache.kafka.streams.processor.internals.ProcessorStateManager.reinitializeStateStoresForPartitions(ProcessorStateManager.java:155)
              at org.apache.kafka.streams.processor.internals.AbstractTask.reinitializeStateStoresForPartitions(AbstractTask.java:229)
              at org.apache.kafka.streams.processor.internals.StoreChangelogReader.restore(StoreChangelogReader.java:94)
              at org.apache.kafka.streams.processor.internals.TaskManager.updateNewAndRestoringTasks(TaskManager.java:325)
              at org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:779)
              at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:750)
              at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:720)
      

      The root cause is due to an OffsetOutOfRangeException that causes the store to be re-initialized:

      [2017-12-12 10:46:04,861] WARN [streams-saak-test-client-StreamThread-2] stream-thread [streams-saak-test-client-StreamThread-2] Restoring StreamTasks failed. Deleting StreamTasks stores to recreate from scratch. (org.apache.kafka.streams.processor.internals.StoreChangelogReader)
      org.apache.kafka.clients.consumer.OffsetOutOfRangeException: Offsets out of range with no configured reset policy for partitions: {streams-soak-test-app-KSTREAM-AGGREGATE-STATE-STORE-0000000048-changelog-9=0}
              at org.apache.kafka.clients.consumer.internals.Fetcher.parseCompletedFetch(Fetcher.java:883)
              at org.apache.kafka.clients.consumer.internals.Fetcher.fetchedRecords(Fetcher.java:528)
              at org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1173)
              at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1106)
              at org.apache.kafka.streams.processor.internals.StoreChangelogReader.restore(StoreChangelogReader.java:84)
              at org.apache.kafka.streams.processor.internals.TaskManager.updateNewAndRestoringTasks(TaskManager.java:325)
              at org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:789)
              at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:750)
              at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:720)
      

        Attachments

        1. streams-error.log
          140 kB
          Damian Guy

          Issue Links

            Activity

              People

              • Assignee:
                damianguy Damian Guy
                Reporter:
                damianguy Damian Guy
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: