Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-12462

Threads in PENDING_SHUTDOWN entering a rebalance can cause an illegal state exception

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 2.7.0, 2.8.0
    • Fix Version/s: 2.8.0, 2.7.1
    • Component/s: streams
    • Labels:

      Description

      A thread was removed, sending it to the PENDING_SHUTDOWN state, but went through a rebalance before completing the shutdown.

      // [2021-03-07 04:33:39,385] DEBUG [i-07430efc31ad166b7-StreamThread-6] stream-thread [i-07430efc31ad166b7-StreamThread-6] Ignoring request to transit from PENDING_SHUTDOWN to PARTITIONS_REVOKED: only DEAD state is a valid next state (org.apache.kafka.streams.processor.internals.StreamThread)
      

      Inside StreamsRebalanceListener#onPartitionsRevoked, we have

      // 
      if (streamThread.setState(State.PARTITIONS_REVOKED) != null && !partitions.isEmpty())
          taskManager.handleRevocation(partitions);
      

      Since PENDING_SHUTDOWN → PARTITIONS_REVOKED is a disallowed transition, we never invoke TaskManager#handleRevocation. Currently handleRevocation is responsible for preparing any active tasks for close, including committing offsets and writing the checkpoint as well as suspending the task. We can’t close the task in handleRevocation since we still support EAGER rebalancing, which invokes handleRevocation at the beginning of a rebalance on all tasks.

      The tasks that are actually revoked will be closed during TaskManager#handleAssignment . The IllegalStateException is specifically because we don’t suspend the task before attempting to close it, and the direct transition from RUNNING → CLOSED is forbidden.

        Attachments

          Activity

            People

            • Assignee:
              ableegoldman A. Sophie Blee-Goldman
              Reporter:
              wcarlson5 Walker Carlson

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment