Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-22698

RabbitMQ source does not stop unless message arrives in queue

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    Description

      In a streaming job with multiple RMQSources, a stop-with-savepoint request has unexpected behavior. Regular checkpoints and savepoints complete successfully, it is only the stop-with-savepoint request where this behavior is seen.
       
      Expected Behavior:
      The stop-with-savepoint request stops the job with a FINISHED state.
       
      Actual Behavior:
      The stop-with-savepoint request either times out or hangs indefinitely unless a message arrives in all the queues that the job consumes from after the stop-with-savepoint request is made.
       
      Current workaround:
      Send a sentinel value to each of the queues consumed by the job that the deserialization schema checks in its isEndOfStream method. This is cumbersome and makes it difficult to do stateful upgrades, as coordination with another system is now necessary.
       
       
      The TaskManager thread dump is attached.
       

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            cmick Michał Ciesielczyk
            austince Austin Cawley-Edwards
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment