Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-11813

Standby per job mode Dispatchers don't know job's JobSchedulingStatus

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 1.6.4, 1.7.2, 1.8.0, 1.9.3, 1.10.3, 1.11.3, 1.13.1, 1.12.4
    • 1.15.0
    • Runtime / Coordination
    • None

    Description

      At the moment, it can happen that standby Dispatchers in per job mode will restart a terminated job after they gained leadership. The problem is that we currently clear the RunningJobsRegistry once a job has reached a globally terminal state. After the leading Dispatcher terminates, a standby Dispatcher will gain leadership. Without having the information from the RunningJobsRegistry it cannot tell whether the job has been executed or whether the Dispatcher needs to re-execute the job. At the moment, the Dispatcher will assume that there was a fault and hence re-execute the job. This can lead to duplicate results.

      I think we need some way to tell standby Dispatchers that a certain job has been successfully executed. One trivial solution could be to not clean up the RunningJobsRegistry but then we will clutter ZooKeeper.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              trohrmann Till Rohrmann
              Votes:
              0 Vote for this issue
              Watchers:
              17 Start watching this issue

              Dates

                Created:
                Updated: