Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-1685

retryTimer not canceled on actor restart in Worker and AppClient

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 0.9.0, 0.9.1, 1.0.0
    • 0.9.2, 1.0.0
    • Spark Core
    • None

    Description

      Both deploy.worker.Worker and deploy.client.AppClient try to registerWithMaster when those Actors start. The attempt at registration is accomplished by starting a retryTimer via the Akka scheduler that will use the registered timeout interval and retry number to make repeated attempts to register with all known Masters before giving up and either marking as dead or calling System.exit.

      The receive methods of these actors can, however, throw exceptions, which will lead to the actor restarting, registerWithMaster being called again on restart, and another retryTimer being scheduled without canceling the already running retryTimer. Assuming that all of the rest of the restart logic is correct for these actors (which I don't believe is actually a given), having multiple retryTimers running presents at least a condition in which the restarted actor may not be able to make the full number of retry attempts before an earlier retryTimer takes the "give up" action.

      Canceling the retryTimer in the actor's postStop hook should suffice.

      Attachments

        Activity

          People

            markhamstra Mark Hamstra
            markhamstra Mark Hamstra
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: