Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-28442

Potentially persist data without leadership

Rank to TopRank to BottomAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersCreate sub-taskConvert to sub-taskLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Incomplete
    • 2.4.3
    • None
    • Deploy
    • None

    Description

      Spark Master could potentially persist data via ZooKeeperPersistenceEngine even if it is not the leader. See the execution order below.

      1. master-1 became the leader.
      2. master-1 received message and wanted to addApplication(or addWorker)
      3. master-1 stuck because of a full gc
      4. master-1 lost leadership on zk. master-2 became the leader.
      5. master-1 received RevokedLeadership message but the message was pending.
      6. master-1 finished persisting data.

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            tison Zili Chen
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment