Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-18342

HDFSBackedStateStore can fail to rename files causing snapshotting and recovery to fail

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 2.0.1
    • Fix Version/s: 2.0.2, 2.1.0
    • Component/s: Structured Streaming
    • Labels:
      None
    • Target Version/s:

      Description

      The HDFSBackedStateStore renames temporary files to delta files as it commits new versions. It however doesn't check whether the rename succeeded. If the rename fails, then recovery will not be possible. It should fail during the rename stage.

        Attachments

          Activity

            People

            • Assignee:
              brkyvz Burak Yavuz
              Reporter:
              brkyvz Burak Yavuz

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment