Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-18342

HDFSBackedStateStore can fail to rename files causing snapshotting and recovery to fail

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 2.0.1
    • Fix Version/s: 2.0.2, 2.1.0
    • Component/s: Structured Streaming
    • Labels:
      None
    • Target Version/s:

      Description

      The HDFSBackedStateStore renames temporary files to delta files as it commits new versions. It however doesn't check whether the rename succeeded. If the rename fails, then recovery will not be possible. It should fail during the rename stage.

        Attachments

          Activity

            People

            • Assignee:
              brkyvz Burak Yavuz
              Reporter:
              brkyvz Burak Yavuz
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: