Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-19779

structured streaming exist needless tmp file

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.0.3, 2.1.1, 2.2.0
    • Fix Version/s: 2.0.3, 2.1.1, 2.2.0
    • Component/s: Structured Streaming
    • Labels:
      None

      Description

      The PR (https://github.com/apache/spark/pull/17012) can to fix restart a Structured Streaming application using hdfs as fileSystem, but also exist a problem that a tmp file of delta file is still reserved in hdfs. And Structured Streaming don't delete the tmp file generated when restart streaming job in future, so we need to delete the tmp file after restart streaming job.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                guifengleaf@gmail.com Feng Gui
                Reporter:
                guifengleaf@gmail.com Feng Gui
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: