Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-17267 Long running structured streaming requirements
  3. SPARK-17165

FileStreamSource should not track the list of seen files indefinitely

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 2.0.1, 2.1.0
    • Structured Streaming
    • None

    Description

      FileStreamSource currently tracks all the files seen indefinitely, which means it can run out of memory or overflow.

      Attachments

        Issue Links

          is related to

          Sub-task - The sub-task of the issue SPARK-17235 MetadataLog should support purging old logs

          • Major - Major loss of function.
          • Resolved
          links to

          Pull request #14728 [Github] Pull Request #14728 (petermaxlee)

          Activity

            People

              petermaxlee Peter Lee
              rxin Reynold Xin
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: