Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-18150 Spark 2.* failes to create partitions for avro files
  3. SPARK-18152

CLONE - FileStreamSource should not track the list of seen files indefinitely

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Invalid
    • None
    • 2.0.1, 2.1.0
    • Structured Streaming
    • None

    Description

      FileStreamSource currently tracks all the files seen indefinitely, which means it can run out of memory or overflow.

      Attachments

        Activity

          People

            petermaxlee Peter Lee
            sunilsbjoshi Sunil Kumar
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: