Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-17267 Long running structured streaming requirements
  3. SPARK-17235

MetadataLog should support purging old logs

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 2.0.1, 2.1.0
    • Structured Streaming
    • None

    Description

      This is a useful primitive operation to have to support checkpointing and forgetting old logs.

      Attachments

        Issue Links

          relates to

          Sub-task - The sub-task of the issue SPARK-17165 FileStreamSource should not track the list of seen files indefinitely

          • Major - Major loss of function.
          • Resolved
          links to

          Pull request #14802 [Github] Pull Request #14802 (petermaxlee)

          Activity

            People

              petermaxlee Peter Lee
              petermaxlee Peter Lee
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: