Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-28594 Allow event logs for running streaming apps to be rolled over
  3. SPARK-29581

Enable cleanup old event log files

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Closed
    • Major
    • Resolution: Invalid
    • 3.0.0
    • None
    • Spark Core
    • None

    Description

      This issue can be start only with SPARK-29579 is addressed properly.

      After SPARK-29579 Spark would guarantee strong compatibility on both live entities and snapshots, which means snapshot file could replace older origin event log files. This issue tracks the efforts on automatically cleaning up old event logs if snapshot file can replace them, which lets overall size of event log on streaming query to be manageable.

      Attachments

        Issue Links

          is blocked by

          Sub-task - The sub-task of the issue SPARK-29579 Guarantee compatibility of snapshot (live entities, KVstore entities)

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              Unassigned Unassigned
              kabhwan Jungtaek Lim
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: