Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-38277

Clear write batch after RocksDB state store's commit

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 3.2.1
    • 3.3.2, 3.4.0
    • Structured Streaming
    • None

    Description

      Current write batch would be cleared when loading the next batch, however, this could be improved once the batch just committed to release unused memory earlier.

      Attachments

        Activity

          People

            yunta Yun Tang
            yunta Yun Tang
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: