Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-24525

Provide an option to limit MemorySink memory usage

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.3.1
    • Fix Version/s: 2.4.0
    • Component/s: Structured Streaming
    • Labels:
      None

      Description

      MemorySink stores stream results in memory and is mostly used for testing and displaying streams, but for large streams, this can OOM the driver. We should add an option to limit the number of rows and the total size of a memory sink and not add any new data once either limit is hit. 

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                mukulmurthy Mukul Murthy
                Reporter:
                mukulmurthy Mukul Murthy
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: