Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-11627

Spark Streaming backpressure mechanism has no initial input rate limit,receivers receive data at the maximum speed , it might cause OOM exception

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 1.5.1, 1.6.0
    • 2.0.0
    • DStreams
    • None

    Description

      Spark Streaming backpressure mechanism has no initial input rate limit, receivers receive data at the maximum speed they can reach in the first batch, the data received might exhaust executors memory resources and cause out of memory exception. Eventually the streaming job would failed, the backpressure mechanism become invalid.

      Attachments

        Activity

          People

            junhaoMg junhaoMg
            junhaoMg junhaoMg
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - 72h
                72h
                Remaining:
                Remaining Estimate - 72h
                72h
                Logged:
                Time Spent - Not Specified
                Not Specified