Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-7398 Add back-pressure to Spark Streaming (umbrella JIRA)
  3. SPARK-8835

Provide pluggable Congestion Strategies to deal with Streaming load

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: In Progress
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: DStreams
    • Labels:
      None

      Description

      Second part of SPARK-7398 (which has an over-arching, high-level design doc).

      An (outdated) PoC implementation exists.

      Note that throttling in the current (Spark 1.5.0) way pushes down the (fast-overflowing) buffer of received elements down to the TCP stack, which then has to deal with the break. This can be witnessed here :

      https://github.com/skyluc/spark-streaming-testbed/tree/test-runs-006/test-runs-006

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                huitseeker Fran├žois Garillot
                Shepherd:
                Tathagata Das
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated: