Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-17510

Set Streaming MaxRate Independently For Multiple Streams

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.0.0
    • 2.1.0
    • DStreams
    • None

    Description

      We use multiple DStreams coming from different Kafka topics in a Streaming application.

      Some settings like maxrate and backpressure enabled/disabled would be better passed as config to KafkaUtils.createStream and KafkaUtils.createDirectStream, instead of setting them in SparkConf.

      Being able to set a different maxrate for different streams is an important requirement for us; we currently work-around the problem by using one receiver-based stream and one direct stream.

      We would like to be able to turn on backpressure for only one of the streams as well.

      Attachments

        Activity

          People

            koeninger Cody Koeninger
            jnadler Jeff Nadler
            Votes:
            2 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: