Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-18580

Use spark.streaming.backpressure.initialRate in DirectKafkaInputDStream

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.2
    • Fix Version/s: None
    • Component/s: DStreams
    • Labels:
      None
    • Target Version/s:

      Description

      Currently the `spark.streaming.kafka.maxRatePerPartition` is used as the initial rate when the backpressure is enabled. This is too exhaustive for the application while it still warms up.
      This is similar to SPARK-11627, applying the solution provided there to DirectKafkaInputDStream.

        Attachments

          Activity

            People

            • Assignee:
              akonopko Oleksandr Konopko
              Reporter:
              omuravskiy Oleg Muravskiy
            • Votes:
              2 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: