Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-23294

Spark Streaming + Rate source + Console Sink : Receiver MaxRate is violated

    XMLWordPrintableJSON

    Details

      Description

      Using following configs while building SparkSession

      ("spark.streaming.backpressure.enabled", "true")
      ("spark.streaming.receiver.maxRate", "100")
      ("spark.streaming.backpressure.initialRate", "100")

       

      Source: Rate Source with following options.

      rowsPerSecond=100000

      Sink: Console Sink. 

       

      I am expecting the process rate to limit at 100 Rows per Second, but maxRate is ignored and streaming job is processing at 100000 rate.

       

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                maropu Takeshi Yamamuro
                Reporter:
                rmatte Ravinder Matte
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: