Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-27420

KinesisInputDStream should expose a way to configure CloudWatch metrics

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.3.3
    • Fix Version/s: 3.0.0
    • Component/s: DStreams, Input/Output
    • Labels:
      None

      Description

      KinesisInputDStream currently does not provide a way to disable CloudWatch metrics push. Kinesis client library (KCL) which is used under the hood provide the ability through `withMetrics` methods.

      To make things worse the default level is "DETAILED" which pushes 10s of metrics every 10 seconds. When dealing with multiple streaming jobs this add up pretty quickly, leading to thousands of dollar in cost. 

      Exposing a way to disable/set the proper level of monitoring is critical to us. We had to send invalid credentials and suppress log as a less-than-ideal workaround : see https://stackoverflow.com/questions/41811039/disable-cloudwatch-for-aws-kinesis-at-spark-streaming/55599002#55599002

       

       

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                sekikn Kengo Seki
                Reporter:
                jgagnon1 Jerome Gagnon
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: