Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-9781

KCL Workers should be configurable from Spark configuration

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Incomplete
    • Affects Version/s: 1.4.1
    • Fix Version/s: None
    • Component/s: DStreams

      Description

      Currently the KinesisClientLibConfiguration for KCL Workers is created within the KinesisReceiver and user is allowed to change only basic settings such as endpoint URL, stream name, credentials, etc.

      However, there is no way to tune some advanced settings, e.g. MaxRecords, IdleTimeBetweenReads, FailoverTime, etc.

      We can add these settings to the Spark configuration and parametrize KinesisClientLibConfiguration with them in KinesisReceiver.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              nehaev Anton Nekhaev
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: