Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-7988

Mechanism to control receiver scheduling

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.5.0
    • Component/s: DStreams
    • Labels:
      None
    • Target Version/s:

      Description

      Streaming receivers are currently scheduled by the underlying Spark scheduler in the same way as other tasks. Will help performance if we added a knob to control receiver scheduling. In most cases, even distribution of receivers among workers should yield best results. In some cases, dense scheduling (e.g, all receivers on one node) may perform better. Maybe support for a new configuration parameter spark.receiver.scheduling.mode (or some such) would help. With 'default' as the current scheduling scheme and 'round-robin' for even distribution among workers (and potentially a third mode for 'dense' scheduling).

        Attachments

          Activity

            People

            • Assignee:
              nravi Nishkam Ravi
              Reporter:
              nravi Nishkam Ravi
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: