Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-6017 Provide transparent secure communication channel on Yarn
  3. SPARK-11140

Replace file server in driver with RPC-based alternative

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.6.0, 2.0.0
    • Component/s: Spark Core
    • Labels:
      None

      Description

      As part of making configuring encryption easy in Spark, it would be better to use the existing RPC channel between driver and executors to transfer files and jars added to the application.

      This would remove the need to start the HTTP server currently used for that purpose, which needs to be configured to use SSL if encryption is wanted. SSL is kinda hard to configure correctly in a multi-user, distributed environment.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                vanzin Marcelo Masiero Vanzin
                Reporter:
                vanzin Marcelo Masiero Vanzin
              • Votes:
                1 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: