Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-18547

Decouple I/O encryption key propagation from UserGroupInformation

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.1.0
    • Fix Version/s: 2.1.0
    • Component/s: Spark Core, YARN
    • Labels:
      None

      Description

      Currently, the encryption key used by the shuffle code is propagated using UserGroupInformation and thus only works on YARN. That makes it really painful to write unit tests in core that include encryption functionality.

      We should change that so that writing these tests is possible, and also because that would allow shuffle encryption to work with other cluster managers.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                vanzin Marcelo Masiero Vanzin
                Reporter:
                vanzin Marcelo Masiero Vanzin
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: