[SPARK-18547] Decouple I/O encryption key propagation from UserGroupInformation

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.1.0
• Fix Version/s: 2.1.0
• Component/s: Spark Core, YARN
• Assignee: Marcelo Masiero Vanzin
• Reporter: Marcelo Masiero Vanzin
• Created: 22/Nov/16 21:21
• Updated: 29/Nov/16 05:11
• Resolved: 29/Nov/16 05:11

Description

Currently, the encryption key used by the shuffle code is propagated using UserGroupInformation and thus only works on YARN. That makes it really painful to write unit tests in core that include encryption functionality.

We should change that so that writing these tests is possible, and also because that would allow shuffle encryption to work with other cluster managers.

Issue Links

• blocks: SPARK-18546 UnsafeShuffleWriter corrupts encrypted shuffle files when merging
• links to: [Github] Pull Request #15981 (vanzin)