[MAPREDUCE-7033] Map outputs implicitly rely on permissive umask for shuffle

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 3.1.0, 3.0.1
• Component/s: mrv2
• Target Version/s: 3.1.0, 3.0.1
• Assignee: Jason Darrell Lowe
• Reporter: Jason Darrell Lowe
• Created: 11/Jan/18 19:23
• Updated: 23/Jul/20 13:41
• Resolved: 01/Feb/18 14:23

Description

Map tasks do not explicitly set the permissions of their output files for shuffle. In a secure cluster the shuffle service is running as a different user than the map task, so the output files require group readability in order to serve up the data during the shuffle phase. If the user's UNIX umask is too restrictive (e.g.: 077) then the map task's file.out and file.out.index permissions can be too restrictive to allow the shuffle handler to access them.

Issue Links

• is related to: MAPREDUCE-7159 FrameworkUploader: ensure proper permissions of generated framework tar.gz if restrictive umask is used
• is related to: TEZ-3894 Tez intermediate outputs implicitly rely on permissive umask for shuffle
• relates to: HADOOP-11347 RawLocalFileSystem#mkdir and create should honor umask