[TEZ-3894] Tez intermediate outputs implicitly rely on permissive umask for shuffle

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.9.2
• Target Version/s: 0.9.2
• Assignee: Jason Darrell Lowe
• Reporter: Jason Darrell Lowe
• Created: 31/Jan/18 21:00
• Updated: 23/Jul/20 13:42
• Resolved: 09/Feb/18 20:06

Description

Tez does not explicitly set the permissions of intermediate output files for shuffle. In a secure cluster the shuffle service is running as a different user than the task, so the output files require group readability in order to serve up the data during the shuffle phase. If the umask is too restrictive (e.g.: 077) then the task's file.out and file.out.index permissions can be too restrictive to allow the shuffle handler to access them.

Issue Links

• is duplicated by: TEZ-4071 shuffle throws exceptions with an external table with multiple hdfs files
• relates to: MAPREDUCE-7033 Map outputs implicitly rely on permissive umask for shuffle
• relates to: HIVE-23518 Tez may skip file permission update on intermediate output
• relates to: TEZ-4185 Tez may skip file permission update on intermediate output