[TEZ-4185] Tez may skip file permission update on intermediate output

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.9.2, 0.10.1, 0.9.3
• Assignee: Attila Magyar
• Reporter: Attila Magyar
• Created: 20/May/20 08:12
• Updated: 05/Jun/20 01:17
• Resolved: 05/Jun/20 01:17

Description

Before updating file permissions TEZ check if the permission change is needed with the following conditional:

if (!SPILL_FILE_PERMS.equals(SPILL_FILE_PERMS.applyUMask(FsPermission.getUMask(conf)))) {
  rfs.setPermission(filename, SPILL_FILE_PERMS);
} 

The problem is that the config object used by this conditional can be different than the config used by the file system (rfs). The reason for this is the underlaying file system instance cache in hadoop.

Either we should remove this conditional altogether (it's a local file system, it won't generate NN calls) or modify it to use rfs.getConf().

The end result is a file, which is not readable by the group, because the permission change was skipped due to the umask mismatch between the 2 configs.

Issue Links

• is related to: TEZ-4057 Fix Unsorted broadcast shuffle umasks
• is related to: TEZ-3894 Tez intermediate outputs implicitly rely on permissive umask for shuffle
• is related to: TEZ-4178 Create separate method for repeated logic of ensuring spill file permission