[TEZ-4057] Fix Unsorted broadcast shuffle umasks

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.9.2
• Fix Version/s: 0.10.0, 0.9.3
• Assignee: Eric Wohlstadter
• Reporter: Gopal Vijayaraghavan
• Created: 27/Mar/19 21:11
• Updated: 25/Aug/20 15:00
• Resolved: 28/Mar/19 20:35

Description

    if (numPartitions == 1 && !pipelinedShuffle) {
      //special case, where in only one partition is available.
      finalOutPath = outputFileHandler.getOutputFileForWrite();
      finalIndexPath = outputFileHandler.getOutputIndexFileForWrite(indexFileSizeEstimate);
      skipBuffers = true;
      writer = new IFile.Writer(conf, rfs, finalOutPath, keyClass, valClass,
          codec, outputRecordsCounter, outputRecordBytesCounter);
    } else {
      skipBuffers = false;
      writer = null;
    }

The broadcast events don't update the file umasks, because they have 1 partition.

total 8.0K
-rw------- 1 hive hadoop 15 Mar 27 20:30 file.out
-rw-r----- 1 hive hadoop 32 Mar 27 20:30 file.out.index

ending up with readable index files and unreadable .out files.

Issue Links

• relates to: HIVE-23518 Tez may skip file permission update on intermediate output
• relates to: TEZ-4185 Tez may skip file permission update on intermediate output