[MAPREDUCE-7370] Parallelize MultipleOutputs#close call

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.0
• Fix Version/s: 3.4.0, 3.3.6
• Component/s: mapreduce-client
• Labels: pull-request-available
• Target Version/s: 3.4.0, 3.3.6
• Hadoop Flags: Reviewed
• Assignee: Ashutosh Gupta
• Reporter: Prabhu Joseph
• Created: 29/Nov/21 18:44
• Updated: 11/Feb/24 10:54
• Resolved: 06/Oct/22 23:19

Description

This call takes more time when there are lot of files to close and there is a high latency to close. Parallelize MultipleOutputs#close call to improve the speed.

  public void close() throws IOException {
    for (RecordWriter writer : recordWriters.values()) {
      writer.close(null);
    }
  }

Idea is from [e-mail]

Issue Links

• links to: GitHub Pull Request #4248