[HADOOP-18374] DistCP: Aggregate IOStatistics Counters in MapReduce Counters

• Type: Sub-task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.3.5
• Component/s: tools/distcp
• Target Version/s: 3.3.9
• Assignee: Mehakmeet Singh
• Reporter: Steve Loughran
• Created: 27/Jul/22 16:54
• Updated: 29/Nov/22 15:55

Description

Distcp can collect IOStatisticsContext counter values and report them to the console. it can't do the timings in min/mean/max though, as there's no way to aggregate them properly.

1. Publish statistics to MapReduce counters in the tasks within CopyMapper.copyFileWithRetry().
2. The counters will be automatically logged in Job.monitorAndPrintJob() when DistCp is executed with the -verbose option; no need for changes there.
3. We could also publish the iOStatistic means by publishing sample count and total sum as two separate counters
4. In AbstractContractDistCpTest, add an override point for subclasses to list which metrics they will issue; assert that values are generated.