[MAPREDUCE-6931] Remove TestDFSIO "Total Throughput" calculation

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.8.0
• Fix Version/s: 2.9.0, 3.0.0-beta1, 2.7.5, 2.8.3
• Component/s: benchmarks, test
• Hadoop Flags: Reviewed
• Assignee: Dennis Huo
• Reporter: Dennis Huo
• Created: 03/Aug/17 00:33
• Updated: 25/Oct/19 20:27
• Resolved: 30/Aug/17 22:13

Description

The new "Total Throughput" line added in https://issues.apache.org/jira/browse/HDFS-9153 is currently calculated as toMB(size) / ((float)execTime) and claims to be in units of "MB/s", but execTime is in milliseconds; thus, the reported number is 1/1000x the actual value:

    String resultLines[] = {
        "----- TestDFSIO ----- : " + testType,
        "            Date & time: " + new Date(System.currentTimeMillis()),
        "        Number of files: " + tasks,
        " Total MBytes processed: " + df.format(toMB(size)),
        "      Throughput mb/sec: " + df.format(size * 1000.0 / (time * MEGA)),
        "Total Throughput mb/sec: " + df.format(toMB(size) / ((float)execTime)),
        " Average IO rate mb/sec: " + df.format(med),
        "  IO rate std deviation: " + df.format(stdDev),
        "     Test exec time sec: " + df.format((float)execTime / 1000),
        "" };

The different calculated fields can also use toMB and a shared milliseconds-to-seconds conversion to make it easier to keep units consistent.

Issue Links

• links to: GitHub Pull Request #259