[HADOOP-18426] Improve the accuracy of MutableStat mean - ASF JIRA

Log work

Agile Board

Rank to Top

Rank to Bottom

Attach files

Attach Screenshot

Bulk Copy Attachments

Bulk Move Attachments

Voters

Stop watching

Watchers

Create sub-task

Convert to sub-task

Move

Link

Clone

Labels

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: Bug
Status: Resolved
Priority: Major
Resolution: Fixed
Affects Version/s: 3.4.0
Fix Version/s: 3.4.0
Component/s: common
Labels:
- pull-request-available

Target Version/s:

3.4.0
Hadoop Flags:

Reviewed

Description

The current MutableStat mean calculation method is more prone to loss accuracy because the sum of samples is too large.
Storing large integers in the double type results in a loss of accuracy. For example, 9223372036854775707 and 9223372036854775708 are both stored as doubles as 9223372036854776000. Therefore, we should try to avoid using the cumulative total sum method to calculate the average, but update the average every time we sample. All in all, we can process each sample on its own to improve mean accuracy.