[HADOOP-14405] Fix performance regression due to incorrect use of DataChecksum

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.8.0, 3.0.0-alpha1
• Fix Version/s: 2.9.0, 3.0.0-alpha4, 2.8.2
• Component/s: native, performance
• Hadoop Flags: Reviewed
• Flags: Patch
• Assignee: LiXin Ge
• Reporter: LiXin Ge
• Created: 08/May/17 07:40
• Updated: 16/Apr/21 00:23
• Resolved: 10/May/17 17:54

Description

Recently I have upgraded my Hadoop version from 2.6 to 3.0, and I find that the write performance decreased by 13%. After some days comparative analysis, It's seems introduced by HADOOP-10865.

Since James Thomas have done the work that native checksum can run against byte[] arrays instead of just against byte buffers, we may use native method preferential because it runs faster than others.

Tsz-wo Sze and Masatake Iwasaki could you take a look at this to see if it make bad effect on your benchmark test? Todd Lipcon could you help to see if I have make mistakes in this patch?
thanks!

Issue Links

• breaks: HADOOP-17638 Checksum type is always forced to be crc32