[HADOOP-16601] Add support for hardware crc32 of nativetask checksums on aarch64 arch

• Type: Improvement
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.2.1
• Fix Version/s: site
• Component/s: common
• Labels: performance
• Target Version/s: 3.2.1
• Flags: Patch
• Assignee: Unassigned
• Reporter: MacChen01
• Created: 24/Sep/19 12:26
• Updated: 06/Jan/20 11:34

Description

Add support for aarch64 CRC instructions in nativetask module, optimize the CRC32 and CRC32C.

Use the benchmark tools : nttest , the improvement is quite substantial:  

CRC32 Zlib polynomial 0x04C11DB7

KeyValueType-IO	Before(MB/s)	After(MB/s)	Improvement
TextType-Write	425.98	602.92	+42%
TextType-Read	796.06	1716.59	+116%
BytesType-Write	474.25	686.84	+45%
BytesType-Read	844.96	1955.03	+131%
UnknownType-Write	434.84	608.81	+40%
UnknownType-Read	805.76	1733.82	+115%

 
 
CRC32C  Castagnoli polynomial 0x1EDC6F41
 

KeyValueType-IO	Before(MB/s)	After(MB/s)	Improvement
TextType-Write	423.39	606.55	+43%
TextType-Read	799.20	1783.28	+123%
BytesType-Write	473.95	696.47	+47%
BytesType-Read	846.30	2018.06	+138%
UnknownType-Write	434.07	612.31	+41%
UnknownType-Read	807.16	1783.95	+121%