[HADOOP-11660] Add support for hardware crc of HDFS checksums on ARM aarch64 architecture

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.8.0
• Fix Version/s: 2.8.0, 3.0.0-alpha1
• Component/s: native
• Labels: performance

Environment

ARM aarch64 development platform

• Target Version/s: 2.8.0
• Release Note: Add support for aarch64 CRC instructions
• Assignee: Edward Nevill
• Reporter: Edward Nevill
• Due: 09/Mar/15
• Created: 02/Mar/15 15:40
• Updated: 04/Dec/17 15:52
• Resolved: 30/Mar/15 21:03

Description

This patch adds support for hardware crc for ARM's new 64 bit architecture

The patch is completely conditionalized on _aarch64_

I have only added support for the non pipelined version as I benchmarked the pipelined version on aarch64 and it showed no performance improvement.

The aarch64 version supports both Castagnoli and Zlib CRCs as both of these are supported on ARM aarch64 hardwre.

To benchmark this I modified the test_bulk_crc32 test to print out the time taken to CRC a 1MB dataset 1000 times.

Before:

CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 2.55
CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 2.55

After:

CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 0.57
CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 0.57

So this represents a 5X performance improvement on raw CRC calculation.

Issue Links

• relates to: HADOOP-15033 Use java.util.zip.CRC32C for Java 9 and above