Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-11660

Add support for hardware crc of HDFS checksums on ARM aarch64 architecture

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.8.0
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: native
    • Labels:
    • Environment:

      ARM aarch64 development platform

    • Target Version/s:
    • Release Note:
      Add support for aarch64 CRC instructions

      Description

      This patch adds support for hardware crc for ARM's new 64 bit architecture

      The patch is completely conditionalized on _aarch64_

      I have only added support for the non pipelined version as I benchmarked the pipelined version on aarch64 and it showed no performance improvement.

      The aarch64 version supports both Castagnoli and Zlib CRCs as both of these are supported on ARM aarch64 hardwre.

      To benchmark this I modified the test_bulk_crc32 test to print out the time taken to CRC a 1MB dataset 1000 times.

      Before:

      CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 2.55
      CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 2.55

      After:

      CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 0.57
      CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 0.57

      So this represents a 5X performance improvement on raw CRC calculation.

        Attachments

        1. jira-11660.patch
          49 kB
          Edward Nevill

          Issue Links

            Activity

              People

              • Assignee:
                enevill Edward Nevill
                Reporter:
                enevill Edward Nevill
              • Votes:
                0 Vote for this issue
                Watchers:
                10 Start watching this issue

                Dates

                • Due:
                  Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - 48h
                  48h
                  Remaining:
                  Remaining Estimate - 48h
                  48h
                  Logged:
                  Time Spent - Not Specified
                  Not Specified