Hadoop Common
  1. Hadoop Common
  2. HADOOP-7446

Implement CRC32C native code using SSE4.2 instructions

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.0
    • Component/s: native, performance
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Once HADOOP-7445 is implemented, we can get further performance improvements by implementing CRC32C using the hardware support available in SSE4.2. This support should be dynamically enabled based on CPU feature flags, and of course should be ifdeffed properly so that it doesn't break the build on architectures/platforms where it's not available.

      1. hadoop-7446.txt
        8 kB
        Todd Lipcon
      2. hadoop-7446.txt
        7 kB
        Todd Lipcon
      3. hadoop-7446.txt
        7 kB
        Todd Lipcon
      4. pipelined_on_todds_patch.txt
        13 kB
        Kihwal Lee
      5. pipelined_with_todds_patch.txt
        18 kB
        Kihwal Lee
      6. crc-pipeline-fix.txt
        4 kB
        Todd Lipcon
      7. pipelined-crc.patch.txt
        18 kB
        Kihwal Lee

        Issue Links

          Activity

          Todd Lipcon created issue -
          Todd Lipcon made changes -
          Field Original Value New Value
          Link This issue is part of HDFS-2080 [ HDFS-2080 ]
          Todd Lipcon made changes -
          Attachment hadoop-7446.txt [ 12486145 ]
          Todd Lipcon made changes -
          Attachment hadoop-7446.txt [ 12489424 ]
          Todd Lipcon made changes -
          Assignee Todd Lipcon [ tlipcon ]
          Todd Lipcon made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Affects Version/s 0.23.0 [ 12315569 ]
          Fix Version/s 0.23.0 [ 12315569 ]
          Todd Lipcon made changes -
          Attachment hadoop-7446.txt [ 12490302 ]
          Kihwal Lee made changes -
          Attachment pipelined_on_todds_patch.txt [ 12490318 ]
          Attachment pipelined_with_todds_patch.txt [ 12490319 ]
          Todd Lipcon made changes -
          Attachment crc-pipeline-fix.txt [ 12491279 ]
          Kihwal Lee made changes -
          Attachment pipelined-crc.patch.txt [ 12493172 ]
          Todd Lipcon made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Resolution Fixed [ 1 ]
          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Suresh Srinivas made changes -
          Component/s performance [ 12316502 ]

            People

            • Assignee:
              Todd Lipcon
              Reporter:
              Todd Lipcon
            • Votes:
              0 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development