Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-7446

Implement CRC32C native code using SSE4.2 instructions

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.23.0
    • 0.23.0
    • native, performance
    • None
    • Reviewed

    Description

      Once HADOOP-7445 is implemented, we can get further performance improvements by implementing CRC32C using the hardware support available in SSE4.2. This support should be dynamically enabled based on CPU feature flags, and of course should be ifdeffed properly so that it doesn't break the build on architectures/platforms where it's not available.

      Attachments

        1. pipelined-crc.patch.txt
          18 kB
          Kihwal Lee
        2. crc-pipeline-fix.txt
          4 kB
          Todd Lipcon
        3. pipelined_with_todds_patch.txt
          18 kB
          Kihwal Lee
        4. pipelined_on_todds_patch.txt
          13 kB
          Kihwal Lee
        5. hadoop-7446.txt
          7 kB
          Todd Lipcon
        6. hadoop-7446.txt
          7 kB
          Todd Lipcon
        7. hadoop-7446.txt
          8 kB
          Todd Lipcon

        Issue Links

          is part of

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-2080 Speed up DFS read path by lessening checksum overhead

          • Major - Major loss of function.
          • Closed

          Activity

            People

              tlipcon Todd Lipcon
              tlipcon Todd Lipcon
              Votes:
              0 Vote for this issue
              Watchers:
              16 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: