Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-8926

hadoop.util.PureJavaCrc32 cache hit-ratio is low for static data


    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.3-alpha
    • Fix Version/s: 2.0.3-alpha, 0.23.5
    • Component/s: util
    • Labels:
    • Environment:

      Ubuntu 10.10 i386

    • Release Note:
      Speed up Crc32 by improving the cache hit-ratio of hadoop.util.PureJavaCrc32


      While running microbenchmarks for HDFS write codepath, a significant part of the CPU fraction was consumed by the DataChecksum.update().

      The attached patch converts the static arrays in CRC32 into a single linear array for a performance boost in the inner loop.

      milli-seconds for 1Gig (16400 loop over a 64kb chunk)

      platform original cache-aware improvement
      x86 3894 2304 40.83
      x86_64 2131 1826 14

      The performance improvement on x86 is rather larger than the 64bit case, due to the extra register/stack pressure caused by the static arrays.

      A closer analysis of the PureJavaCrc32 JIT code shows the following assembly fragment

        0x40f1e345: mov    $0x184,%ecx
        0x40f1e34a: mov    0x4415b560(%ecx),%ecx  ;*getstatic T8_5
                                              ; - PureJavaCrc32::update@95 (line 61)
                                              ;   {oop('PureJavaCrc32')}
        0x40f1e350: mov    %ecx,0x2c(%esp)

      Basically, the static variables T8_0 through to T8_7 are being spilled to the stack because of register pressure. The x86_64 case has a lower likelihood of such pessimistic JIT code due to the increased number of registers.


        1. crc32-faster+readable.patch
          17 kB
          Gopal V
        2. crc32-faster+test.patch
          13 kB
          Gopal V
        3. pure-crc32-cache-hit.patch
          6 kB
          Gopal V

          Issue Links



              • Assignee:
                gopalv Gopal V
                gopalv Gopal V
              • Votes:
                0 Vote for this issue
                17 Start watching this issue


                • Created: