Hadoop Common
  1. Hadoop Common
  2. HADOOP-8971

Backport: hadoop.util.PureJavaCrc32 cache hit-ratio is low for static data (HADOOP-8926)

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.2.0
    • Fix Version/s: 1.2.0
    • Component/s: util
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Backport cache-aware improvements for PureJavaCrc32 from trunk (HADOOP-8926)

      Description

      Backport (HADOOP-8926) cache-aware improvements made to PureJavaCrc32 to branch-1.

      With patch, observed significant improvement in crc throughput at larger chunk sizes on branch-1

      java.version = 1.6.0_37
      java.runtime.name = Java(TM) SE Runtime Environment
      java.runtime.version = 1.6.0_37-b06
      java.vm.version = 20.12-b01
      java.vm.vendor = Sun Microsystems Inc.
      java.vm.name = Java HotSpot(TM) 64-Bit Server VM
      java.vm.specification.version = 1.0
      java.specification.version = 1.6
      os.arch = amd64
      os.name = Linux
      os.version = 3.5.0-17-generic

      Performance Table (The unit is MB/sec)

      Num Bytes CRC32 PureJavaCrc32 PureJavaCrc32+patch
      1 16.548 163.916 122.289
      2 30.046 213.591 246.657
      4 59.853 301.521 322.862
      8 107.435 491.757 512.487
      16 179.278 509.099 709.656
      32 265.496 651.438 908.673
      64 350.994 766.566 1000.775
      128 410.514 795.719 1073.625
      256 434.938 828.010 1188.074
      512 451.997 883.219 1239.929
      1024 443.369 902.318 1250.906
      2048 427.923 864.701 1228.599
      4096 430.814 863.258 1210.633
      8192 437.079 883.766 1242.756
      16384 433.349 851.653 1211.161
      32768 433.188 906.142 1171.551
      65536 437.420 890.266 1089.157
      131072 431.189 869.041 1180.171
      262144 430.636 910.133 1212.344
      524288 429.365 889.867 1197.842
      1048576 433.012 887.909 1153.719
      2097152 432.520 861.701 1179.221
      4194304 439.791 899.085 1178.736
      8388608 442.422 856.197 1211.953
      16777216 419.015 866.560 1152.946

        Issue Links

          Activity

          Hide
          Gopal V added a comment -

          Backported verbatim (paths changed)

          Show
          Gopal V added a comment - Backported verbatim (paths changed)
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12550644/backport-HADOOP-8926.patch
          against trunk revision .

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1670//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12550644/backport-HADOOP-8926.patch against trunk revision . -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1670//console This message is automatically generated.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          +1 patch looks good.

          Show
          Tsz Wo Nicholas Sze added a comment - +1 patch looks good.
          Hide
          Suresh Srinivas added a comment -

          I committed the patch. Thank you Gopal.

          Show
          Suresh Srinivas added a comment - I committed the patch. Thank you Gopal.
          Hide
          Matt Foley added a comment -

          Closed upon release of Hadoop 1.2.0.

          Show
          Matt Foley added a comment - Closed upon release of Hadoop 1.2.0.

            People

            • Assignee:
              Gopal V
              Reporter:
              Gopal V
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development