[HADOOP-7333] Performance improvement in PureJavaCrc32

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 0.21.0
• Fix Version/s: 0.23.0
• Component/s: performance, util

Environment

Linux x64

• Hadoop Flags: Reviewed
• Assignee: Eric Caspole
• Reporter: Eric Caspole
• Created: 26/May/11 15:03
• Updated: 24/Jul/12 17:51
• Resolved: 27/May/11 18:58

Description

I would like to propose a small patch to

org.apache.hadoop.util.PureJavaCrc32.update(byte[] b, int off, int len)

Currently the method stores the intermediate result back into the data member "crc." I noticed this method gets
inlined into DataChecksum.update() and that method appears as one of the hotter methods in a simple hprof profile collected while running terasort and gridmix.

If the code is modified to save the temporary result into a local and just once store the final result back into the data member, it results in slightly more efficient hotspot codegen.

I tested this change using the the "org.apache.hadoop.util.TestPureJavaCrc32$PerformanceTest" which is embedded in the existing unit test for this class, TestPureJavaCrc32 on a variety of linux x64 AMD and Intel multi-socket and multi-core systems I have available to test.

The patch removes several stores of the intermediate result to memory yielding a 0%-10% speedup in the "org.apache.hadoop.util.TestPureJavaCrc32$PerformanceTest" which is embedded in the existing unit test for this class, TestPureJavaCrc32.

If you use a debug hotspot JVM with -XX:+PrintOptoAssembly, you can see the intermediate stores such as:

414 movq R9, rsp + #24 # spill
419 movl R9 + #12 (8-bit), RDX # int ! Field PureJavaCrc32.crc
41d xorl R10, RDX # int

The patch results in just one final store of the fully computed value.

Issue Links

• is related to: HADOOP-8617 backport pure Java CRC32 calculator changes to branch-1
• is related to: HADOOP-6148 Implement a pure Java CRC32 calculator