[HADOOP-8971] Backport: hadoop.util.PureJavaCrc32 cache hit-ratio is low for static data (HADOOP-8926)

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.2.0
• Fix Version/s: 1.2.0
• Component/s: util
• Hadoop Flags: Reviewed
• Release Note: Backport cache-aware improvements for PureJavaCrc32 from trunk (HADOOP-8926)
• Assignee: Gopal Vijayaraghavan
• Reporter: Gopal Vijayaraghavan
• Created: 24/Oct/12 16:04
• Updated: 15/May/13 05:15
• Resolved: 25/Oct/12 04:15

Description

Backport (HADOOP-8926) cache-aware improvements made to PureJavaCrc32 to branch-1.

With patch, observed significant improvement in crc throughput at larger chunk sizes on branch-1

java.version = 1.6.0_37
java.runtime.name = Java(TM) SE Runtime Environment
java.runtime.version = 1.6.0_37-b06
java.vm.version = 20.12-b01
java.vm.vendor = Sun Microsystems Inc.
java.vm.name = Java HotSpot(TM) 64-Bit Server VM
java.vm.specification.version = 1.0
java.specification.version = 1.6
os.arch = amd64
os.name = Linux
os.version = 3.5.0-17-generic

Performance Table (The unit is MB/sec)

Num Bytes	CRC32	PureJavaCrc32	PureJavaCrc32+patch
1	16.548	163.916	122.289
2	30.046	213.591	246.657
4	59.853	301.521	322.862
8	107.435	491.757	512.487
16	179.278	509.099	709.656
32	265.496	651.438	908.673
64	350.994	766.566	1000.775
128	410.514	795.719	1073.625
256	434.938	828.010	1188.074
512	451.997	883.219	1239.929
1024	443.369	902.318	1250.906
2048	427.923	864.701	1228.599
4096	430.814	863.258	1210.633
8192	437.079	883.766	1242.756
16384	433.349	851.653	1211.161
32768	433.188	906.142	1171.551
65536	437.420	890.266	1089.157
131072	431.189	869.041	1180.171
262144	430.636	910.133	1212.344
524288	429.365	889.867	1197.842
1048576	433.012	887.909	1153.719
2097152	432.520	861.701	1179.221
4194304	439.791	899.085	1178.736
8388608	442.422	856.197	1211.953
16777216	419.015	866.560	1152.946

Issue Links

• relates to: HADOOP-8926 hadoop.util.PureJavaCrc32 cache hit-ratio is low for static data