[AVRO-1241] improve trevni performance on string deserialization

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.7.3
• Fix Version/s: 1.7.4
• Component/s: java
• Release Note: Optimized string de-serialization
• Assignee: Joseph Adler
• Reporter: Joseph Adler
• Created: 01/Feb/13 20:38
• Updated: 27/Feb/13 00:54
• Resolved: 04/Feb/13 21:59

Description

I have been trying to implement a storage function for Apache Pig that writes data in Trevni format. I found that the storage function was very slow when reading whole records.

I did some profiling (with Yourkit) and found that most of the CPU time was being spent in org.apache.trevni.InputBuffer$readString() (specifically in the String() method). I changed to java.nio.charset.CharsetDecoder.decode for deserialization and saw a big improvement. Changes are included in the patch.

Issue Links

• is related to: PIG-3015 Rewrite of AvroStorage
• is related to: HIVE-3585 Integrate Trevni as another columnar oriented file format
• relates to: PIG-3015 Rewrite of AvroStorage