[HIVE-266] Improve SerDe performance by using Text instead of String

• Type: Improvement
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 0.4.0
• Fix Version/s: 0.4.0
• Component/s: Serializers/Deserializers
• Hadoop Flags: Incompatible change, Reviewed
• Assignee: Zheng Shao
• Reporter: Zheng Shao
• Created: 02/Feb/09 05:54
• Updated: 17/Dec/11 00:07
• Resolved: 14/Apr/09 22:56

Description

A recent performance study showed that 2 places in Hive code has exhibited large cpu usage percentage:
1. String.getBytes() (UTF-8 encoding)
2. String.split()

We should replace String with Text object to:
1. Avoid UTF-8 decoding and encoding
2. Reuse the Text object and avoid creating new objects for each column in each row like in String.split()

This is expected to give a big (20%+) performance improvement to Hive.

Issue Links

• is blocked by: HIVE-337 LazySimpleSerDe should support multi-level nested array, map, struct types
• is blocked by: HIVE-375 LazySimpleSerDe to directly serialize (append) int/long/byte/short etc to UTF-8 buffer
• is related to: HIVE-270 Add a lazy-deserialized SerDe for space and cpu efficient serialization of rows with primitive types