[HIVE-7144] GC pressure during ORC StringDictionary writes

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.14.0
• Fix Version/s: 0.14.0
• Component/s: File Formats
• Labels: ORC Performance

Environment

ORC Table ~ 12 string columns

• Release Note: Use Text writables directly in ORC dictionaries to avoid String allocations.
• Assignee: Gopal Vijayaraghavan
• Reporter: Gopal Vijayaraghavan
• Created: 29/May/14 20:55
• Updated: 13/Nov/14 19:44
• Resolved: 07/Jul/14 22:31

Description

When ORC string dictionary writes data out, it suffers from bad GC performance due to a few allocations in-loop.

The conversions are as follows

StringTreeWriter::getStringValue() causes 2 conversions

LazyString -> Text (LazyString::getWritableObject)
Text -> String (LazyStringObjectInspector::getPrimitiveJavaObject)

Then StringRedBlackTree::add() does one conversion

String -> Text

This causes some GC pressure with un-necessary String and byte[] array allocations.

Issue Links

• is related to: HIVE-7364 Trunk cannot be built on -Phadoop1 after HIVE-7144