[HIVE-15290] Stripe size smaller than specified.

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.2.0, 1.2.1, 2.0.0, 2.0.1, 2.1.0
• Component/s: ORC
• Assignee: Unassigned
• Reporter: Yuxing Yao
• Created: 28/Nov/16 08:20
• Updated: 19/Jan/17 07:37

Description

In Hive-1.2.0, the real stripe size of output orc file will be very small if most of table data are empty, result in too many Column Statistics objects consumes most of the memory.
I found it become better in Hive-2.0.1, but the stripe size still much smaller than expected.
I saw there's a Jira item: https://issues.apache.org/jira/browse/HIVE-13232 moved the compressed = null out of if block, this changes helps a lot, but for completely fix this, another change is needed in `OutStream.getBufferSize()`
I've created the PR:
https://github.com/apache/hive/pull/118
Please take a look.