[HIVE-5324] Extend record writer and ORC reader/writer interfaces to provide statistics

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.13.0
• Fix Version/s: 0.13.0
• Component/s: ORC, Serializers/Deserializers
• Labels: backward-incompatible orcfile statistics
• Assignee: Prasanth Jayachandran
• Reporter: Prasanth Jayachandran
• Created: 20/Sep/13 00:36
• Updated: 06/Nov/15 01:24
• Resolved: 28/Sep/13 04:27

Description

The current implementation for computing statistics (number of rows and raw data size) happens for every single row processed. The processOp() method in FileSinkOperator gets raw data size for each row from the serde and accumulates the size in hashmap while counting the number of rows. This accumulated statistics is then published to metastore.
In case of ORC, ORC already stores enough statistics internally which can be made use of when publishing the stats to metastore. This will avoid the duplication of work that is happening in the processOp(). Also getting the statistics directly from ORC is very cheap (can directly read from the file footer).

Issue Links

• breaks: HIVE-6952 Hive 0.13 HiveOutputFormat breaks backwards compatibility
• is related to: HIVE-4340 ORC should provide raw data size
• relates to: HIVE-5325 Implement statistics providing ORC writer and reader interfaces