[HIVE-2185] extend table statistics to store the size of uncompressed data (+extend interfaces for collecting other types of statistics)

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.8.0
• Component/s: Serializers/Deserializers, Statistics
• Hadoop Flags: Reviewed
• Release Note: This patch added getSerDeStats() methods to the Serializer and Deserializer interfaces. Consequently, any SerDes which were compiled against the old interfaces will need to be recompiled against the new interfaces in order to work against Hive 0.8.0.
• Assignee: Tomasz Nykiel
• Reporter: Tomasz Nykiel
• Created: 25/May/11 22:27
• Updated: 02/Apr/12 22:04
• Resolved: 03/Jun/11 16:46

Description

Currently, when executing INSERT OVERWRITE and ANALYZE TABLE commands we collect statistics about the number of rows per partition/table. Other statistics (e.g., total table/partition size) are derived from the file system.

Here, we want to collect information about the sizes of uncompressed data, to be able to determine the efficiency of compression.
Currently, a large part of statistics collection mechanism is hardcoded and not-easily extensible for other statistics.
On top of adding the new statistic collected, it would be desirable to extend the collection mechanism, so any new statistics could be added easily.