[HIVE-3777] add a property in the partition to figure out if stats are accurate

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.13.0
• Fix Version/s: 0.13.0
• Component/s: Query Processor
• Assignee: Ashutosh Chauhan Assign to me
• Reporter: Namit Jain
• Created: 07/Dec/12 04:36
• Updated: 30/Oct/14 05:29
• Resolved: 08/Nov/13 15:57

Description

Currently, stats task tries to update the statistics in the table/partition
being updated after the table/partition is loaded. In case of a failure to
update these stats (due to the any reason), the operation either succeeds
(writing inaccurate stats) or fails depending on whether hive.stats.reliable
is set to true. This can be bad for applications who do not always care about
reliable stats, since the query may have taken a long time to execute and then
fail eventually.

Another property should be added to the partition: areStatsAccurate. If hive.stats.reliable is
set to false, and stats could not be computed correctly, the operation would
still succeed, update the stats, but set areStatsAccurate to false.
If the application cares about accurate stats, it can be obtained in the
background.

Issue Links

• is depended upon by: HIVE-4150 optimize queries like 'select count(1) from T where <conditions on partition columns>'
• is related to: HIVE-8648 numRows cannot be set by user
• relates to: HIVE-1653 Ability to enforce correct stats