[HIVE-18851] make Hive basic stats valid for ACID; clean up and refactor the code

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: Statistics, Transactions
• Labels: ACID
• Assignee: Unassigned
• Reporter: Sergey Shelukhin
• Created: 02/Mar/18 22:13
• Updated: 02/Mar/18 22:50

Description

HIVE-18571 started as a couple small fixes for MM tables, but ended up as a somewhat major cleanup of stats for ACID tables; however it doesn't do that rigorously and not for all cases.
This is a follow-up JIRA to implement stats for ACID properly (potentially also with ACID semantics similar to those of queries, but that could be another follow-up - for now, at least they should be based on the correct set of files).
Overall I've discovered that Hive stats code is spread all over in random places in code base and is brittle and inconsistent, esp. for any complex scenario like ACID tables.
So, instead of making ad-hoc fixes everywhere, I think at the minimum it should be moved to a single spot (so that e.g. BasicStatsTask, BasicStatsTaskNoJob, metastore "quick" stats generation, etc all use the same code with the same logic) and made valid for ACID.

Issue Links

• is related to: HIVE-18571 stats issues for MM tables; ACID doesn't check state for CTAS