Major Description
Currently in Spark, statistics are generated by "analyze" commands.
When user updates the table and collects stats in Hive, "totalSize"/"numRows" will be updated in metastore.
Then, in spark side, table stats becomes stale and is different from Hive's stats.
This is expected. Currently, we have two sources of statistics, i.e. Spark's stats and Hive's stats. In our design, once Spark's stats is available, we respect it over Hive's stats.
If a user generated stats at Spark side, it's his responsibility to update Spark's stats by re-running analyze commands.
But we should clearly document in related code the mechanism to choose between these two sources of stats.
Attachments
Issue Links
- links to
