[SPARK-21969] CommandUtils.updateTableStats should call refreshTable

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.3.0
• Fix Version/s: 2.3.0
• Component/s: SQL
• Target Version/s: 2.3.0
• Assignee: Anton Okolnychyi
• Reporter: Bogdan Raducanu
• Created: 10/Sep/17 19:49
• Updated: 19/Sep/17 21:19
• Resolved: 19/Sep/17 21:19

Description

The table is cached so even though statistics are removed, they will still be used by the existing sessions.

spark.range(100).write.saveAsTable("tab1")
sql("analyze table tab1 compute statistics")
sql("explain cost select distinct * from tab1").show(false)

Produces:

Relation[id#103L] parquet, Statistics(sizeInBytes=784.0 B, rowCount=100, hints=none)

spark.range(100).write.mode("append").saveAsTable("tab1")
sql("explain cost select distinct * from tab1").show(false)

After append something, the same stats are used

Relation[id#135L] parquet, Statistics(sizeInBytes=784.0 B, rowCount=100, hints=none)

Manually refreshing the table removes the stats

spark.sessionState.catalog.refreshTable(TableIdentifier("tab1"))
sql("explain cost select distinct * from tab1").show(false)

Relation[id#155L] parquet, Statistics(sizeInBytes=1568.0 B, hints=none)

Issue Links

• is part of: SPARK-21237 Invalidate stats once table data is changed
• links to: [Github] Pull Request #19252 (aokolnychyi)