[SPARK-22637] CatalogImpl.refresh() has quadratic complexity for a view

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.2.0
• Fix Version/s: 2.2.2, 2.3.0
• Component/s: SQL
• Target Version/s: 2.2.2, 2.3.0
• Assignee: Herman van Hövell
• Reporter: Herman van Hövell
• Created: 28/Nov/17 21:05
• Updated: 29/Nov/17 05:43
• Resolved: 29/Nov/17 00:04

Description

org.apache.spark.sql.internal.CatalogImpl.refreshTable uses foreach(..) to refresh all tables in a view. This traverses all nodes in the subtree and calls LogicalPlan.refresh() on these nodes. However LogicalPlan.refresh() is also refreshing its children, as a result refreshing a large view can be quite expensive.

Issue Links

• links to: [Github] Pull Request #19837 (hvanhovell)