[SPARK-8680] PropagateTypes is very slow when there are lots of columns

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.3.1, 1.4.0
• Fix Version/s: 1.5.0
• Component/s: SQL
• Target Version/s: 1.5.0
• Assignee: L. C. Hsieh
• Reporter: Davies Liu
• Created: 27/Jun/15 18:05
• Updated: 30/Jun/15 15:18
• Resolved: 30/Jun/15 15:18

Description

The time for PropagateTypes is O(N*N), N is the number of columns, which is very slow if there many columns (>1000)

There easiest optimization could be put `q.inputSet` outside of transformExpressions which could have about 4 times improvement for N=3000

Issue Links

• links to: [Github] Pull Request #7087 (viirya)