[SPARK-18661] Creating a partitioned datasource table should not scan all files for table

• Type: Sub-task
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 2.1.0
• Fix Version/s: 2.1.0
• Component/s: SQL
• Target Version/s: 2.1.0
• Assignee: Eric Liang
• Reporter: Eric Liang
• Created: 30/Nov/16 23:43
• Updated: 04/Dec/16 13:54
• Resolved: 04/Dec/16 13:54

Description

Even though in 2.1 creating a partitioned datasource table will not populate the partition data by default (until the user issues MSCK REPAIR TABLE), it seems we still scan the filesystem for no good reason.

We should avoid doing this when the user specifies a schema.

Issue Links

• links to: [Github] Pull Request #16090 (ericl)