[SPARK-29166] Add parameters to limit the number of dynamic partitions for data source table

• Type: Bug
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.4.4, 3.0.0
• Component/s: SQL
• Assignee: Unassigned
• Reporter: Lantao Jin
• Created: 19/Sep/19 01:23
• Updated: 30/Apr/20 16:47

Description

Dynamic partition in Hive table has some restrictions to limit the max number of partitions. See https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DML#LanguageManualDML-DynamicPartitionInserts

It's very useful to prevent to create mistake partitions like ID. Also it can protect the NameNode from mass RPC calls of creating.

Data source table also needs similar limitation.

Issue Links

• links to: GitHub Pull Request #25840