[SPARK-8008] JDBC data source can overload the external database system due to high concurrency

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Won't Fix
• Component/s: SQL
• Assignee: Unassigned
• Reporter: Rene Treffer
• Created: 01/Jun/15 08:32
• Updated: 16/Mar/18 00:11
• Resolved: 20/Jul/15 00:41

Description

Spark tries to load as many partitions as possible in parallel, which can in turn overload the database although it would be possible to load all partitions given a lower concurrency.

It would be nice to either limit the maximum concurrency or to at least warn about this behavior.