[SPARK-37391] SIGNIFICANT bottleneck introduced by fix for SPARK-32001

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1.0, 3.1.1, 3.1.2, 3.2.0
• Fix Version/s: 3.1.3, 3.2.1, 3.3.0
• Component/s: SQL

Environment

N/A

• Assignee: Danny Guinther
• Reporter: Danny Guinther
• Created: 19/Nov/21 19:27
• Updated: 12/Dec/22 18:11
• Resolved: 24/Dec/21 01:11

Description

The fix for https://issues.apache.org/jira/browse/SPARK-32001 ( https://github.com/apache/spark/pull/29024/files#diff-345beef18081272d77d91eeca2d9b5534ff6e642245352f40f4e9c9b8922b085R58 ) does not seem to have consider the reality that some apps may rely on being able to establish many JDBC connections simultaneously for performance reasons.

The fix forces concurrency to 1 when establishing database connections and that strikes me as a significant user impacting change and a significant bottleneck.

Can anyone propose a workaround for this? I have an app that makes connections to thousands of databases and I can't upgrade to any version >3.1.x because of this significant bottleneck.

 

Thanks in advance for your help!

Issue Links

• links to: [Github] Pull Request #34745 (tdg5)
• links to: [Github] Pull Request #34745 (tdg5)
• links to: [Github] Pull Request #34988 (tdg5)
• links to: [Github] Pull Request #34989 (tdg5)
• links to: [Github] Pull Request #34989 (tdg5)