Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-37391

SIGNIFICANT bottleneck introduced by fix for SPARK-32001

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.1.0, 3.1.1, 3.1.2, 3.2.0
    • 3.1.3, 3.2.1, 3.3.0
    • SQL
    • None
    • N/A

    Description

      The fix for https://issues.apache.org/jira/browse/SPARK-32001 ( https://github.com/apache/spark/pull/29024/files#diff-345beef18081272d77d91eeca2d9b5534ff6e642245352f40f4e9c9b8922b085R58 ) does not seem to have consider the reality that some apps may rely on being able to establish many JDBC connections simultaneously for performance reasons.

      The fix forces concurrency to 1 when establishing database connections and that strikes me as a significant user impacting change and a significant bottleneck.

      Can anyone propose a workaround for this? I have an app that makes connections to thousands of databases and I can't upgrade to any version >3.1.x because of this significant bottleneck.

       

      Thanks in advance for your help!

      Attachments

        1. so-much-blocking.jpg
          409 kB
          Danny Guinther
        2. spark-regression-dashes.jpg
          300 kB
          Danny Guinther

        Activity

          People

            danny-seismic Danny Guinther
            danny-seismic Danny Guinther
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: