[BEAM-9629] JdbcIO seems to run out of connections in the connection pool and freezes pipeline

• Type: Bug
• Status: Resolved
• Priority: P2
• Resolution: Fixed
• Affects Version/s: 2.18.0, 2.19.0, 2.21.0, 2.22.0, 2.23.0
• Fix Version/s: 2.24.0
• Component/s: io-java-jdbc
• Labels: performance
• Environment: Dataflow, Direct Runner on macOS Catalina.
• External issue URL: https://github.com/spotify/scio/issues/2774
• Assignee: Luke Cwik
• Reporter: Boris Shilov
• Created: 28/Mar/20 14:21
• Updated: 22/Apr/21 00:05
• Resolved: 10/Jul/20 17:45

Description

Greetings,

I am using JdbcIO via the Scala wrappers provided in the Scio project. I am trying to read a few dozen tables in parallel from MySQL, but above 8 concurrent SELECT operations the pipeline freezes. With help of the Scio maintainers we've been able to isolate the issue as likely originating in JdbcIO running out of connections in the connection pool and idling indefinitely. The issue occurs both on the Direct Runner and Dataflow.

Please see linked issue for more context: https://github.com/spotify/scio/issues/2774

Issue Links

• links to: GitHub Pull Request #12209