[DERBY-7123] Opening many client connections at the same time can hang

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 10.14.2.0, 10.15.2.0
• Component/s: Network Server
• Environment: Test run on ; $ uname -a ; Linux mattiasl-laptop2020 5.8.0-63-generic #71-Ubuntu SMP Tue Jul 13 15:59:12 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
• Urgency: Normal
• Issue & fix info: Repro attached
• Bug behavior facts: Crash, Seen in production
• Assignee: Unassigned
• Reporter: Mattias Lundstrom
• Created: 09/Aug/21 08:21
• Updated: 09/Aug/21 14:54

Description

If many connections are opened at the same time towards a Derby network server using ClientConnectionPoolDataSource.getPooledConnection, then this operation may hang indefinitely.

This has been tested with network server version 10.14.2.0 and 10.15.2.0 and the same client versions.

Attached test client program (for 10.15.2.0) to show the behavior. It just repeatedly tries to open 1000 connections in parallel to the same server (in the production system, there were much fewer connections, but that makes it is harder to reproduce). It is generally slow and sometimes seems to hang indefinitely.

Test can be run by just having a local network server and create a database and then

$ javac -cp derbyclient.jar:derbytools.jar DerbyTest.java
$ java -cp .:derbyclient.jar:derbytools.jar DerbyTest localhost 1527 seconddb 

(where "seconddb" is the test database)

Testing seems to show that if the getPooledConnection operation is synchronized, the problem disappears (see commented out code in test program). We are using this workaround in our application for now.

Running the test program, sometimes the connection seems to hang indefinitely or at the very least be extremely slow. In these situations neither client nor server consumes any CPU and the client seems to wait for IO from the server.

Stacktrace from client

java.base@11.0.11/java.net.SocketInputStream.socketRead0(Native Method)
java.base@11.0.11/java.net.SocketInputStream.socketRead(SocketInputStream.java:115)
java.base@11.0.11/java.net.SocketInputStream.read(SocketInputStream.java:168)
java.base@11.0.11/java.net.SocketInputStream.read(SocketInputStream.java:140)
app//org.apache.derby.client.net.Reply.fill(Reply.java:172)
app//org.apache.derby.client.net.Reply.ensureALayerDataInBuffer(Reply.java:216)
app//org.apache.derby.client.net.Reply.readDssHeader(Reply.java:319)
app//org.apache.derby.client.net.Reply.startSameIdChainParse(Reply.java:1085)
app//org.apache.derby.client.net.NetConnectionReply.readExchangeServerAttributes(NetConnectionReply.java:56)
app//org.apache.derby.client.net.NetConnection.readServerAttributesAndKeyExchange(NetConnection.java:764)
app//org.apache.derby.client.net.NetConnection.flowServerAttributesAndKeyExchange(NetConnection.java:708)
app//org.apache.derby.client.net.NetConnection.flowUSRIDONLconnect(NetConnection.java:562)
app//org.apache.derby.client.net.NetConnection.flowConnect(NetConnection.java:359)
app//org.apache.derby.client.net.NetConnection.initialize(NetConnection.java:274)
app//org.apache.derby.client.net.NetConnection.<init>(NetConnection.java:260)
app//org.apache.derby.client.net.ClientJDBCObjectFactoryImpl.newNetConnection(ClientJDBCObjectFactoryImpl.java:327)
app//org.apache.derby.client.ClientPooledConnection.<init>(ClientPooledConnection.java:130)
app//org.apache.derby.client.net.ClientJDBCObjectFactoryImpl.newClientPooledConnection(ClientJDBCObjectFactoryImpl.java:85)
app//org.apache.derby.client.BasicClientDataSource.getPooledConnectionX(BasicClientDataSource.java:1619)
app//org.apache.derby.client.BasicClientDataSource.getPooledConnectionMinion(BasicClientDataSource.java:1560)
app//org.apache.derby.jdbc.ClientConnectionPoolDataSource.getPooledConnection(ClientConnectionPoolDataSource.java:69)