[IOTDB-832] reconnection failed: the SessionPool has wait for 60 seconds to get a new connection

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.10.0, 0.9.3, 0.11.0, 0.10.1
• Fix Version/s: 0.10.1
• Component/s: Client/Java
• Labels: pull-request-available
• Assignee: Unassigned
• Reporter: Xiangdong Huang
• Created: 10/Aug/20 16:09
• Updated: 11/Aug/20 04:15
• Resolved: 11/Aug/20 04:15

Description

SessionPool can reconnect the server automatically if the connection is broken.

However, if the server is down and therefore session pool reconnects failed, the pool still considers the size of session connections that it owns should be +1, which leads to no more sessions can be created.

 

// Some comments here

23:41:27.902 [Thread-1] DEBUG org.apache.iotdb.session.pool.SessionPool - Create a new Session 127.0.0.1, 6667, root, root
23:41:27.902 [Thread-0] DEBUG org.apache.iotdb.session.pool.SessionPool - Create a new Session 127.0.0.1, 6667, root, root
23:41:27.902 [Thread-2] DEBUG org.apache.iotdb.session.pool.SessionPool - Create a new Session 127.0.0.1, 6667, root, root
23:41:32.923 [Thread-1] DEBUG org.apache.iotdb.session.pool.SessionPool - Remove a broken Session 127.0.0.1, 6667, root, root
23:41:32.923 [Thread-2] WARN org.apache.thrift.transport.TIOStreamTransport - Error closing output stream.
java.net.SocketException: Socket closed
	at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:118)
	at java.net.SocketOutputStream.write(SocketOutputStream.java:155)
	at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
	at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
	at java.io.FilterOutputStream.close(FilterOutputStream.java:158)
	at org.apache.thrift.transport.TIOStreamTransport.close(TIOStreamTransport.java:108)
	at org.apache.thrift.transport.TSocket.close(TSocket.java:235)
	at org.apache.thrift.transport.TFastFramedTransport.close(TFastFramedTransport.java:115)
	at org.apache.iotdb.session.Session.close(Session.java:192)
	at org.apache.iotdb.session.pool.SessionPool.closeSession(SessionPool.java:215)
	at org.apache.iotdb.session.pool.SessionPool.cleanSessionAndMayThrowConnectionException(SessionPool.java:223)
	at org.apache.iotdb.session.pool.SessionPool.insertRecord(SessionPool.java:391)
	at org.apache.iotdb.SessionPoolExample.lambda$insertRecord$0(SessionPoolExample.java:74)
	at java.lang.Thread.run(Thread.java:748)
23:41:32.923 [Thread-0] DEBUG org.apache.iotdb.session.pool.SessionPool - Remove a broken Session 127.0.0.1, 6667, root, root
23:41:32.923 [Thread-0] DEBUG org.apache.iotdb.session.pool.SessionPool - Create a new Session 127.0.0.1, 6667, root, root
23:41:32.923 [Thread-2] DEBUG org.apache.iotdb.session.pool.SessionPool - Remove a broken Session 127.0.0.1, 6667, root, root
23:41:32.923 [Thread-2] DEBUG org.apache.iotdb.session.pool.SessionPool - Create a new Session 127.0.0.1, 6667, root, root
23:41:32.924 [Thread-1] DEBUG org.apache.iotdb.session.pool.SessionPool - Create a new Session 127.0.0.1, 6667, root, root
org.apache.iotdb.rpc.IoTDBConnectionException: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
	at org.apache.iotdb.session.Session.open(Session.java:130)
	at org.apache.iotdb.session.Session.open(Session.java:115)
	at org.apache.iotdb.session.pool.SessionPool.getSession(SessionPool.java:144)
	at org.apache.iotdb.session.pool.SessionPool.insertRecord(SessionPool.java:384)
	at org.apache.iotdb.SessionPoolExample.lambda$insertRecord$0(SessionPoolExample.java:74)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
	at org.apache.thrift.transport.TSocket.open(TSocket.java:226)
	at org.apache.thrift.transport.TFastFramedTransport.open(TFastFramedTransport.java:125)
	at org.apache.iotdb.session.Session.open(Session.java:128)
	... 5 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.net.Socket.connect(Socket.java:589)
	at org.apache.thrift.transport.TSocket.open(TSocket.java:221)
	... 7 more
org.apache.iotdb.rpc.IoTDBConnectionException: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
	at org.apache.iotdb.session.Session.open(Session.java:130)
	at org.apache.iotdb.session.Session.open(Session.java:115)
	at org.apache.iotdb.session.pool.SessionPool.getSession(SessionPool.java:144)
	at org.apache.iotdb.session.pool.SessionPool.insertRecord(SessionPool.java:384)
	at org.apache.iotdb.SessionPoolExample.lambda$insertRecord$0(SessionPoolExample.java:74)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
	at org.apache.thrift.transport.TSocket.open(TSocket.java:226)
	at org.apache.thrift.transport.TFastFramedTransport.open(TFastFramedTransport.java:125)
	at org.apache.iotdb.session.Session.open(Session.java:128)
	... 5 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.net.Socket.connect(Socket.java:589)
	at org.apache.thrift.transport.TSocket.open(TSocket.java:221)
	... 7 more
org.apache.iotdb.rpc.IoTDBConnectionException: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
	at org.apache.iotdb.session.Session.open(Session.java:130)
	at org.apache.iotdb.session.Session.open(Session.java:115)
	at org.apache.iotdb.session.pool.SessionPool.getSession(SessionPool.java:144)
	at org.apache.iotdb.session.pool.SessionPool.insertRecord(SessionPool.java:384)
	at org.apache.iotdb.SessionPoolExample.lambda$insertRecord$0(SessionPoolExample.java:74)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
	at org.apache.thrift.transport.TSocket.open(TSocket.java:226)
	at org.apache.thrift.transport.TFastFramedTransport.open(TFastFramedTransport.java:125)
	at org.apache.iotdb.session.Session.open(Session.java:128)
	... 5 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.net.Socket.connect(Socket.java:589)
	at org.apache.thrift.transport.TSocket.open(TSocket.java:221)
	... 7 more
23:41:42.946 [Thread-0] WARN org.apache.iotdb.session.pool.SessionPool - the SessionPool has wait for 10 seconds to get a new connection: 127.0.0.1:6667 with root, root
23:41:42.946 [Thread-0] WARN org.apache.iotdb.session.pool.SessionPool - current occupied size 0, queue size 0, considered size 3 
org.apache.iotdb.rpc.IoTDBConnectionException: timeout to get a connection from 127.0.0.1:6667
	at org.apache.iotdb.session.pool.SessionPool.getSession(SessionPool.java:126)
	at org.apache.iotdb.session.pool.SessionPool.insertRecord(SessionPool.java:384)
	at org.apache.iotdb.SessionPoolExample.lambda$insertRecord$0(SessionPoolExample.java:74)
	at java.lang.Thread.run(Thread.java:748)

 

Issue Links

• links to: GitHub Pull Request #1610