[POOL-347] borrowObject waits for maxWaitMillis over in pool full

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.4.3, 2.5.0, 2.6.0
• Fix Version/s: 2.6.1
• Assignee: Unassigned
• Reporter: Shunsuke Nakamura
• Created: 20/Jul/18 16:43
• Updated: 24/Jul/18 22:12
• Resolved: 22/Jul/18 14:42

Description

Since POOL-303's fix, even if we specify maxWaitMillis, object creation continues waiting for longer time without any hard limit at https://github.com/apache/commons-pool/blob/3e6dfcd61ddcd88b18934738ebda05c84c948a80/src/main/java/org/apache/commons/pool2/impl/GenericObjectPool.java#L848.

Here's the actual stacktrace:

   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x0000000741158358> (a java.lang.Object)
	at java.lang.Object.wait(Object.java:502)
	at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:848)
	- locked <0x0000000741158358> (a java.lang.Object)
	at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:417)
	at org.apache.commons.pool2.impl.TestGenericObjectPool.testReturnBorrowObjectWithingMaxWaitMillis(TestGenericObjectPool.java:2658)

As example of this issue,
we use Jedis2.9 with commons-pool 2.4.3 and maxWaitMillis=500ms in our environment.
However, when master node is down and the connection pool for the node is full, succeeding JedisConnections wait there forever until pool is free.
Therefore, borrowObject (and the aborting) of last connections takes 40 ~ 80 sec at worst case.

In order to avoid such situations, we should set hard limit to wait by reusing maxWaitMillis or another value.

Issue Links

• is caused by: POOL-303 GenericObjectPool's borrowObject may stuck if create() always fail
• relates to: POOL-340 borrowObject is stuck, if create fails
• links to: GitHub Pull Request #10