Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.5.4
    • Fix Version/s: 2.0
    • Labels:
      None

      Description

      I'm working on a project that uses Hector (Cassandra client). Hector uses commons-pool (we're using 1.5.4) to pool connections to hosts within a Cassandra cluster. Hector provides a JMX MBean that exposes a "NumActive" property, which is the cumulative call to retrieve numActive from all of the individual connection pools. When querying this property via JMS on our production servers, we often see negative values. For example, on a server that has three connection pools, the "NumActive" property reported was -3899.

      I know this issue has been reported before (POOL-29), and was supposedly fixed. The fix discussed there was to merely check the value of _numActive to prevent it from going negative. However, that does not fix the real problem here, which is that it is possible to decrement _numActive more than once for each activated object.

      For example, from a quick look at the code (GenericObjectPool.java, v1.5.4), it would be possible to do the following:

      1) Create a pool with 10 objects.
      2) Borrow all 10 objects from the pool.
      3) Call getNumActive (returns 10).
      4) Call invalidateObject for ONE of the objects 11 times.
      5) Call getNumActive (returns -1).

      The invalidateObject method calls the _factory to destroy the object, and subsequent calls to destroy the same object may or may not result in an exception. Regardless, _numActive is decremented within a finally block, and therefore would always be decremented even if the object had already been invalidated and destroyed.

      I'd like to suggest using a HashSet instead of a counter to keep track of active objects. If borrowing an object added it to a HashSet, and returning or invaliding the object removed it from the HashSet (subsequent removes would be no-ops), the example given above would not result in an incorrect value when getNumActive is called (it would just return the current size of the HashSet).

      Note that although unrelated to this bug, it might also be wise to use a HashSet instead of the int counter _numInternalProcessing.

        Issue Links

          Activity

          Hide
          Mark Thomas added a comment -

          +1 to everything Phil said. Marking this as fixed in 2.0

          Show
          Mark Thomas added a comment - +1 to everything Phil said. Marking this as fixed in 2.0
          Hide
          Phil Steitz added a comment -

          In the 1.x versions of pool, there is no guarantee that GenericObjectPool counters will maintain integrity when clients violate the pool contract by returning or invalidating the same object multiple times. See the note in the javadoc for returnObject. I don't think this is something that can be "fixed" in the 1.x line, because 1.x pools do not maintain references to objects checked out to clients. This has been changed in the 2.0 branch. As Thomas points out, 2.0 will throw in these cases.

          As a workaround using the 1.x code, you can either directly use DBCP's AbandonedObjectPool (which will no-op multiple returns / invalidates) or imitate the approach there, which is to subclass GOP and add tracking for checked out objects.

          Obviously, the best solution is to fix whatever bug in the client code is causing the multiple returns or invalidates.

          I am inclined to close this as WONTFIX. Alternatively, we put fix version as 2.0 and agree that throwing is acceptable in these cases.

          Show
          Phil Steitz added a comment - In the 1.x versions of pool, there is no guarantee that GenericObjectPool counters will maintain integrity when clients violate the pool contract by returning or invalidating the same object multiple times. See the note in the javadoc for returnObject. I don't think this is something that can be "fixed" in the 1.x line, because 1.x pools do not maintain references to objects checked out to clients. This has been changed in the 2.0 branch. As Thomas points out, 2.0 will throw in these cases. As a workaround using the 1.x code, you can either directly use DBCP's AbandonedObjectPool (which will no-op multiple returns / invalidates) or imitate the approach there, which is to subclass GOP and add tracking for checked out objects. Obviously, the best solution is to fix whatever bug in the client code is causing the multiple returns or invalidates. I am inclined to close this as WONTFIX. Alternatively, we put fix version as 2.0 and agree that throwing is acceptable in these cases.
          Hide
          Bruno Candido Volpato da Cunha added a comment - - edited

          Hello.

          I am having this problem in Commons-Pool 1.6 too...
          When I return an Object that doesn't exist in Pool (or even when I return a null), the numActive go negative.

          Test.java
          
          	@Test
          	public void testCommonsPool()  {
          		GenericObjectPool<ProgressConnection> pool = ProgressConnectionManager.getInstance().getPool();
          		
          		System.out.println(pool.getNumActive());
          		
          		try {
          			ProgressConnection conn = pool.borrowObject();
          			pool.returnObject(null); //null or conn have the same behavior
          			pool.returnObject(null); //null or conn have the same behavior
          		} catch(Exception e) {
          			e.printStackTrace();
          			
          		}
          		System.out.println(pool.getNumActive());
          		
          		
          	}
          

          The result that I get is:
          0
          -1

          Show
          Bruno Candido Volpato da Cunha added a comment - - edited Hello. I am having this problem in Commons-Pool 1.6 too... When I return an Object that doesn't exist in Pool (or even when I return a null), the numActive go negative. Test.java @Test public void testCommonsPool() { GenericObjectPool<ProgressConnection> pool = ProgressConnectionManager.getInstance().getPool(); System .out.println(pool.getNumActive()); try { ProgressConnection conn = pool.borrowObject(); pool.returnObject( null ); // null or conn have the same behavior pool.returnObject( null ); // null or conn have the same behavior } catch (Exception e) { e.printStackTrace(); } System .out.println(pool.getNumActive()); } The result that I get is: 0 -1
          Hide
          Thomas Neidhart added a comment -

          I tested the described scenario with the latest trunk version (2.0-SNAPSHOT), and the behavior is now different:

              @Test
              public void testNumActive() throws Exception {
                  pool.setMaxTotal(10);
                  Object o = null;
                  for (int i = 0; i < 10; i++) {
                      o = pool.borrowObject();
                  }
                  System.out.println(pool.getNumActive());
                  for (int i = 0; i < 11; i++) {
                      pool.invalidateObject(o);
                  }
                  System.out.println(pool.getNumActive());
              }
          

          An exception is thrown when trying to invalidate an object that has already been removed from the pool.
          Though, there may be a possible race condition in the current invalidateObject method when different threads try to invalidate the same object, as the method provides no synchronization at all.

          In POOL-125 it is mentioned that this problem has been solved, but I fail to see how this is done for the re-factored version.

          Show
          Thomas Neidhart added a comment - I tested the described scenario with the latest trunk version (2.0-SNAPSHOT), and the behavior is now different: @Test public void testNumActive() throws Exception { pool.setMaxTotal(10); Object o = null; for (int i = 0; i < 10; i++) { o = pool.borrowObject(); } System.out.println(pool.getNumActive()); for (int i = 0; i < 11; i++) { pool.invalidateObject(o); } System.out.println(pool.getNumActive()); } An exception is thrown when trying to invalidate an object that has already been removed from the pool. Though, there may be a possible race condition in the current invalidateObject method when different threads try to invalidate the same object, as the method provides no synchronization at all. In POOL-125 it is mentioned that this problem has been solved, but I fail to see how this is done for the re-factored version.
          Hide
          Mark Thomas added a comment -

          pool2 is a major refactoring that is close to the first release. I am just going through doing some clean up (reduce duplication of code and Javadoc, add missing Javadoc, expose some additional attributes via JMX) with a view to doing a release shortly. The core code is pretty stable and passes all the unit tests. It isn't a release yet but you are free to build from source. If you find any problems, they should get fixed pretty quickly.

          Show
          Mark Thomas added a comment - pool2 is a major refactoring that is close to the first release. I am just going through doing some clean up (reduce duplication of code and Javadoc, add missing Javadoc, expose some additional attributes via JMX) with a view to doing a release shortly. The core code is pretty stable and passes all the unit tests. It isn't a release yet but you are free to build from source. If you find any problems, they should get fixed pretty quickly.
          Hide
          Mark Mindenhall added a comment -

          I spoke too soon. The current 1.6 release (that can be downloaded) has exactly the same problem with the invalidateObject method:

              public void invalidateObject(T obj) throws Exception {
                  try {
                      if (_factory != null) {
                          _factory.destroyObject(obj);
                      }
                  } finally {
                      synchronized (this) {
                          _numActive--;
                      }
                      allocate();
                  }
              }
          

          When you suggested version 1.6, I looked at the version of GenericObjectPool in trunk, which fixes the problem. What's the status of that release? It looks like a major refactoring (even the package name has changed from org.apache.commons.pool to org.apache.commons.pool2).

          Show
          Mark Mindenhall added a comment - I spoke too soon. The current 1.6 release (that can be downloaded) has exactly the same problem with the invalidateObject method: public void invalidateObject(T obj) throws Exception { try { if (_factory != null ) { _factory.destroyObject(obj); } } finally { synchronized ( this ) { _numActive--; } allocate(); } } When you suggested version 1.6, I looked at the version of GenericObjectPool in trunk, which fixes the problem. What's the status of that release? It looks like a major refactoring (even the package name has changed from org.apache.commons.pool to org.apache.commons.pool2).
          Hide
          Mark Mindenhall added a comment -

          Thanks, I'll try that...I figured something would break if I just dropped in the generics version.

          Show
          Mark Mindenhall added a comment - Thanks, I'll try that...I figured something would break if I just dropped in the generics version.
          Hide
          Gary Gregory added a comment -

          You can just replace the 1.5 jar with the 1.6 jar without changing any source code.

          Show
          Gary Gregory added a comment - You can just replace the 1.5 jar with the 1.6 jar without changing any source code.
          Hide
          Mark Mindenhall added a comment -

          It looks like 1.6 would solve the problem, but using 1.6 would require me to refactor the Hector code to use generics.

          Show
          Mark Mindenhall added a comment - It looks like 1.6 would solve the problem, but using 1.6 would require me to refactor the Hector code to use generics.
          Hide
          Gary Gregory added a comment -

          Can you try the latest version (1.6)?

          Show
          Gary Gregory added a comment - Can you try the latest version (1.6)?

            People

            • Assignee:
              Unassigned
              Reporter:
              Mark Mindenhall
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development