Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 3.2.3, 3.3
    • Fix Version/s: 3.2.4, 3.3.1
    • Component/s: servicemix-core
    • Labels:
      None
    • Environment:

      Java version: 1.6.0_12 and 1.6.0_06
      OS name: "linux" version: "2.6.9-42.elsmp" arch: "i386" Family: "unix"

      Description

      Look at this test code

      import java.util.concurrent.locks.*;
      import java.util.concurrent.*;
      
      /**
       * Test ReentrantReadWrite Lock.
       * This code shows how we cannot get a read lock if the sequence is
       * thread 1:  rwlock.readLock().lock(); //should sucess to get the readLock
       * thread 2:  rwlock.writeLock().lock();//should fail to get the writeLock since the readLock already hold by another thread
       * thread 3: rwlock.readLock().lock(); //should success to get the readLock since no other thread hold the writeLock but failed only with JDK6, (OK with JDK 5)
       */
      public class TestTryWriteInRead {
      
          void log(String s) {
              System.out.printf("%s: %s%n", Thread.currentThread().getName(), s);
          }
      
      
          public static void main(String[] args) throws Exception {
      
              new TestTryWriteInRead().runTest();
          }
      
          public void runTest() throws Exception {
              ReentrantReadWriteLock rwlock = new ReentrantReadWriteLock(false);
      
              // obtain read lock
              rwlock.readLock().lock(); //should sucess to get the readLock
      
              // start 1 threads
              for (int i = 0; i < 1; i++) {
                  new Thread(this.new WriteLockThread(rwlock), "TryWrite" + i).start();
                  Thread.sleep(3000);
                  new Thread(this.new ReadLockThread(rwlock), "TryRead" + i).start();
              }
          }
      
          class WriteLockThread implements Runnable {
      
              private ReentrantReadWriteLock rwlock;
      
              public WriteLockThread(ReentrantReadWriteLock rwlock) {
                  this.rwlock = rwlock;
              }
              public void run() {
                  try {
                      log("try get writelock");
                      rwlock.writeLock().lock(); //should fail to get the writeLock since the readLock already hold by another thread
                      log("can get writelock"); // can't print out
                  } finally {
                  }
      
              }
      
          }
      
          class ReadLockThread implements Runnable {
      
              private ReentrantReadWriteLock rwlock;
      
              public ReadLockThread(ReentrantReadWriteLock rwlock) {
                  this.rwlock = rwlock;
              }
              public void run() {
                  try {
                      log("try get readlock");
                      rwlock.readLock().lock();//should success to get the readLock since no other thread hold the writeLock but failed only with JDK6, (OK with JDK 5)
                      log("can get readlock");//get this print out with JDK5 as expected, but not with JDK6, which means the error in JDK6
                  } finally {
                      log("unlock readlock");
                      rwlock.readLock().unlock();
                  }
      
              }
      
          }
      }
      
      

      Run this code with JDK 5 we get the expected behavior , the console output is
      TryWrite0: try get writelock
      TryRead0: try get readlock
      TryRead0: can get readlock
      TryRead0: unlock readlock,
      rwLock.readLock() can get readLock in trhead with name TryRead0
      But run this code with JDK 6 we get console output like
      TryWrite0: try get writelock
      TryRead0: try get readlock
      which means can't get readLock even there is no other thread hold the writeLock, that's conflict with the JDK API DOC.

      Actually this test code is the abstraction of the behavior when we redeploy SAs, especially at the same time externel client still send message to the endpoint.
      We can find the ReentrantReadWrite used in AbstractFlow, and in some case if the ReentrantReadWrite read and write lock accquire seqence is same as the test code, then the SMX bus hang.

      1. SM-1824.patch
        5 kB
        Freeman Fang

        Activity

        Hide
        Freeman Fang added a comment -

        Just report this bug to SUN, need internal review before it is visible on the Sun Developer Network (SDN).

        Show
        Freeman Fang added a comment - Just report this bug to SUN, need internal review before it is visible on the Sun Developer Network (SDN).
        Hide
        Freeman Fang added a comment -

        SUN just confirm this bug, we get bug id now
        http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6816565
        NOTE: It may take a day or two before this bug shows up in this link

        Show
        Freeman Fang added a comment - SUN just confirm this bug, we get bug id now http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6816565 NOTE: It may take a day or two before this bug shows up in this link
        Hide
        Ron Gavlin added a comment -

        Hi Freeman,

        Is there a way to work-around this bug until a patch for this issue can be incorporated into JDK 6? An SMX bus hang on SA redeployment is a serious problem from my perspective.

        /Ron

        Show
        Ron Gavlin added a comment - Hi Freeman, Is there a way to work-around this bug until a patch for this issue can be incorporated into JDK 6? An SMX bus hang on SA redeployment is a serious problem from my perspective. /Ron
        Hide
        Guillaume Nodet added a comment -

        I think we discussed this issue on IRC ...
        I don't think the api javadoc does not really explicitely defines the behavior when readLock and writeLock are trying to be acquired at the same time.
        Anyway, in smx 3.3 + the new endpoint / assembly behavior (where stopping the sa waits for all exchanges to be processed), the lock may not be needed anymore.
        It needs a bit more investigation.

        Show
        Guillaume Nodet added a comment - I think we discussed this issue on IRC ... I don't think the api javadoc does not really explicitely defines the behavior when readLock and writeLock are trying to be acquired at the same time. Anyway, in smx 3.3 + the new endpoint / assembly behavior (where stopping the sa waits for all exchanges to be processed), the lock may not be needed anymore. It needs a bit more investigation.
        Hide
        Freeman Fang added a comment -

        After hava chat with Guillaume, the possible workaround of this issue for smx 3.2 is use Reentrant lock from Apache Harmony instead the one from Sun JDK.
        And for smx 3.3, since SU honor the stop state (i.e. process exchanges, but do not consumer new external exchagnes), when stopping the SA in smx 3.3, the container will wait for all exchanges to be processed before shutting it down. So we get chance to remove Reentrant writelock, although this need a liitle bit more test.
        I will test both of it.

        Show
        Freeman Fang added a comment - After hava chat with Guillaume, the possible workaround of this issue for smx 3.2 is use Reentrant lock from Apache Harmony instead the one from Sun JDK. And for smx 3.3, since SU honor the stop state (i.e. process exchanges, but do not consumer new external exchagnes), when stopping the SA in smx 3.3, the container will wait for all exchanges to be processed before shutting it down. So we get chance to remove Reentrant writelock, although this need a liitle bit more test. I will test both of it.
        Hide
        Freeman Fang added a comment -

        Did a quick test with using ReentrantReadWriteLock from backport-util-concurrent-2.2.jar, no luck. Both jdk 5 and jdk6 hang. Seems the ReentrantReadWriteLock from backport-util-concurrent-2.2.jar is not what we want.
        Will try the one from Apache Harmony

        Show
        Freeman Fang added a comment - Did a quick test with using ReentrantReadWriteLock from backport-util-concurrent-2.2.jar, no luck. Both jdk 5 and jdk6 hang. Seems the ReentrantReadWriteLock from backport-util-concurrent-2.2.jar is not what we want. Will try the one from Apache Harmony
        Hide
        Freeman Fang added a comment -

        yeah ReentrantReadWriteLock from Apache Harmony works by my initial test

        Show
        Freeman Fang added a comment - yeah ReentrantReadWriteLock from Apache Harmony works by my initial test
        Hide
        Freeman Fang added a comment -

        http://svn.apache.org/viewvc?rev=760773&view=rev
        use ReentrantReadWriteLock from Apache Harmony for 3.2 branch

        Show
        Freeman Fang added a comment - http://svn.apache.org/viewvc?rev=760773&view=rev use ReentrantReadWriteLock from Apache Harmony for 3.2 branch
        Hide
        Freeman Fang added a comment -

        Just test and verify that for smx 3.3, with the appeded patch, don't use flow.suspend(which in turn don't use Reentrant writelock), redeploy SAs is ok, and not hang for both jdk5 and 6.

        Show
        Freeman Fang added a comment - Just test and verify that for smx 3.3, with the appeded patch, don't use flow.suspend(which in turn don't use Reentrant writelock), redeploy SAs is ok, and not hang for both jdk5 and 6.
        Hide
        Freeman Fang added a comment -
        Show
        Freeman Fang added a comment - commit fix http://svn.apache.org/viewvc?rev=760819&view=rev for trunk

          People

          • Assignee:
            Freeman Fang
            Reporter:
            Freeman Fang
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development