Uploaded image for project: 'Stratos'
  1. Stratos
  2. STRATOS-1439

Detecting Issues with Read Write Locks

    XMLWordPrintableJSON

    Details

    • Type: Task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 4.1.0
    • Fix Version/s: 4.1.0
    • Component/s: Documentation
    • Labels:
      None

      Description

      Derive the information provided by Imesh in the email thread [1] and add it to the developer guide.

      [1] [Discuss] Detecting Issues with Read Write Locks
      ----------------------
      Hi Devs,

      Currently we have significant amount of read write locks in Stratos codebase to synchronize data structure updates made by multiple threads.

      As I figured out recently there are several possibilities that these locks could cause problems if not properly implemented:

      Problem 1: A thread trying to acquire a write lock while having a read lock.
      Problem 2: Locks acquired but not released

      I just implemented a ReadWriteLock class in stratos common module which could detect these problems. This class make use of java.util.concurrent.locks.ReentrantReadWriteLock and implements additional logic to detect problems. By default problem 2 detection logic is disabled, it can be enabled by setting following system property:

      read.write.lock.monitor.enabled=true

      I have now updated all read write locks with the above class.
      Following is an example scenario where I detected an unreleased lock in the REST API:

      org.apache.stratos.common.exception.LockNotReleasedException
      at org.apache.stratos.common.concurrent.locks.ReadWriteLockMonitor.checkTimeout(ReadWriteLockMonitor.java:71)
      at org.apache.stratos.common.concurrent.locks.ReadWriteLockMonitor.run(ReadWriteLockMonitor.java:50)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
      at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at java.lang.Thread.run(Thread.java:745)
      [2015-01-20 18:30:12,952] ERROR

      {org.apache.stratos.common.concurrent.locks.ReadWriteLockMonitor}

      - System error, lock has not released for 30 seconds: [lock-name] topology [lock-type] Read [thread-id] 405 [thread-name] http-nio-9443-exec-21 [stack-trace]
      java.lang.Thread.getStackTrace(Thread.java:1589)
      org.apache.stratos.common.concurrent.locks.ReadWriteLock.acquireReadLock(ReadWriteLock.java:160)
      org.apache.stratos.messaging.message.receiver.topology.TopologyManager.acquireReadLockForCluster(TopologyManager.java:173)
      org.apache.stratos.rest.endpoint.api.StratosApiV41Utils.addClustersInstancesToGroupInstanceBean(StratosApiV41Utils.java:1015)
      org.apache.stratos.rest.endpoint.api.StratosApiV41Utils.setSubGroupInstances(StratosApiV41Utils.java:1027)
      org.apache.stratos.rest.endpoint.api.StratosApiV41Utils.addGroupsInstancesToApplicationInstanceBean(StratosApiV41Utils.java:979)
      org.apache.stratos.rest.endpoint.api.StratosApiV41Utils.getApplicationRuntime(StratosApiV41Utils.java:963)
      org.apache.stratos.rest.endpoint.api.StratosApiV41.getApplicationRuntime(StratosApiV41.java:579)

      Thanks
      Imesh

        Attachments

          Activity

            People

            • Assignee:
              mariangela Mariangela Hills
              Reporter:
              mariangela Mariangela Hills
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: