Axis
  1. Axis
  2. AXIS-2498

TypeMappingImpl is not thread safe

    Details

      Description

      This issue originates from http://marc.theaimsgroup.com/?l=axis-user&m=114993403032512&w=2.

      Original report and Cyrille's initial response below:

      ================================

      Hello Manuel,

      My understanding is the Stub#createCall invokes methods on an
      instance of TypeMapping that is shared by the AxisEngine via the
      TypeMappingRegistry. Unfortunately, this TypeMappingImpl is not
      threadsafe.

      Here is a a draft patch that should fix it waiting for an official
      patch. Note that I didn't succeed in reproducing the deadlock, I
      patched according to my understanding of the problem.
      Moreover, I preferred to send you this draft/emergency patch before
      doing extensive testings. It works on my samples but I didn't look at
      the impact on performances (it should be limited ).

      HOW TO US THIS PATCH ?

      Here is a draft patch to fix this issue. To use it, the easiest way
      is to unzip axis-1.4-threadSafetyPatch.jar under /WEB-INF/classes
      (classloader order).

      PATCH DESCRIPTION
      TypeMappingImpl.java is the only modified class. Modifications:

      • use synchronized maps instead of plain HashMaps
      • synchronize internalRegister() method to ensure atomicity of access
        to maps in modification
      • synchronize access to "namespaces" variable in setSupportedEncodings
        to ensure atomicity of the clear-add sequence

      Hope this helps,

      Cyrille


      Cyrille Le Clerc
      cyrille.leclerc@pobox.com
      cyrille.leclerc@fr.ibm.com
      +33 6.61.33.69.86

      On 6/10/06, Manuel Mall <manuel@apache.org> wrote:
      > I am aware that this topic has been discussed a number of times, e.g.
      > http://marc.theaimsgroup.com/?l=axis-user&m=113771126214607&w=2.
      >
      > I followed the advice that the locator is thread safe but the stub is
      > not and as I have only stateless calls I have therefore a single
      > instance of the serviceLocator and a new instance of the stub for each
      > call.
      >
      > However, when I have many simultaneous client calls I get
      > random "lock-ups" of the system. The threads always end up stuck in the
      > same state, that is in a call to HashMap.containsKey (see below). The
      > trace snippet is taken from a 'kill -3' output. There were around 20
      > threads all with the identical trace. The only difference being the
      > address in the '- locked <...> (a ...)' line, which is to be expected
      > as I have separate instances of the stub.
      >
      > I had hoped that the fix to
      > https://issues.apache.org/jira/browse/AXIS-2284 which is in Axis 1.4
      > had addressed this issue. But even after switching to Axis 1.4 the
      > problem remains. So it seems to me that there is another concurrent
      > HashMap modification problem left in the Axis code in the area of the
      > Type Mapping Registry.
      >
      > Needless to say that the problem only occurs on a multi CPU system.
      >
      > As I am really not familiar enough with the Axis internals I am
      > wondering if anyone with more insight has a suggestion how to avoid
      > this.
      >
      > Thanks
      >
      > Manuel
      > -----
      > "LandataWebPollRequestEventSlave:" prio=1 tid=0x8b48f0a0 nid=0x25cf
      > runnable [0x8ac74000..0x8ac745f0]
      > at java.util.HashMap.containsKey(HashMap.java:349)
      > at java.util.HashMap$KeySet.contains(HashMap.java:873)
      > at
      > org.apache.axis.encoding.TypeMappingImpl.isRegistered(TypeMappingImpl.java:195)
      > at
      > org.apache.axis.encoding.TypeMappingDelegate.isRegistered(TypeMappingDelegate.java:136)
      > at org.apache.axis.client.Call.registerTypeMapping(Call.java:2280)
      > at org.apache.axis.client.Call.registerTypeMapping(Call.java:2322)
      > at
      > au.com.anstat.oscar.landata.webservices.externalwsdl.Portal_340Stub.createCall(Portal_340Stub.java:2494)
      > - locked <0x91cb2dc8> (a
      > au.com.anstat.oscar.landata.webservices.externalwsdl.Portal_340Stub)
      > at
      > au.com.anstat.oscar.landata.webservices.externalwsdl.Portal_340Stub.propertyCertificateList(Portal_340Stub.java:3979)
      > at
      > au.com.anstat.oscar.common.landata.Operation.doPropertyCertificateList(Operation.java:574)
      >

      1. Screenshot Eclipse Memory Analyzer Hashmap.png
        39 kB
        Simon Frettloeh
      2. AXIS-2498.patch
        8 kB
        Cyrille Le Clerc
      3. SynchronizedMap.java
        4 kB
        Cyrille Le Clerc
      4. TypeMappingImpl.java
        30 kB
        Cyrille Le Clerc

        Issue Links

          Activity

          Hide
          Cyrille Le Clerc added a comment -

          Proposed patch description :
          Synchronize access to the internal maps and an internal array of TypeMappingImpl.

          As the four maps (qName2Pair, class2Pair, pair2SF, pair2DF) work together, Their access are synchronized with a shared mutex. Moreover, as java.util.Collections#synchronizedMap() does not allow to pass the mutex as parameter, I created a org.apache.utils.SynchronizedMap to support this feature.

          Note that the use of a readWrite lock (e.g. java.util.concurrent.locks.ReentrantReadWriteLock) would giver better performances but I didn't find one in axis code nor in its dependencies.
          An idea may be to add backport-util-concurrent.jar or Doug Lea's util.concurrent.jar .

          To ease readibility of this patch, I also attached the source of the two modified classes TypeMappingImpl.java and SynchronizedMap.java

          Show
          Cyrille Le Clerc added a comment - Proposed patch description : Synchronize access to the internal maps and an internal array of TypeMappingImpl. As the four maps (qName2Pair, class2Pair, pair2SF, pair2DF) work together, Their access are synchronized with a shared mutex. Moreover, as java.util.Collections#synchronizedMap() does not allow to pass the mutex as parameter, I created a org.apache.utils.SynchronizedMap to support this feature. Note that the use of a readWrite lock (e.g. java.util.concurrent.locks.ReentrantReadWriteLock) would giver better performances but I didn't find one in axis code nor in its dependencies. An idea may be to add backport-util-concurrent.jar or Doug Lea's util.concurrent.jar . To ease readibility of this patch, I also attached the source of the two modified classes TypeMappingImpl.java and SynchronizedMap.java
          Hide
          Gautam Singh added a comment -

          Has this been integrated or approved? Can this be used in production?

          I

          Show
          Gautam Singh added a comment - Has this been integrated or approved? Can this be used in production? I
          Hide
          Lukas Eder added a comment -

          This seems to be an old yet still active issue. In one of our productive systems, we have the same issues. Is there going to be a patch release axis 1.4.1 in the future, including such fixes?

          BTW: AXIS-2792 seems to be another duplicate of this issue.

          Show
          Lukas Eder added a comment - This seems to be an old yet still active issue. In one of our productive systems, we have the same issues. Is there going to be a patch release axis 1.4.1 in the future, including such fixes? BTW: AXIS-2792 seems to be another duplicate of this issue.
          Hide
          eddie esquivel added a comment -

          Hello. I can confirm that this is still an active issue. Our production environments (multi-cpu) are having this same issue:

          (StuckThreadMaxTime) of "600" seconds. Stack trace:
          java.util.HashMap.put(HashMap.java:385) org.apache.axis.encoding.TypeMappingImpl.internalRegister(TypeMappingImpl.java:269) org.apache.axis.encoding.TypeMappingImpl.register(TypeMappingImpl.java:221) org.apache.axis.encoding.TypeMappingDelegate.register(TypeMappingDelegate.java:73)
          org.apache.axis.client.Call.registerTypeMapping(Call.java:2285)
          org.apache.axis.client.Call.registerTypeMapping(Call.java:2322)

          Any ideas on when this will be fixed?

          Thanks

          Show
          eddie esquivel added a comment - Hello. I can confirm that this is still an active issue. Our production environments (multi-cpu) are having this same issue: (StuckThreadMaxTime) of "600" seconds. Stack trace: java.util.HashMap.put(HashMap.java:385) org.apache.axis.encoding.TypeMappingImpl.internalRegister(TypeMappingImpl.java:269) org.apache.axis.encoding.TypeMappingImpl.register(TypeMappingImpl.java:221) org.apache.axis.encoding.TypeMappingDelegate.register(TypeMappingDelegate.java:73) org.apache.axis.client.Call.registerTypeMapping(Call.java:2285) org.apache.axis.client.Call.registerTypeMapping(Call.java:2322) Any ideas on when this will be fixed? Thanks
          Hide
          Sridhar added a comment -

          We are facing this issue on production. Is there any solution or workaround to this issue.

          Show
          Sridhar added a comment - We are facing this issue on production. Is there any solution or workaround to this issue.
          Hide
          Simon Frettloeh added a comment - - edited

          Also have this on our system. See attached screenshot of an eclipse memory anlyzer thread stack. See "Example of a corrupted hashmap and it's effect leading to an infinite loop".

          Show
          Simon Frettloeh added a comment - - edited Also have this on our system. See attached screenshot of an eclipse memory anlyzer thread stack. See "Example of a corrupted hashmap and it's effect leading to an infinite loop".
          Hide
          Lukas Eder added a comment -

          Guys, the attached patch works for me, you should try it...

          Show
          Lukas Eder added a comment - Guys, the attached patch works for me, you should try it...
          Hide
          Arun Bhat added a comment -

          We ran into a similar issue in production. Still waiting for a thread dump from the client. It looks like https://issues.apache.org/jira/browse/AXIS-2647 is also related. I am wondering if both patches are needed.

          Show
          Arun Bhat added a comment - We ran into a similar issue in production. Still waiting for a thread dump from the client. It looks like https://issues.apache.org/jira/browse/AXIS-2647 is also related. I am wondering if both patches are needed.
          Hide
          srini added a comment -

          Where can we get this(axis-1.4-threadSafetyPatch.jar) patch? Is it available at apache axis downloads?

          Show
          srini added a comment - Where can we get this(axis-1.4-threadSafetyPatch.jar) patch? Is it available at apache axis downloads?
          Hide
          Jean-Francois Larouche added a comment - - edited

          We got this error too in production yesterday.

          Cyrille i checked your SynchronizedMap, did you forget to syncronize clear or it is by purpose?

          I also cannot find the axis-1.4-threadSafetyPatch.jar

          Thanks.

          Show
          Jean-Francois Larouche added a comment - - edited We got this error too in production yesterday. Cyrille i checked your SynchronizedMap, did you forget to syncronize clear or it is by purpose? I also cannot find the axis-1.4-threadSafetyPatch.jar Thanks.
          Hide
          Hudson added a comment -

          Integrated in axis-trunk #200 (See https://builds.apache.org/job/axis-trunk/200/)
          Cleaned up the code in MultithreadTestCase and extended it to cover the scenario described in AXIS-2498. (Revision 1400282)

          Result = SUCCESS
          veithen :
          Files :

          • /axis/axis1/java/trunk/integration/src/test/java/test/wsdl/multithread/Invoker.java
          • /axis/axis1/java/trunk/integration/src/test/java/test/wsdl/multithread/MultithreadTestCase.java
          • /axis/axis1/java/trunk/integration/src/test/java/test/wsdl/multithread/Report.java
          • /axis/axis1/java/trunk/integration/src/test/java/test/wsdl/multithread/StubSupplier.java
          Show
          Hudson added a comment - Integrated in axis-trunk #200 (See https://builds.apache.org/job/axis-trunk/200/ ) Cleaned up the code in MultithreadTestCase and extended it to cover the scenario described in AXIS-2498 . (Revision 1400282) Result = SUCCESS veithen : Files : /axis/axis1/java/trunk/integration/src/test/java/test/wsdl/multithread/Invoker.java /axis/axis1/java/trunk/integration/src/test/java/test/wsdl/multithread/MultithreadTestCase.java /axis/axis1/java/trunk/integration/src/test/java/test/wsdl/multithread/Report.java /axis/axis1/java/trunk/integration/src/test/java/test/wsdl/multithread/StubSupplier.java
          Hide
          Hudson added a comment -

          Integrated in axis-trunk #205 (See https://builds.apache.org/job/axis-trunk/205/)

          • Attempt to increase the probability of observing the issue described in AXIS-2498 in MultithreadTestCase.
          • Ensure that MultithreadTestCase will report hanging threads (instead of blocking the build). (Revision 1400644)

          Result = SUCCESS
          veithen :
          Files :

          • /axis/axis1/java/trunk/integration/src/test/java/test/wsdl/multithread/Invoker.java
          • /axis/axis1/java/trunk/integration/src/test/java/test/wsdl/multithread/MultithreadTestCase.java
          Show
          Hudson added a comment - Integrated in axis-trunk #205 (See https://builds.apache.org/job/axis-trunk/205/ ) Attempt to increase the probability of observing the issue described in AXIS-2498 in MultithreadTestCase. Ensure that MultithreadTestCase will report hanging threads (instead of blocking the build). (Revision 1400644) Result = SUCCESS veithen : Files : /axis/axis1/java/trunk/integration/src/test/java/test/wsdl/multithread/Invoker.java /axis/axis1/java/trunk/integration/src/test/java/test/wsdl/multithread/MultithreadTestCase.java
          Hide
          Andreas Veithen added a comment -

          Using synchronized maps in TypeMappingImpl as proposed by Cyrille Le Clerc may have an undesired impact on performance. In addition it misses the root cause of the issue, which can be found in the generated code in the createCall method of the stub:

          // All the type mapping information is registered
          // when the first call is made.
          // The type mapping information is actually registered in
          // the TypeMappingRegistry of the service, which
          // is the reason why registration is only needed for the first call.
          synchronized (this) {
          if (firstCall()) {
          // must set encoding style before registering serializers
          _call.setEncodingStyle(null);
          for (int i = 0; i < cachedSerFactories.size(); ++i) {
          java.lang.Class cls = (java.lang.Class) cachedSerClasses.get;
          javax.xml.namespace.QName qName =
          (javax.xml.namespace.QName) cachedSerQNames.get;
          java.lang.Object x = cachedSerFactories.get;
          if (x instanceof Class)

          { java.lang.Class sf = (java.lang.Class) cachedSerFactories.get(i); java.lang.Class df = (java.lang.Class) cachedDeserFactories.get(i); _call.registerTypeMapping(cls, qName, sf, df, false); }


          else if (x instanceof javax.xml.rpc.encoding.SerializerFactory)

          { org.apache.axis.encoding.SerializerFactory sf = (org.apache.axis.encoding.SerializerFactory) cachedSerFactories.get(i); org.apache.axis.encoding.DeserializerFactory df = (org.apache.axis.encoding.DeserializerFactory) cachedDeserFactories.get(i); _call.registerTypeMapping(cls, qName, sf, df, false); }


          }
          }
          }

          As noted in the comment, although the code calls registerTypeMapping on the Call object, the type mappings are actually registered in the service. On the other hand, the code synchronizes on the stub instance. If multiple stub instances created using the same service (locator) instance are used concurrently, then it is possible for multiple threads to enter the synchronized block and to attempt to modify the type mappings of the same service instance.

          A quick fix could be to replace synchronized(this) by synchronized(service). However, as noted in AXIS-2880, it would be actually more natural to initialize the type mappings in the locator instead of the stub, and this would solve the concurrency issues once and for all.

          Show
          Andreas Veithen added a comment - Using synchronized maps in TypeMappingImpl as proposed by Cyrille Le Clerc may have an undesired impact on performance. In addition it misses the root cause of the issue, which can be found in the generated code in the createCall method of the stub: // All the type mapping information is registered // when the first call is made. // The type mapping information is actually registered in // the TypeMappingRegistry of the service, which // is the reason why registration is only needed for the first call. synchronized (this) { if (firstCall()) { // must set encoding style before registering serializers _call.setEncodingStyle(null); for (int i = 0; i < cachedSerFactories.size(); ++i) { java.lang.Class cls = (java.lang.Class) cachedSerClasses.get ; javax.xml.namespace.QName qName = (javax.xml.namespace.QName) cachedSerQNames.get ; java.lang.Object x = cachedSerFactories.get ; if (x instanceof Class) { java.lang.Class sf = (java.lang.Class) cachedSerFactories.get(i); java.lang.Class df = (java.lang.Class) cachedDeserFactories.get(i); _call.registerTypeMapping(cls, qName, sf, df, false); } else if (x instanceof javax.xml.rpc.encoding.SerializerFactory) { org.apache.axis.encoding.SerializerFactory sf = (org.apache.axis.encoding.SerializerFactory) cachedSerFactories.get(i); org.apache.axis.encoding.DeserializerFactory df = (org.apache.axis.encoding.DeserializerFactory) cachedDeserFactories.get(i); _call.registerTypeMapping(cls, qName, sf, df, false); } } } } As noted in the comment, although the code calls registerTypeMapping on the Call object, the type mappings are actually registered in the service. On the other hand, the code synchronizes on the stub instance. If multiple stub instances created using the same service (locator) instance are used concurrently, then it is possible for multiple threads to enter the synchronized block and to attempt to modify the type mappings of the same service instance. A quick fix could be to replace synchronized(this) by synchronized(service). However, as noted in AXIS-2880 , it would be actually more natural to initialize the type mappings in the locator instead of the stub, and this would solve the concurrency issues once and for all.
          Hide
          Bhupendra added a comment -

          Hi Hudson,

          I tried to download the latest axis jar "axis-1.4.1-20121216.170351-233.jar" but I am unable to figure out the changes done by you for this particular problem as well as if I try using this jar then my build seems to fail. Could you please highlight the changed files and changes as part of the fix of this problem?

          Thanks,
          Bhupendra.

          Show
          Bhupendra added a comment - Hi Hudson, I tried to download the latest axis jar "axis-1.4.1-20121216.170351-233.jar" but I am unable to figure out the changes done by you for this particular problem as well as if I try using this jar then my build seems to fail. Could you please highlight the changed files and changes as part of the fix of this problem? Thanks, Bhupendra.
          Hide
          peng wang added a comment -

          We also meet this issue in our product. After apply this code diff, it works well. I puzzle why this bug still open? Does Axis has workaround for this issue?

          Show
          peng wang added a comment - We also meet this issue in our product. After apply this code diff, it works well. I puzzle why this bug still open? Does Axis has workaround for this issue?

            People

            • Assignee:
              Unassigned
              Reporter:
              Manuel Mall
            • Votes:
              7 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:

                Development