Uploaded image for project: 'Axis'
  1. Axis
  2. AXIS-2698

SAX2EventRecorder causes Out of memory during Deserialization

    Details

    • Type: Bug
    • Status: Open
    • Priority: Critical
    • Resolution: Unresolved
    • Affects Version/s: 1.2
    • Fix Version/s: None
    • Labels:
      None
    • Environment:
      IBM AIX 5.2., Client: java 1.4.2 Server: PERL, SOAPLite

      Description

      After undetermined time (3 or 4 days) the java client crashes with an out of memeory exception. Analyzing the head dump showes that during the deserialization the SAX2EventRecorder$objArrayVector.add produces a huge amount of MessageElemets. The returned message is a 3times nested array containg in the inner arry just a strings [ [ [ key~value, ..., key~value ] ] ]. Here a snippit of the array

      [
      [
      [
      'range.type~INTEGER',
      'range.mode~LASTCHANGE',
      'attribute.state~error',
      'lower-limit.value~2',
      'lower-limit.isIncluded~FALSE',
      'type.id~fhkh00',
      'attribute.id~timestamp.last-change-physical',
      'fhkh00.host~lwfis24',
      'fhkh00.id~DVD337SF',
      'fhkh00.instance~A11F2411'
      ],
      [
      'range.type~INTEGER',
      'range.mode~LASTCHANGE',
      'attribute.state~error',
      'upper-limit.value~0',
      'upper-limit.isIncluded~FALSE',
      'type.id~fhkh00',
      'attribute.id~timestamp.last-change-physical',
      'fhkh00.host~lwfis24',
      'fhkh00.id~DVD337SF',
      'fhkh00.instance~A11F2411'
      ],
      [
      'range.type~INTEGER',
      'range.mode~LASTCHANGE',
      'attribute.state~ok',
      'lower-limit.value~0',
      'lower-limit.isIncluded~TRUE',
      'upper-limit.value~2',
      'upper-limit.isIncluded~TRUE',
      'type.id~fhkh00',
      'attribute.id~timestamp.last-change-physical',
      'fhkh00.host~lwfis24',
      'fhkh00.id~DVD337SF',
      'fhkh00.instance~A11F2411'
      ]
      ]
      ]

        Activity

        Hide
        ehxhe gaochao added a comment -

        I also meet the similar problem,
        The thread org.apache.tomcat.util.threads.ThreadWithAttributes @ 0x4f068898 http-0.0.0.0-8080-Processor63 keeps local variables with total size 435,096,504 (46.12%) bytes.
        The memory is accumulated in one instance of "org.apache.axis.encoding.DeserializationContext" loaded by "org.jboss.mx.loading.UnifiedClassLoader3 @ 0x4cc718d8".

        94M used by SAX2EventRecorder, and there are also lots MessageElements, each size of them is 472, maybe totally 2219524 entries.

        Show
        ehxhe gaochao added a comment - I also meet the similar problem, The thread org.apache.tomcat.util.threads.ThreadWithAttributes @ 0x4f068898 http-0.0.0.0-8080-Processor63 keeps local variables with total size 435,096,504 (46.12%) bytes. The memory is accumulated in one instance of "org.apache.axis.encoding.DeserializationContext" loaded by "org.jboss.mx.loading.UnifiedClassLoader3 @ 0x4cc718d8". 94M used by SAX2EventRecorder, and there are also lots MessageElements, each size of them is 472, maybe totally 2219524 entries.
        Hide
        prakash.rao Prakash C Rao added a comment -

        Unfortunately we have chosen Axis 1.4 for our production system and we get out of memory every now & then. I analyzed 6-7 heap dumps and it is all pointing to SAX2EventRecorder.objArrayVector. I see multiple references to objArray of size 1024000. These objects in total are consuming about 400-650 MB of memory and it is causing out of memory.

        The SAX2EventRecorder.objArrayVector gets called for every character parsing & in our case we call the service SOAP API in a loop and after few hours we get out of memory. Is there a way to fix this? Why these objects are not being collected by FULL GC after receiving response and it's being processed for one request. I see clear API in SAX2EventRecorder which assigns a new objArrayVector to events object but don't see getting called anywhere. Isn't objArrayVector to be nullified after response is processed so that it becomes eligible for GC?

        Show
        prakash.rao Prakash C Rao added a comment - Unfortunately we have chosen Axis 1.4 for our production system and we get out of memory every now & then. I analyzed 6-7 heap dumps and it is all pointing to SAX2EventRecorder.objArrayVector. I see multiple references to objArray of size 1024000. These objects in total are consuming about 400-650 MB of memory and it is causing out of memory. The SAX2EventRecorder.objArrayVector gets called for every character parsing & in our case we call the service SOAP API in a loop and after few hours we get out of memory. Is there a way to fix this? Why these objects are not being collected by FULL GC after receiving response and it's being processed for one request. I see clear API in SAX2EventRecorder which assigns a new objArrayVector to events object but don't see getting called anywhere. Isn't objArrayVector to be nullified after response is processed so that it becomes eligible for GC?
        Hide
        sharath2k Sharath added a comment -

        I have a critical issue with deserializing Java Array with 3000+ length
        Does any one have fix for this issue or is it being worked out.

        Show
        sharath2k Sharath added a comment - I have a critical issue with deserializing Java Array with 3000+ length Does any one have fix for this issue or is it being worked out.
        Hide
        molson Michael Olson added a comment -

        I am using Axis 1.4, and I consistently get this error. It is not a matter of whether or not the error occurs, only when. Also, it does not take 3 or 4 days for me; if I leave my threads running at 1 request per second, they have all died before 24 hours has elapsed. I would agree with whoever marked this bug as critical. This makes the deserialization function incredibly fragile and undependable for long term processing. Is there any chance of getting this assigned to someone?

        Show
        molson Michael Olson added a comment - I am using Axis 1.4, and I consistently get this error. It is not a matter of whether or not the error occurs, only when. Also, it does not take 3 or 4 days for me; if I leave my threads running at 1 request per second, they have all died before 24 hours has elapsed. I would agree with whoever marked this bug as critical. This makes the deserialization function incredibly fragile and undependable for long term processing. Is there any chance of getting this assigned to someone?
        Hide
        rojens Jens Rottgardt added a comment -

        Hello,

        I have followed Bjorn's advice and updated AIXS from 1.2.1 to 1.4 but the problem still exists. The out of memory occures exactly at the same line.

        java.lang.OutOfMemoryError
        at org.apache.axis.message.SAX2EventRecorder$objArrayVector.add(SAX2EventRecorder.java(Compiled Code))
        at org.apache.axis.message.SAX2EventRecorder.endElement(SAX2EventRecorder.java(Inlined Compiled Code))
        at org.apache.axis.encoding.DeserializationContext.endElement(DeserializationContext.java(Compiled Code))
        at org.apache.xerces.parsers.AbstractSAXParser.endElement(Unknown Source)
        at org.apache.xerces.impl.XMLNSDocumentScannerImpl.scanEndElement(Unknown Source)
        at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown Source)
        at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)
        at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
        at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
        at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
        at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
        at javax.xml.parsers.SAXParser.parse(Unknown Source)
        at org.apache.axis.encoding.DeserializationContext.parse(DeserializationContext.java(Compiled Code))
        at org.apache.axis.SOAPPart.getAsSOAPEnvelope(SOAPPart.java(Compiled Code))
        at org.apache.axis.Message.getSOAPEnvelope(Message.java(Inlined Compiled Code))
        at org.apache.axis.handlers.soap.MustUnderstandChecker.invoke(MustUnderstandChecker.java(Compiled Code))
        at org.apache.axis.client.AxisClient.invoke(AxisClient.java(Compiled Code))
        at org.apache.axis.client.Call.invokeEngine(Call.java(Compiled Code))
        at org.apache.axis.client.Call.invoke(Call.java(Compiled Code))
        at org.apache.axis.client.Call.invoke(Call.java(Compiled Code))
        at org.apache.axis.client.Call.invoke(Call.java(Compiled Code))
        at org.apache.axis.client.Call.invoke(Call.java(Compiled Code))
        at com.vwag.fh.common.fsp.AbstractConnector.genericGetAllData(AbstractConnector.java(Inlined Compiled Code))
        at com.vwag.fh.common.fsp.HDTServiceImpl.getAllHDTData(HDTServiceImpl.java(Inlined Compiled Code))
        at fh.sysmon.adapter.service.fsp.implementation.HDTConnectorDelegator.getAvailableItemMap(HDTConnectorDelegator.java(Compiled Code))
        at fh.sysmon.adapter.service.fsp.implementation.HDTConnectorDelegator.getAvailableMbeans(HDTConnectorDelegator.java(Compiled Code))
        at fh.sysmon.adapter.service.fsp.implementation.FSPResourceInfoBulkUpdateJob.updateContent(FSPResourceInfoBulkUpdateJob.java(Compiled Code))
        at fh.sysmon.adapter.service.fsp.implementation.FSPResourceInfoBulkUpdateJob.execute(FSPResourceInfoBulkUpdateJob.java(Compiled Code))
        at fh.sysmon.common.scheduler.DummyScheduler.alarm(DummyScheduler.java(Compiled Code))
        at fh.evo.common.timer.Alarm.run(Alarm.java(Compiled Code))
        at java.lang.Thread.run(Thread.java:570)

        ... Jens

        Show
        rojens Jens Rottgardt added a comment - Hello, I have followed Bjorn's advice and updated AIXS from 1.2.1 to 1.4 but the problem still exists. The out of memory occures exactly at the same line. java.lang.OutOfMemoryError at org.apache.axis.message.SAX2EventRecorder$objArrayVector.add(SAX2EventRecorder.java(Compiled Code)) at org.apache.axis.message.SAX2EventRecorder.endElement(SAX2EventRecorder.java(Inlined Compiled Code)) at org.apache.axis.encoding.DeserializationContext.endElement(DeserializationContext.java(Compiled Code)) at org.apache.xerces.parsers.AbstractSAXParser.endElement(Unknown Source) at org.apache.xerces.impl.XMLNSDocumentScannerImpl.scanEndElement(Unknown Source) at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown Source) at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source) at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source) at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source) at org.apache.xerces.parsers.XMLParser.parse(Unknown Source) at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source) at javax.xml.parsers.SAXParser.parse(Unknown Source) at org.apache.axis.encoding.DeserializationContext.parse(DeserializationContext.java(Compiled Code)) at org.apache.axis.SOAPPart.getAsSOAPEnvelope(SOAPPart.java(Compiled Code)) at org.apache.axis.Message.getSOAPEnvelope(Message.java(Inlined Compiled Code)) at org.apache.axis.handlers.soap.MustUnderstandChecker.invoke(MustUnderstandChecker.java(Compiled Code)) at org.apache.axis.client.AxisClient.invoke(AxisClient.java(Compiled Code)) at org.apache.axis.client.Call.invokeEngine(Call.java(Compiled Code)) at org.apache.axis.client.Call.invoke(Call.java(Compiled Code)) at org.apache.axis.client.Call.invoke(Call.java(Compiled Code)) at org.apache.axis.client.Call.invoke(Call.java(Compiled Code)) at org.apache.axis.client.Call.invoke(Call.java(Compiled Code)) at com.vwag.fh.common.fsp.AbstractConnector.genericGetAllData(AbstractConnector.java(Inlined Compiled Code)) at com.vwag.fh.common.fsp.HDTServiceImpl.getAllHDTData(HDTServiceImpl.java(Inlined Compiled Code)) at fh.sysmon.adapter.service.fsp.implementation.HDTConnectorDelegator.getAvailableItemMap(HDTConnectorDelegator.java(Compiled Code)) at fh.sysmon.adapter.service.fsp.implementation.HDTConnectorDelegator.getAvailableMbeans(HDTConnectorDelegator.java(Compiled Code)) at fh.sysmon.adapter.service.fsp.implementation.FSPResourceInfoBulkUpdateJob.updateContent(FSPResourceInfoBulkUpdateJob.java(Compiled Code)) at fh.sysmon.adapter.service.fsp.implementation.FSPResourceInfoBulkUpdateJob.execute(FSPResourceInfoBulkUpdateJob.java(Compiled Code)) at fh.sysmon.common.scheduler.DummyScheduler.alarm(DummyScheduler.java(Compiled Code)) at fh.evo.common.timer.Alarm.run(Alarm.java(Compiled Code)) at java.lang.Thread.run(Thread.java:570) ... Jens
        Hide
        bjorn Bjorn Townsend added a comment -

        Have you considered trying the latest version of Axis, verison 1.4? Many memory leaks have been fixed since 1.2.

        Show
        bjorn Bjorn Townsend added a comment - Have you considered trying the latest version of Axis, verison 1.4? Many memory leaks have been fixed since 1.2.

          People

          • Assignee:
            Unassigned
            Reporter:
            rojens Jens Rottgardt
          • Votes:
            3 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:

              Development