Geronimo
  1. Geronimo
  2. GERONIMO-3930

IllegalArgumentException reading Transaction Log

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 2.0.x, 2.1, 2.1.1, 2.2
    • Fix Version/s: 2.0.3, 2.1.2, 2.2
    • Component/s: None
    • Security Level: public (Regular issues)
    • Labels:
      None

      Description

      Beniamin has reported the following problem on user@.

      After processing 20k request to my webservice whose are translated to ~120k
      XA transactions (postgres + jms) Geronimo hangs up and does not respond on
      requests via HTTP, request to JMS engine (from HermesJMS) and ignores tries
      to shutdown server.

      I stopped Geronimo with kill -9 and tried to start it again and got
      exception:

      Module 11/69 org.apache.geronimo.configs/activemq-ra/2.1-SNAPSHOT/car
      10:22:15,325 ERROR [GBeanInstanceState] Error while starting; GBean is now in the FAILED state:
      abstractName="org.apache.geronimo.configs/transaction/2.1-SNAPSHOT/car?ServiceModule=org.apache.geronimo.configs/transaction/2.1-SNAPSHOT/car,j2eeType=TransactionLog,name=HOWLTransactionLog"
      java.lang.IllegalArgumentException: Negative position
      at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:613)
      at org.objectweb.howl.log.BlockLogBuffer.read(BlockLogBuffer.java:412)
      at org.objectweb.howl.log.LogFileManager.read(LogFileManager.java:641)
      at org.objectweb.howl.log.LogBufferManager.replay(LogBufferManager.java:869)
      at org.objectweb.howl.log.Logger.replay(Logger.java:396)
      at org.objectweb.howl.log.xa.XALogger.open(XALogger.java:897)
      at org.apache.geronimo.transaction.log.HOWLLog.doStart(HOWLLog.java:224)
      at org.apache.geronimo.gbean.runtime.GBeanInstance.createInstance(GBeanInstance.java:996)
      at org.apache.geronimo.gbean.runtime.GBeanInstanceState.attemptFullStart(GBeanInstanceState.java:268)
      at org.apache.geronimo.gbean.runtime.GBeanInstanceState.start(GBeanInstanceState.java:102)
      at org.apache.geronimo.gbean.runtime.GBeanInstance.start(GBeanInstance.java:539)
      at org.apache.geronimo.gbean.runtime.GBeanDependency.attemptFullStart(GBeanDependency.java:111)
      at org.apache.geronimo.gbean.runtime.GBeanDependency.addTarget(GBeanDependency.java:146)
      at org.apache.geronimo.gbean.runtime.GBeanDependency$1.running(GBeanDependency.java:120)
      at org.apache.geronimo.kernel.basic.BasicLifecycleMonitor.fireRunningEvent(BasicLifecycleMonitor.java:176)
      at org.apache.geronimo.kernel.basic.BasicLifecycleMonitor.access$300(BasicLifecycleMonitor.java:44)
      at org.apache.geronimo.kernel.basic.BasicLifecycleMonitor$RawLifecycleBroadcaster.fireRunningEvent(BasicLifecycleMonitor.java:254)
      at org.apache.geronimo.gbean.runtime.GBeanInstanceState.attemptFullStart(GBeanInstanceState.java:294)
      at org.apache.geronimo.gbean.runtime.GBeanInstanceState.start(GBeanInstanceState.java:102)
      at org.apache.geronimo.gbean.runtime.GBeanInstanceState.startRecursive(GBeanInstanceState.java:124)
      at org.apache.geronimo.gbean.runtime.GBeanInstance.startRecursive(GBeanInstance.java:553)
      at org.apache.geronimo.kernel.basic.BasicKernel.startRecursiveGBean(BasicKernel.java:379)
      at org.apache.geronimo.kernel.config.ConfigurationUtil.startConfigurationGBeans(ConfigurationUtil.java:448)
      at org.apache.geronimo.kernel.config.KernelConfigurationManager.start(KernelConfigurationManager.java:187)
      at org.apache.geronimo.kernel.config.SimpleConfigurationManager.startConfiguration(SimpleConfigurationManager.java:534)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.apache.geronimo.gbean.runtime.ReflectionMethodInvoker.invoke(ReflectionMethodInvoker.java:34)
      at org.apache.geronimo.gbean.runtime.GBeanOperation.invoke(GBeanOperation.java:124)
      at org.apache.geronimo.gbean.runtime.GBeanInstance.invoke(GBeanInstance.java:830)
      at org.apache.geronimo.gbean.runtime.RawInvoker.invoke(RawInvoker.java:57)
      at org.apache.geronimo.kernel.basic.RawOperationInvoker.invoke(RawOperationInvoker.java:35)
      at org.apache.geronimo.kernel.basic.ProxyMethodInterceptor.intercept(ProxyMethodInterceptor.java:96)
      at org.apache.geronimo.gbean.GBeanLifecycle$$EnhancerByCGLIB$$5ebee84a.startConfiguration(<generated>)
      at org.apache.geronimo.system.main.EmbeddedDaemon.doStartup(EmbeddedDaemon.java:156)
      at org.apache.geronimo.system.main.EmbeddedDaemon.execute(EmbeddedDaemon.java:78)
      at org.apache.geronimo.kernel.util.MainConfigurationBootstrapper.main(MainConfigurationBootstrapper.java:45)
      at org.apache.geronimo.cli.AbstractCLI.executeMain(AbstractCLI.java:67)
      at org.apache.geronimo.cli.daemon.DaemonCLI.main(DaemonCLI.java:30)
      Server Startup failed

        Activity

        Joe Bohn made changes -
        Fix Version/s 2.0.3 [ 12313315 ]
        Fix Version/s 2.0.x [ 12312601 ]
        Donald Woods made changes -
        Fix Version/s 2.1.x [ 12313103 ]
        Fix Version/s 2.1.2 [ 12313123 ]
        Hide
        Donald Woods added a comment -

        also merged into branches/2.0 as Rev664019.

        Show
        Donald Woods added a comment - also merged into branches/2.0 as Rev664019.
        Kevan Miller made changes -
        Resolution Fixed [ 1 ]
        Status Open [ 1 ] Closed [ 6 ]
        Assignee Kevan Miller [ kevan ]
        Hide
        Kevan Miller added a comment -

        The tran logs cannot exceed 2 gigabytes in size. We were configuring the max size to be 32K * Integer.MAX_VALUE

        32K byte buffers is pretty large. There'd be a lot of wasted space... 4K should be a reasonable value...

        The default max size is now 2 megs (4k buffers * 512).

        Show
        Kevan Miller added a comment - The tran logs cannot exceed 2 gigabytes in size. We were configuring the max size to be 32K * Integer.MAX_VALUE 32K byte buffers is pretty large. There'd be a lot of wasted space... 4K should be a reasonable value... The default max size is now 2 megs (4k buffers * 512).
        Joe Bohn made changes -
        Field Original Value New Value
        Fix Version/s 2.1.1 [ 12312941 ]
        Fix Version/s 2.1.x [ 12313103 ]
        Hide
        Joe Bohn added a comment -

        Kevan,
        It appears this issue will not be resolved for 2.1.1 so I'm moving the fix version to 2.1.x.
        Joe

        Show
        Joe Bohn added a comment - Kevan, It appears this issue will not be resolved for 2.1.1 so I'm moving the fix version to 2.1.x. Joe
        Hide
        Joe Bohn added a comment -

        Any word back on the results of setting the limit to be unbounded? If so, then can this be included in 2.1.1 as the default or are there other issues that might push this from 2.1.1 to 2.2?

        Show
        Joe Bohn added a comment - Any word back on the results of setting the limit to be unbounded? If so, then can this be included in 2.1.1 as the default or are there other issues that might push this from 2.1.1 to 2.2?
        Hide
        Kevan Miller added a comment -

        I suspect that we're allowing the transaction log file to become too large. See this HOWL bug report for a bit more information – http://forge.objectweb.org/tracker/index.php?func=detail&aid=306425&group_id=92&atid=100092

        The default HOWL configuration is defined in the following HOWLTransactionLog GBean in ~/geronimo/server/branches/2.1/plugins/connector/transaction/src/main/plan/plan.xml:

        <gbean name="HOWLTransactionLog" class="org.apache.geronimo.transaction.log.HOWLLogGBean">
        <attribute name="bufferClassName">org.objectweb.howl.log.BlockLogBuffer</attribute>
        <attribute name="bufferSizeKBytes">32</attribute>
        <attribute name="checksumEnabled">true</attribute>
        <attribute name="adler32Checksum">true</attribute>
        <attribute name="flushSleepTimeMilliseconds">50</attribute>
        <attribute name="logFileDir">var/txlog</attribute>
        <attribute name="logFileExt">log</attribute>
        <attribute name="logFileName">howl</attribute>
        <attribute name="maxBlocksPerFile">-1</attribute>
        <attribute name="maxBuffers">0</attribute>
        <attribute name="maxLogFiles">2</attribute>
        <attribute name="minBuffers">4</attribute>
        <attribute name="threadsWaitingForceThreshold">-1</attribute>
        <reference name="XidFactory">
        <name>XidFactory</name>
        </reference>
        <reference name="ServerInfo">
        <name>ServerInfo</name>
        </reference>
        </gbean>

        The maxBlocksPerFile setting is the cause of this problem, I suspect. Configuring to a more reasonable value (i.e. 65534 with a buffer size of 32k), should keep the logfile size below Integer.MAX_VALUE. Note that you can't dynamically change this setting. HOWL won't let the server start, if it detects that the current maxBlocksPerFile doesn't match the old setting in the howl log files. You'll have to delete the old howl log files.

        Adding the following to your config.xml will override the default maxBlocksPerFile setting:

        <module name="org.apache.geronimo.configs/transaction/2.1/car">
        <gbean name="HOWLTransactionLog">
        <attribute name="maxBlocksPerFile">65534</attribute>
        </gbean>
        </module>

        I'm going to wait to hear that this fixes the problem, before changing our default behavior.

        Show
        Kevan Miller added a comment - I suspect that we're allowing the transaction log file to become too large. See this HOWL bug report for a bit more information – http://forge.objectweb.org/tracker/index.php?func=detail&aid=306425&group_id=92&atid=100092 The default HOWL configuration is defined in the following HOWLTransactionLog GBean in ~/geronimo/server/branches/2.1/plugins/connector/transaction/src/main/plan/plan.xml: <gbean name="HOWLTransactionLog" class="org.apache.geronimo.transaction.log.HOWLLogGBean"> <attribute name="bufferClassName">org.objectweb.howl.log.BlockLogBuffer</attribute> <attribute name="bufferSizeKBytes">32</attribute> <attribute name="checksumEnabled">true</attribute> <attribute name="adler32Checksum">true</attribute> <attribute name="flushSleepTimeMilliseconds">50</attribute> <attribute name="logFileDir">var/txlog</attribute> <attribute name="logFileExt">log</attribute> <attribute name="logFileName">howl</attribute> <attribute name="maxBlocksPerFile">-1</attribute> <attribute name="maxBuffers">0</attribute> <attribute name="maxLogFiles">2</attribute> <attribute name="minBuffers">4</attribute> <attribute name="threadsWaitingForceThreshold">-1</attribute> <reference name="XidFactory"> <name>XidFactory</name> </reference> <reference name="ServerInfo"> <name>ServerInfo</name> </reference> </gbean> The maxBlocksPerFile setting is the cause of this problem, I suspect. Configuring to a more reasonable value (i.e. 65534 with a buffer size of 32k), should keep the logfile size below Integer.MAX_VALUE. Note that you can't dynamically change this setting. HOWL won't let the server start, if it detects that the current maxBlocksPerFile doesn't match the old setting in the howl log files. You'll have to delete the old howl log files. Adding the following to your config.xml will override the default maxBlocksPerFile setting: <module name="org.apache.geronimo.configs/transaction/2.1/car"> <gbean name="HOWLTransactionLog"> <attribute name="maxBlocksPerFile">65534</attribute> </gbean> </module> I'm going to wait to hear that this fixes the problem, before changing our default behavior.
        Kevan Miller created issue -

          People

          • Assignee:
            Kevan Miller
            Reporter:
            Kevan Miller
          • Votes:
            1 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development