[DERBY-4173] java/lang/OutOfMemoryError" "native memory exhausted" received on z/OS for process launched to shutdown network server and launch the replication run in Suites.All

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Cannot Reproduce
• Affects Version/s: 10.5.1.1
• Component/s: Network Server

Environment

java version "1.6.0"
Java(TM) SE Runtime Environment (build pmz3160sr2ifix-20081021_01(SR2+IZ32776+IZ33456))
IBM J9 VM (build 2.4, J2RE 1.6.0 IBM J9 2.4 z/OS s390-31 jvmmz3160ifx-20081010_24288 (JIT enabled, AOT enabled)
J9VM - 20081009_024288_bHdSMr
JIT - r9_20080721_1330ifx2
GC - 20080724_AA)
JCL - 20080808_02

10.5.1.1 RC2

• Bug behavior facts: Crash
• Assignee: Katherine Marsden
• Reporter: Katherine Marsden
• Created: 21/Apr/09 21:44
• Updated: 18/May/09 14:48
• Resolved: 06/May/09 22:35

Description

With pmz3160sr2ifix-20081021_01(SR2+IZ32776+IZ33456 there were 6 javacore files produced during the test run with jvm crashes shutting down network server.

1TISIGINFO Dump Event "systhrow" (00040000) Detail "java/lang/OutOfMemoryError" "native memory exhausted" received
1TIDATETIME Date: 2009/04/16 at 23:04:20

The command line in two javacore was starting the replication run.

1CICMDLINE /u/vanlm/pmz3160sr2ifx-20081021_01/J6.0/bin/java -Dderby.tests.trace=true -Dtest.serverHost=localhost -Dtest.serverPort=1527 -Dtest.inserts=10000 -Dtest.commitFreq=1000 -classpath .:/u/vanlm/10.5/jars/derbyrun.jar:/u/vanlm/10.5/jars/junit.jar:/u/vanlm/10.5/jars/derbyTesting.jar:/u/vanlm/10.5/jars/jakarta-oro-2.0.8.jar junit.textui.TestRunner org.apache.derbyTesting.functionTests.tests.replicationTests.ReplicationTestRun

I never knew the replication run was launched in another process like this.

The others were just simple server shutdowns:
1CICMDLINE /u/vanlm/pmz3160sr2ifx-20081021_01/J6.0/bin/java -Dderby.infolog.append=true -cp .:/u/vanlm/10.5/jars/derbyrun.jar:/u/vanlm/10.5/jars/junit.jar:/u/vanlm/10.5/jars/derbyTesting.jar:/u/vanlm/10.5/jars/jakarta-oro-2.0.8.jar org.apache.derby.drda.NetworkServerControl shutdown -h localhost -p 4527

It is not clear to me how a simple shutdown could run out of native memory, since it only simply starts java and writes a few bytes to a socket to shutdown the server.

Then 40 minutes later the system shutdown with complaints that it was out of AUX space
09106 23:40:09.87 STC20757 00000010 ACTR001I VANLM9 STEP1 BPXPRFC 0001
09106 23:40:11.79 STC20746 00000201 IEA794I SVC DUMP HAS CAPTURED: 379
379 00000201 DUMPID=004 REQUESTED BY JOB (VANLM )
379 00000201 DUMP TITLE=COMPON=ISG,COMPID=SCSDS,ISSUER=ISGRREC ,MODULE=ISGGR
379 00000201 T +????,ABEND=S0738,REASON=023E0004
09106 23:40:11.83 00000201 IEF196I IGD100I 2030 ALLOCATED TO DDNAME SYS00007 DATACLAS ( )
09106 23:40:30.76 00000010 IRA205I 50% AUXILIARY STORAGE ALLOCATED
09106 23:40:36.76 00000010 *IRA200E AUXILIARY STORAGE SHORTAGE

It appears the Aux space issue occurred because there was not sufficient space to write the dumps from the prior events, but it is not clear why it didn't happen for 40 minutes.

The crashes did not cause any tests to fail and the run completed reporting only:
There was 1 failure:
1) testMkdirsInvalidAbsolute(org.apache.derbyTesting.unitTests.junit.VirtualFileTest)junit.framework.AssertionFailedErro
r
at org.apache.derbyTesting.unitTests.junit.VirtualFileTest.testMkdirsInvalidAbsolute(VirtualFileTest.java:94)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:45)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
at org.apache.derbyTesting.junit.BaseTestCase.runBare(BaseTestCase.java:105)

FAILURES!!!
Tests run: 10939, Failures: 1, Errors: 0

$
I did not notice this happening with the 64bit jvm when I ran against 10.5.1.0 RC1 but there is a chance that it occurred and I didn't notice beaause it does not actually cause tests to fail for some reason.