Uploaded image for project: 'Derby'
  1. Derby
  2. DERBY-7075

OutOfMemory Exceptions in derby server process

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 10.8.3.0, 10.14.2.0
    • Fix Version/s: None
    • Component/s: Network Server
    • Labels:
      None
    • Urgency:
      Urgent

      Description

      We have two runtime environments v7.5.4 and v7.5.3 similar in terms of OS memory, java version and java launch command settings for the derby and other java processes required for our application in which similar projects/workloads are running.  But in one of the environments we are seeing out of memory (OOM) exceptions in the derby server java process.  In this environment where OOM is occurring, we even refreshed Apache derby to the latest version available for Java 8 and still encountered OOM exceptions.  Please help resolve this OOM issue. 

       

      Exception in thread "DRDAConnThread_39" Exception in thread "DRDAConnThread_41" java/lang/OutOfMemoryError: Java heap space
      at java/nio/CharBuffer.wrap (CharBuffer.java:373)
      at sun/nio/cs/StreamEncoder.implWrite (StreamEncoder.java:277)
      at sun/nio/cs/StreamEncoder.write (StreamEncoder.java:137)
      at java/io/OutputStreamWriter.write (OutputStreamWriter.java:244)
      at java/io/BufferedWriter.flushBuffer (BufferedWriter.java:140)
      at java/io/PrintStream.write (PrintStream.java:550)
      at java/io/PrintStream.print (PrintStream.java:693)
      at java/io/PrintStream.append (PrintStream.java:1089)
      at java/io/PrintStream.append (PrintStream.java:70)
      at com/ibm/oti/util/Util.appendTo (Util.java:225)
      at java/lang/Throwable.printStackTrace (Throwable.java:506)
      at java/lang/Throwable.printStackTrace (Throwable.java:302)
      at java/lang/Throwable.printStackTrace (Throwable.java:247)
      Exception in thread "DRDAConnThread_43" java.lang.OutOfMemoryErrorjava.lang.OutOfMemoryError(?)java.lang.OutOfMemoryError: Java heap space
      at java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:872)
      at java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:8java/lang/OutOfMemoryError: Java heap space
      at java/lang/ThreadGroup.uncaughtException (ThreadGroup.java:872)
      at java/lang/ThreadGroup.uncaughtException (ThreadGroup.java:866)
      at java/lang/ThreadGroup.uncaughtException (ThreadGroup.java:866)
      at java/lang/Thread.uncaughtException (Thread.java:1332)

      -----------------------

      :v7.5.4
      OS: RHELv8 with 4GB RAM
      Java: IBM Java SDK v8.0.6.0
      Apache Derby v10.8.3.3, Apache Derby v10.14.2.0

      [bceglc385/Standalone ~]# /usr/java/default/jre/bin/java -version
      java version "1.8.0_231"
      Java(TM) SE Runtime Environment (build 8.0.6.0 - pxa6480sr6-20191107_01(SR6))
      IBM J9 VM (build 2.9, JRE 1.8.0 Linux amd64-64-Bit Compressed References 20191106_432135 (JIT enabled, AOT enabled)
      OpenJ9   - f0b6be7
      OMR      - 18d8f94
      IBM      - 233dfb5)
      JCL - 20191016_01 based on Oracle jdk8u231-b10

       

      /usr/java/default/bin/java -server -Xdump:heap+system+snap:none -Xdump:heap:events=user,file=/var/log/javaheapdump-user-derby.ph  -Xdump:heap:events=systhrow,filter=java/lang/OutOfMemoryError,file=/var/log/javaheapdump-oom-derby.phd,range=1..1 -Xdump!https://emoji.slack-edge.com/T27SFGS2W/java/c0a6f7d2f3faab62.png!defaults:file=/var/log/javacore-derby.txt -Xmx512m -Xms512m -Xmn192m -agentlib:CastIron -Xbootclasspath/p:/usr/ironhide/maestro/bundles/com.approuter.management.jmxremote-1.0.0.jar: -Xgcpolicy:gencon -Xverbosegclog:/var/log/derbyserver_gc.log,10,1000 -XX:+HeapManagementMXBeanCompatibility -Dorg.apache.commons.logging.Log=org.apache.commons.logging.impl.Jdk14Logger -Dderby.system.home=/usr/ironhide/var/db/staging -Dcom.sun.management.jmxremote -Dcom.approuter.management.jmxremote.connectorserver.port=31997 -Djmx.remote.protocol.provider.pkgs=com.approuter.management.jmxremote -Dcom.sun.management.jmxremote.port=1997 -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=true -Dcom.sun.management.jmxremote.password.file=/usr/ironhide/maestro/management/jmxremote.password -Dcom.sun.management.jmxremote.access.file=/usr/ironhide/maestro/management/jmxremote.access -Dcom.approuter.logging.filename=derbyserver.log -Djava.rmi.server.hostname=9.121.242.177 -Djava.net.preferIPv4Stack=true -Dcom.approuter.sysconf.appliance=true -Dcom.mchange.v2.log.MLog=com.mchange.v2.log.jdk14logging.Jdk14MLog -Dcom.mchange.v2.resourcepool.experimental.useScatteredAcquireTask=true -Dc3p0.checkoutTimeout=30000 -Dc3p0.maxIdleTimeExcessConnections=30 -Dc3p0.maxStatementsPerConnection=100 -Dc3p0.testConnectionOnCheckin=true -Dc3p0.testConnectionOnCheckout=true -Dc3p0.numHelperThreads=10 -Dosgi.clean=true -Dosgi.configuration.area=/usr/ironhide/maestro/bundles/derbyserver -Dorg.osgi.framework.bootdelegation=* -Declipse.application.noDefault=true -Declipse.consoleLog=true -Dcom.approuter.orcmon.purger.compressionmode=sequential -DIH_ROOT=/usr/ironhide -Djava.util.logging.config.file=/usr/ironhide/maestro/etc/logging.properties -jar /usr/ironhide/maestro/bundles/org.eclipse.osgi_3.10.1.v20140909-1633.jar

       

      [bceglc390/Standalone ~]# cat /proc/meminfo
      MemTotal:        4057920 kB
      MemFree:          152896 kB
      Buffers:          342624 kB
      Cached:           729112 kB
      SwapCached:         3428 kB
      Active:          2644068 kB
      Inactive:        1066404 kB
      Active(anon):    1951704 kB
      Inactive(anon):   712696 kB
      Active(file):     692364 kB
      Inactive(file):   353708 kB
      Unevictable:          20 kB
      Mlocked:              20 kB
      SwapTotal:       3999736 kB
      SwapFree:        3988760 kB
      Dirty:               720 kB
      Writeback:             0 kB
      AnonPages:       2636560 kB
      Mapped:            38508 kB
      Shmem:             25648 kB
      Slab:             129600 kB
      SReclaimable:      48320 kB
      SUnreclaim:        81280 kB
      KernelStack:        6568 kB
      PageTables:        11116 kB
      NFS_Unstable:          0 kB
      Bounce:                0 kB
      WritebackTmp:          0 kB
      CommitLimit:     6028696 kB
      Committed_AS:    1658244 kB
      VmallocTotal:   34359738367 kB
      VmallocUsed:      152396 kB
      VmallocChunk:   34359582800 kB
      AnonHugePages:   2433024 kB
      HugePages_Total:       0
      HugePages_Free:        0
      HugePages_Rsvd:        0
      HugePages_Surp:        0
      Hugepagesize:       2048 kB
      DirectMap4k:        4096 kB
      DirectMap2M:     4190208 kB

       

      ----------
      v7.5.3
      OS: Bedrock v5 which is custom OS based on RHELv7 with 4GB RAM
      Java: IBM Java SDK v8.0.6.0
      Apache Derby v10.8.3.3

       

      [bceglc390/Standalone ~]# /usr/java/default/jre/bin/java -version
      java version "1.8.0_231"
      Java(TM) SE Runtime Environment (build 8.0.6.0 - pxa6480sr6-20191107_01(SR6))
      IBM J9 VM (build 2.9, JRE 1.8.0 Linux amd64-64-Bit Compressed References 20191106_432135 (JIT enabled, AOT enabled)
      OpenJ9   - f0b6be7
      OMR      - 18d8f94
      IBM      - 233dfb5)
      JCL - 20191016_01 based on Oracle jdk8u231-b10

       

      /usr/java/default/bin/java -server -Xdump:heap+system+snap:none -Xdump:heap:events=user,file=/var/log/javaheapdump-user-derby.phd -Xdump:heap:events=systhrow,filter=java/lang/OutOfMemoryError,file=/var/log/javaheapdump-oom-derby.phd,range=1..1 -Xdump!https://emoji.slack-edge.com/T27SFGS2W/java/c0a6f7d2f3faab62.png!defaults:file=/var/log/javacore-derby.txt -Xmx512m -Xms512m -Xmn192m -agentlib:CastIron -Xbootclasspath/p:/usr/ironhide/maestro/bundles/com.approuter.management.jmxremote-1.0.0.jar: -Xgcpolicy:gencon -Xverbosegclog:/var/log/derbyserver_gc.log,10,1000 -XX:+HeapManagementMXBeanCompatibility -Dorg.apache.commons.logging.Log=org.apache.commons.logging.impl.Jdk14Logger -Dderby.system.home=/usr/ironhide/var/db/staging -Dcom.sun.management.jmxremote -Dcom.approuter.management.jmxremote.connectorserver.port=31997 -Djmx.remote.protocol.provider.pkgs=com.approuter.management.jmxremote -Dcom.sun.management.jmxremote.port=1997 -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=true -Dcom.sun.management.jmxremote.password.file=/usr/ironhide/maestro/management/jmxremote.password -Dcom.sun.management.jmxremote.access.file=/usr/ironhide/maestro/management/jmxremote.access -Dcom.approuter.logging.filename=derbyserver.log -Djava.rmi.server.hostname=9.124.112.240 -Djava.net.preferIPv4Stack=true -Dcom.approuter.sysconf.appliance=true -Dcom.mchange.v2.log.MLog=com.mchange.v2.log.jdk14logging.Jdk14MLog -Dcom.mchange.v2.resourcepool.experimental.useScatteredAcquireTask=true -Dc3p0.checkoutTimeout=30000 -Dc3p0.maxIdleTimeExcessConnections=30 -Dc3p0.maxStatementsPerConnection=100 -Dc3p0.testConnectionOnCheckin=true -Dc3p0.testConnectionOnCheckout=true -Dc3p0.numHelperThreads=10 -Dosgi.clean=true -Dosgi.configuration.area=/usr/ironhide/maestro/bundles/derbyserver -Dorg.osgi.framework.bootdelegation=* -Declipse.application.noDefault=true -Declipse.consoleLog=true -Dcom.approuter.orcmon.purger.compressionmode=sequential -DIH_ROOT=/usr/ironhide -Djava.util.logging.config.file=/usr/ironhide/maestro/etc/logging.properties -jar /usr/ironhide/maestro/bundles/org.eclipse.osgi_3.10.1.v20140909-1633.jar

       

      [cldegdh9/Standalone log]# cat /proc/meminfo
      MemTotal:        3872868 kB
      MemFree:          343700 kB
      MemAvailable:     374676 kB
      Buffers:           33100 kB
      Cached:           170624 kB
      SwapCached:         3848 kB
      Active:          2540784 kB
      Inactive:         710092 kB
      Active(anon):    2429812 kB
      Inactive(anon):   621324 kB
      Active(file):     110972 kB
      Inactive(file):    88768 kB
      Unevictable:          20 kB
      Mlocked:              20 kB
      SwapTotal:       5345272 kB
      SwapFree:        5280760 kB
      Dirty:              1880 kB
      Writeback:             0 kB
      AnonPages:       3044312 kB
      Mapped:            33676 kB
      Shmem:              3984 kB
      Slab:             119980 kB
      SReclaimable:      71120 kB
      SUnreclaim:        48860 kB
      KernelStack:       15116 kB
      PageTables:        13936 kB
      NFS_Unstable:          0 kB
      Bounce:                0 kB
      WritebackTmp:          0 kB
      CommitLimit:     7281704 kB
      Committed_AS:    1755044 kB
      VmallocTotal:   34359738367 kB
      VmallocUsed:           0 kB
      VmallocChunk:          0 kB
      HardwareCorrupted:     0 kB
      AnonHugePages:   2885632 kB
      ShmemHugePages:        0 kB
      ShmemPmdMapped:        0 kB
      HugePages_Total:       0
      HugePages_Free:        0
      HugePages_Rsvd:        0
      HugePages_Surp:        0
      Hugepagesize:       2048 kB
      Hugetlb:               0 kB
      DirectMap4k:      126912 kB
      DirectMap2M:     4067328 kB

        Attachments

        1. image-101420-2.png
          196 kB
          Vamsavardhana Reddy
        2. image-101420-3.png
          198 kB
          Vamsavardhana Reddy
        3. image-101420-1.png
          202 kB
          Vamsavardhana Reddy
        4. image-10833-2.png
          138 kB
          Vamsavardhana Reddy
        5. image-10833-1.png
          93 kB
          Vamsavardhana Reddy
        6. javaheapdump-oom-derby-101420.zip
          14.66 MB
          Vamsavardhana Reddy
        7. javaheapdump-oom-derby-v10833.zip
          12.93 MB
          Vamsavardhana Reddy

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              vamsic Vamsavardhana Reddy
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: