Type:
Bug
Status:
Closed
Priority:
Critical
Resolution:
Fixed
Affects Version/s:
None
Component/s:
None
OOME in IPC server handler causes the IPC handler to abort, but the client never learns about this, so it waits and waits and waits... I have seen Heritrix writer threads that have been waiting for 7+ hours. And, the OOME does not take down the HRS, so it stays up in some degraded state. E.g.:
java.lang.OutOfMemoryError: Java heap space
Dumping heap to java_pid13008.hprof
Exception in thread "IPC Server handler 5 on 60020" java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2786)
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
at java.io.DataOutputStream.write(DataOutputStream.java:90)
at org.apache.hadoop.hbase.util.Bytes.writeByteArray(Bytes.java:82)
at org.apache.hadoop.hbase.io.Cell.write(Cell.java:162)
at org.apache.hadoop.hbase.io.HbaseMapWritable.write(HbaseMapWritable.java:200)
at org.apache.hadoop.hbase.io.RowResult.write(RowResult.java:249)
at org.apache.hadoop.hbase.io.HbaseObjectWritable.writeObject(HbaseObjectWritable.java:300)
at org.apache.hadoop.hbase.io.HbaseObjectWritable.write(HbaseObjectWritable.java:262)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:917)
Exception in thread "IPC Server handler 7 on 60020" java.lang.OutOfMemoryError: Java heap space
Exception in thread "IPC Server handler 4 on 60020" java.lang.OutOfMemoryError: Java heap space
Exception in thread "IPC Server handler 2 on 60020" java.lang.OutOfMemoryError: Java heap space
Exception in thread "IPC Server handler 3 on 60020" java.lang.OutOfMemoryError: Java heap space
Exception in thread "IPC Server handler 0 on 60020" java.lang.OutOfMemoryError: Java heap space
Exception in thread "IPC Server handler 6 on 60020" java.lang.OutOfMemoryError: Java heap space
Exception in thread "IPC Server handler 9 on 60020" java.lang.OutOfMemoryError: Java heap space
Exception in thread "IPC Server handler 1 on 60020" java.lang.OutOfMemoryError: Java heap space
Exception in thread "IPC Server handler 8 on 60020" java.lang.OutOfMemoryError: Java heap space
There are no Sub-Tasks for this issue.
{"report":{"fcp":4474.600000023842,"ttfb":634.3000000119209,"pageVisibility":"visible","entityId":12414602,"key":"jira.project.issue.view-issue","isInitial":true,"threshold":1000,"elementTimings":{},"userDeviceMemory":8,"userDeviceProcessors":16,"apdex":0,"journeyId":"c32e596a-1562-42ca-bb16-025a42132b3e","navigationType":0,"readyForUser":4733.100000023842,"redirectCount":0,"resourceLoadedEnd":3149.300000011921,"resourceLoadedStart":639.7000000476837,"resourceTiming":[{"duration":426.2999999523163,"initiatorType":"link","name":"https://issues.apache.org/jira/s/b62489a2eaac59d9b8a093c1a51d034f-CDN/xd97tr/820010/13pdxe5/49fa3aa3d35a2cc689cbf274e66cc41a/_/download/contextbatch/css/_super/batch.css","startTime":639.7000000476837,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":639.7000000476837,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":1066,"responseStart":0,"secureConnectionStart":0},{"duration":427.9000000357628,"initiatorType":"link","name":"https://issues.apache.org/jira/s/56490edcf9d54e35149505f78cca6a47-CDN/xd97tr/820010/13pdxe5/72cb823bcc50211a60c1ebe830467cae/_/download/contextbatch/css/jira.browse.project,jira.view.issue,project.issue.navigator,atl.general,atl.global,jira.global,jira.general,-_super/batch.css?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&richediton=true&slack-enabled=true","startTime":640,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":640,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":1067.9000000357628,"responseStart":0,"secureConnectionStart":0},{"duration":1937.199999988079,"initiatorType":"script","name":"https://issues.apache.org/jira/s/5263129088916436ab9aeb2417075b3f-CDN/xd97tr/820010/13pdxe5/49fa3aa3d35a2cc689cbf274e66cc41a/_/download/contextbatch/js/_super/batch.js?locale=en-UK","startTime":640.1000000238419,"connectEnd":1862.6000000238419,"connectStart":1538,"domainLookupEnd":1538,"domainLookupStart":1538,"fetchStart":640.1000000238419,"redirectEnd":0,"redirectStart":0,"requestStart":1863.6000000238419,"responseEnd":2577.300000011921,"responseStart":2000.9000000357628,"secureConnectionStart":1697.800000011921},{"duration":2509,"initiatorType":"script","name":"https://issues.apache.org/jira/s/611c208bd094adb71a6f4f3e7f6fff3d-CDN/xd97tr/820010/13pdxe5/72cb823bcc50211a60c1ebe830467cae/_/download/contextbatch/js/jira.browse.project,jira.view.issue,project.issue.navigator,atl.general,atl.global,jira.global,jira.general,-_super/batch.js?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&locale=en-UK&richediton=true&slack-enabled=true","startTime":640.3000000119209,"connectEnd":1863.300000011921,"connectStart":1538,"domainLookupEnd":1538,"domainLookupStart":1538,"fetchStart":640.3000000119209,"redirectEnd":0,"redirectStart":0,"requestStart":1863.7000000476837,"responseEnd":3149.300000011921,"responseStart":1985.2000000476837,"secureConnectionStart":1698.6000000238419},{"duration":1058.5,"initiatorType":"script","name":"https://issues.apache.org/jira/s/d41d8cd98f00b204e9800998ecf8427e-CDN/xd97tr/820010/13pdxe5/1.0/_/download/batch/jira.webresources:calendar-en/jira.webresources:calendar-en.js","startTime":640.6000000238419,"connectEnd":640.6000000238419,"connectStart":640.6000000238419,"domainLookupEnd":640.6000000238419,"domainLookupStart":640.6000000238419,"fetchStart":640.6000000238419,"redirectEnd":0,"redirectStart":0,"requestStart":1538.300000011921,"responseEnd":1699.1000000238419,"responseStart":1698,"secureConnectionStart":640.6000000238419},{"duration":1222.699999988079,"initiatorType":"script","name":"https://issues.apache.org/jira/s/d41d8cd98f00b204e9800998ecf8427e-CDN/xd97tr/820010/13pdxe5/1.0/_/download/batch/jira.webresources:calendar-localisation-moment/jira.webresources:calendar-localisation-moment.js","startTime":640.7000000476837,"connectEnd":640.7000000476837,"connectStart":640.7000000476837,"domainLookupEnd":640.7000000476837,"domainLookupStart":640.7000000476837,"fetchStart":640.7000000476837,"redirectEnd":0,"redirectStart":0,"requestStart":1699.2000000476837,"responseEnd":1863.4000000357628,"responseStart":1862.6000000238419,"secureConnectionStart":640.7000000476837},{"duration":431.9000000357628,"initiatorType":"link","name":"https://issues.apache.org/jira/s/981f587853769311cda7c3b845131a06-CDN/xd97tr/820010/13pdxe5/cb5a5495a038c0744457f25821ba9ee8/_/download/contextbatch/css/jira.global.look-and-feel,-_super/batch.css","startTime":640.8000000119209,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":640.8000000119209,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":1072.7000000476837,"responseStart":0,"secureConnectionStart":0},{"duration":1361.4000000357628,"initiatorType":"script","name":"https://issues.apache.org/jira/rest/api/1.0/shortcuts/820010/5840efff50357da9055d4714dc0713f/shortcuts.js?context=issuenavigation&context=issueaction","startTime":641,"connectEnd":1862.300000011921,"connectStart":1537.5,"domainLookupEnd":1537.5,"domainLookupStart":1537.5,"fetchStart":641,"redirectEnd":0,"redirectStart":0,"requestStart":1863.6000000238419,"responseEnd":2002.4000000357628,"responseStart":2001.5,"secureConnectionStart":1697.5},{"duration":402.89999997615814,"initiatorType":"link","name":"https://issues.apache.org/jira/s/3ac36323ba5e4eb0af2aa7ac7211b4bb-CDN/xd97tr/820010/13pdxe5/efa42a25652b26dfd802540c024826b3/_/download/contextbatch/css/com.atlassian.jira.projects.sidebar.init,-_super,-jira.view.issue,-project.issue.navigator/batch.css?jira.create.linked.issue=true&richediton=true","startTime":672.4000000357628,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":672.4000000357628,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":1075.300000011921,"responseStart":0,"secureConnectionStart":0},{"duration":538.3999999761581,"initiatorType":"script","name":"https://issues.apache.org/jira/s/efa8931cd5ac13ed95c56ca8a1dc1967-CDN/xd97tr/820010/13pdxe5/efa42a25652b26dfd802540c024826b3/_/download/contextbatch/js/com.atlassian.jira.projects.sidebar.init,-_super,-jira.view.issue,-project.issue.navigator/batch.js?jira.create.linked.issue=true&locale=en-UK&richediton=true","startTime":672.6000000238419,"connectEnd":672.6000000238419,"connectStart":672.6000000238419,"domainLookupEnd":672.6000000238419,"domainLookupStart":672.6000000238419,"fetchStart":672.6000000238419,"redirectEnd":0,"redirectStart":0,"requestStart":1080.1000000238419,"responseEnd":1211,"responseStart":1210.4000000357628,"secureConnectionStart":672.6000000238419},{"duration":1388.6000000238419,"initiatorType":"script","name":"https://issues.apache.org/jira/s/d41d8cd98f00b204e9800998ecf8427e-CDN/xd97tr/820010/13pdxe5/1.0/_/download/batch/jira.webresources:bigpipe-js/jira.webresources:bigpipe-js.js","startTime":746.5,"connectEnd":746.5,"connectStart":746.5,"domainLookupEnd":746.5,"domainLookupStart":746.5,"fetchStart":746.5,"redirectEnd":0,"redirectStart":0,"requestStart":2021.800000011921,"responseEnd":2135.100000023842,"responseStart":2134.5,"secureConnectionStart":746.5},{"duration":1423.300000011921,"initiatorType":"script","name":"https://issues.apache.org/jira/s/d41d8cd98f00b204e9800998ecf8427e-CDN/xd97tr/820010/13pdxe5/1.0/_/download/batch/jira.webresources:bigpipe-init/jira.webresources:bigpipe-init.js","startTime":746.6000000238419,"connectEnd":746.6000000238419,"connectStart":746.6000000238419,"domainLookupEnd":746.6000000238419,"domainLookupStart":746.6000000238419,"fetchStart":746.6000000238419,"redirectEnd":0,"redirectStart":0,"requestStart":2002.7000000476837,"responseEnd":2169.900000035763,"responseStart":2169.400000035763,"secureConnectionStart":746.6000000238419},{"duration":777.8999999761581,"initiatorType":"xmlhttprequest","name":"https://issues.apache.org/jira/rest/webResources/1.0/resources","startTime":2722.400000035763,"connectEnd":2722.400000035763,"connectStart":2722.400000035763,"domainLookupEnd":2722.400000035763,"domainLookupStart":2722.400000035763,"fetchStart":2722.400000035763,"redirectEnd":0,"redirectStart":0,"requestStart":3325.600000023842,"responseEnd":3500.300000011921,"responseStart":3499.7000000476837,"secureConnectionStart":2722.400000035763}],"fetchStart":0,"domainLookupStart":33,"domainLookupEnd":74,"connectStart":74,"connectEnd":353,"secureConnectionStart":239,"requestStart":353,"responseStart":634,"responseEnd":745,"domLoading":638,"domInteractive":4849,"domContentLoadedEventStart":4849,"domContentLoadedEventEnd":4907,"domComplete":5914,"loadEventStart":5914,"loadEventEnd":5918,"userAgent":"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)","marks":[{"name":"bigPipe.sidebar-id.start","time":4820.200000047684},{"name":"bigPipe.sidebar-id.end","time":4821},{"name":"bigPipe.activity-panel-pipe-id.start","time":4821.200000047684},{"name":"bigPipe.activity-panel-pipe-id.end","time":4822.300000011921},{"name":"activityTabFullyLoaded","time":4954.5}],"measures":[],"correlationId":"e545a573e9e552","effectiveType":"4g","downlink":10,"rtt":0,"serverDuration":238,"dbReadsTimeInMs":5,"dbConnsTimeInMs":13,"applicationHash":"ace47f9899e9ee25d7157d59aa17ab06aee30d3d","experiments":[]}}
Specifically in the case of my usage pattern, an OOME cascade like the above will damage IPC during a scan, and subsequent writes from the client are what stall forever.