Details
-
Bug
-
Status: Closed
-
Major
-
Resolution: Won't Do
-
1.7
-
None
-
None
-
* OS: Amazon Linux AMI release 2014.03 (centos-like)
* CPU: Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GH
* MemTotal: 3843028 kB
* Startup command: apollo -ea -server -XX:+AggressiveOpts -XX:+OptimizeStringConcat -XX:+UseFastAccessorMethods -XX:+UseBiasedLocking -Xmx2662m -Xms128m -Dcom.sun.management.jmxremote -Dapollo.home=/opt/apache-apollo-1.7 -Dapollo.base=/var/lib/subscriptions -classpath /opt/apache-apollo-1.7/lib/apollo-boot.jar org.apache.activemq.apollo.boot.Apollo /var/lib/subscriptions/lib\;/opt/apache-apollo-1.7/lib org.apache.activemq.apollo.cli.Apollo run
* Java: OpenJDK 64-Bit Server VM 1.7.0_55* OS: Amazon Linux AMI release 2014.03 (centos-like) * CPU: Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GH * MemTotal: 3843028 kB * Startup command: apollo -ea -server -XX:+AggressiveOpts -XX:+OptimizeStringConcat -XX:+UseFastAccessorMethods -XX:+UseBiasedLocking -Xmx2662m -Xms128m -Dcom.sun.management.jmxremote -Dapollo.home=/opt/apache-apollo-1.7 -Dapollo.base=/var/lib/subscriptions -classpath /opt/apache-apollo-1.7/lib/apollo-boot.jar org.apache.activemq.apollo.boot.Apollo /var/lib/subscriptions/lib\;/opt/apache-apollo-1.7/lib org.apache.activemq.apollo.cli.Apollo run * Java: OpenJDK 64-Bit Server VM 1.7.0_55
Description
Apollo has been failing under load, and I can consistently reproduce it with this formula. I've attached two things to this bug report:
- All apollo logs
- The load-test application (both source code and compiled binary)
After a few hours of load testing (one producer, five consumers, roughly 7000 messages/second, message size 1.4k), leveldb has a "Short Write" exception then Apollo goes into recovery mode and runs out of memory. All logs are attached, relevant section of stacktrace.log looks like this:2014-06-25 23:54:55,593 | WARN | stackref=146d56da433 java.io.IOException: Short write at org.apache.activemq.apollo.broker.store.leveldb.RecordLog$LogAppender.flush(RecordLog.scala:206) at org.apache.activemq.apollo.broker.store.leveldb.RecordLog.appender(RecordLog.scala:464) at org.apache.activemq.apollo.broker.store.leveldb.LevelDBClient$$anonfun$store$1.apply$mcV$sp(LevelDBClient.scala:870) at org.apache.activemq.apollo.broker.store.leveldb.LevelDBClient$$anonfun$store$1.apply(LevelDBClient.scala:870) at org.apache.activemq.apollo.broker.store.leveldb.LevelDBClient$$anonfun$store$1.apply(LevelDBClient.scala:870) at org.apache.activemq.apollo.broker.store.leveldb.LevelDBClient.using_index(LevelDBClient.scala:664) at org.apache.activemq.apollo.broker.store.leveldb.LevelDBClient$$anonfun$retry_using_index$1.apply(LevelDBClient.scala:670) at org.apache.activemq.apollo.broker.store.leveldb.LevelDBClient.retry(LevelDBClient.scala:758) at org.apache.activemq.apollo.broker.store.leveldb.LevelDBClient.retry_using_index(LevelDBClient.scala:670) at org.apache.activemq.apollo.broker.store.leveldb.LevelDBClient.store(LevelDBClient.scala:869) at org.apache.activemq.apollo.broker.store.leveldb.LevelDBStore$$anonfun$store$1.apply$mcV$sp(LevelDBStore.scala:82) at org.fusesource.hawtdispatch.package$$anon$4.run(hawtdispatch.scala:357) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) 2014-06-26 00:10:39,219 | WARN | stackref=146d56da434 java.lang.OutOfMemoryError: Java heap space 2014-06-26 00:14:44,191 | WARN | stackref=146d56da435 java.lang.OutOfMemoryError: Java heap space 2014-06-26 00:16:40,941 | WARN | stackref=146d56da438 java.lang.OutOfMemoryError: Java heap space 2014-06-26 00:22:30,467 | WARN | stackref=146d56da439 java.lang.OutOfMemoryError: Java heap space
And the same section of apollo.log looks like this:
2014-06-25 23:54:55,584 | WARN | DB operation failed. (entering recovery mode): java.io.IOException: Short write | 146d56da433 2014-06-26 00:01:35,273 | WARN | handle failed | java.lang.OutOfMemoryError: Java heap space at java.io.ByteArrayOutputStream.<init>(ByteArrayOutputStream.java:77) at sun.security.ssl.OutputRecord.<init>(OutputRecord.java:76) at sun.security.ssl.EngineOutputRecord.<init>(EngineOutputRecord.java:65) at sun.security.ssl.HandshakeOutStream.<init>(HandshakeOutStream.java:63) at sun.security.ssl.Handshaker.activate(Handshaker.java:491) at sun.security.ssl.SSLEngineImpl.kickstartHandshake(SSLEngineImpl.java:711) at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:809) at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:761) at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) at org.eclipse.jetty.io.nio.SslConnection.unwrap(SslConnection.java:519) at org.eclipse.jetty.io.nio.SslConnection.process(SslConnection.java:354) at org.eclipse.jetty.io.nio.SslConnection.access$900(SslConnection.java:43) at org.eclipse.jetty.io.nio.SslConnection$SslEndPoint.fill(SslConnection.java:661) at org.eclipse.jetty.http.HttpParser.fill(HttpParser.java:1030) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:275) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:230) at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:77) at org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.java:191) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:622) at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:46) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) 2014-06-26 00:04:00,378 | WARN | | java.lang.OutOfMemoryError: Java heap space at sun.security.ssl.InputRecord.<init>(InputRecord.java:84) at sun.security.ssl.EngineInputRecord.<init>(EngineInputRecord.java:63) at sun.security.ssl.SSLEngineImpl.init(SSLEngineImpl.java:407) at sun.security.ssl.SSLEngineImpl.<init>(SSLEngineImpl.java:343) at sun.security.ssl.SSLContextImpl.engineCreateSSLEngine(SSLContextImpl.java:208) at javax.net.ssl.SSLContext.createSSLEngine(SSLContext.java:362) at org.eclipse.jetty.util.ssl.SslContextFactory.newSslEngine(SslContextFactory.java:1495) at org.eclipse.jetty.server.ssl.SslSelectChannelConnector.createSSLEngine(SslSelectChannelConnector.java:588) at org.eclipse.jetty.server.ssl.SslSelectChannelConnector.newConnection(SslSelectChannelConnector.java:551) at org.eclipse.jetty.server.nio.SelectChannelConnector$ConnectorSelectorManager.newConnection(SelectChannelConnector.java:320) at org.eclipse.jetty.server.nio.SelectChannelConnector.newEndPoint(SelectChannelConnector.java:267) at org.eclipse.jetty.server.nio.SelectChannelConnector$ConnectorSelectorManager.newEndPoint(SelectChannelConnector.java:326) at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.createEndPoint(SelectorManager.java:836) at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:491) at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:285) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) 2014-06-26 00:10:39,219 | WARN | java.lang.OutOfMemoryError: Java heap space | 146d56da434 2014-06-26 00:10:39,220 | WARN | handle failed | java.lang.OutOfMemoryError: Java heap space at java.math.BigInteger.multiplyToLen(BigInteger.java:1226) at java.math.BigInteger.oddModPow(BigInteger.java:1764) at java.math.BigInteger.modPow(BigInteger.java:1599) at sun.security.rsa.RSACore.crtCrypt(RSACore.java:154) at sun.security.rsa.RSACore.rsa(RSACore.java:101) at sun.security.rsa.RSASignature.engineSign(RSASignature.java:177) at java.security.Signature$Delegate.engineSign(Signature.java:1160) at java.security.Signature.sign(Signature.java:553) at sun.security.ssl.HandshakeMessage$ECDH_ServerKeyExchange.<init>(HandshakeMessage.java:1012) at sun.security.ssl.ServerHandshaker.clientHello(ServerHandshaker.java:772) at sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:167) at sun.security.ssl.Handshaker.processLoop(Handshaker.java:868) at sun.security.ssl.Handshaker$1.run(Handshaker.java:808) at sun.security.ssl.Handshaker$1.run(Handshaker.java:806) at java.security.AccessController.doPrivileged(Native Method) at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1227) at org.eclipse.jetty.io.nio.SslConnection.process(SslConnection.java:370) at org.eclipse.jetty.io.nio.SslConnection.access$900(SslConnection.java:43) at org.eclipse.jetty.io.nio.SslConnection$SslEndPoint.fill(SslConnection.java:661) at org.eclipse.jetty.http.HttpParser.fill(HttpParser.java:1030) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:275) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:230) at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:77) at org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.java:191) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:622) at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:46) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) 2014-06-26 00:12:09,137 | WARN | handle failed | java.lang.OutOfMemoryError: Java heap space 2014-06-26 00:12:54,045 | WARN | handle failed | java.lang.OutOfMemoryError: Java heap space 2014-06-26 00:14:44,191 | WARN | java.lang.OutOfMemoryError: Java heap space | 146d56da435 2014-06-26 00:16:33,678 | WARN | java.lang.OutOfMemoryError: Java heap space | 146d56da438 2014-06-26 00:20:41,583 | WARN | | java.lang.OutOfMemoryError: Java heap space 2014-06-26 00:21:53,264 | WARN | | java.lang.OutOfMemoryError: Java heap space 2014-06-26 00:22:07,730 | WARN | handle failed | java.lang.OutOfMemoryError: Java heap space 2014-06-26 00:22:07,731 | WARN | java.lang.OutOfMemoryError: Java heap space | 146d56da439 2014-06-26 00:22:07,731 | WARN | handle failed | java.lang.OutOfMemoryError: Java heap space 2014-06-26 00:23:28,222 | WARN | handle failed | java.lang.OutOfMemoryError: Java heap space 2014-06-26 00:23:49,662 | WARN | handle failed | java.lang.OutOfMemoryError: Java heap space 2014-06-26 00:25:22,976 | WARN | handle failed | java.lang.OutOfMemoryError: Java heap space 2014-06-26 00:25:37,279 | WARN | handle failed | java.lang.OutOfMemoryError: Java heap space 2014-06-26 00:26:05,740 | WARN | | java.lang.OutOfMemoryError: Java heap space 2014-06-26 00:26:34,485 | WARN | | java.lang.OutOfMemoryError: Java heap space 2014-06-26 00:27:17,858 | WARN | handle failed | java.lang.OutOfMemoryError: Java heap space 2014-06-26 00:27:39,331 | WARN | handle failed | java.lang.OutOfMemoryError: Java heap space