Details
-
Bug
-
Status: Open
-
Minor
-
Resolution: Unresolved
-
0.10.2.0
-
None
-
None
-
None
-
Ubuntu 14.04, Java 8(1.8.0_65)
Description
One of our kafka brokers shut down after a load test and while there are some corrupted index files , the broker is failing to start with a unsafe memory access error
[2017-07-23 15:52:32,019] FATAL Fatal error during KafkaServerStartable startup. Prepare to shutdown (kafka.server.KafkaServerStartable) java.lang.InternalError: a fault occurred in a recent unsafe memory access operation in compiled Java code at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:53) at org.apache.kafka.common.utils.Utils.readFully(Utils.java:854) at org.apache.kafka.common.utils.Utils.readFullyOrFail(Utils.java:827) at org.apache.kafka.common.record.FileLogInputStream$FileChannelLogEntry.loadRecord(FileLogInputStream.java:136) at org.apache.kafka.common.record.FileLogInputStream$FileChannelLogEntry.record(FileLogInputStream.java:149) at kafka.log.LogSegment$$anonfun$recover$1.apply(LogSegment.scala:225) at kafka.log.LogSegment$$anonfun$recover$1.apply(LogSegment.scala:224) at scala.collection.Iterator$class.foreach(Iterator.scala:893) at scala.collection.AbstractIterator.foreach(Iterator.scala:1336) at scala.collection.IterableLike$class.foreach(IterableLike.scala:72) at scala.collection.AbstractIterable.foreach(Iterable.scala:54) at kafka.log.LogSegment.recover(LogSegment.scala:224) at kafka.log.Log$$anonfun$loadSegments$4.apply(Log.scala:231) at kafka.log.Log$$anonfun$loadSegments$4.apply(Log.scala:188) at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733) at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186) at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732) at kafka.log.Log.loadSegments(Log.scala:188) at kafka.log.Log.<init>(Log.scala:116) at kafka.log.LogManager$$anonfun$loadLogs$2$$anonfun$3$$anonfun$apply$10$$anonfun$apply$1.apply$mcV$sp(LogManager.scala:157) at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:57) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)
This doesn't seem to be same as https://issues.apache.org/jira/browse/KAFKA-1554 because these topics are actively in use and the other empty indices are recovered fine..
Kafka on the machine had died because the disk was full.
It seems to have resolved after the disk issue. Should kafka just check disk at startup and refuse to continue starting up?
Attachments
Issue Links
- is related to
-
KAFKA-5705 Kafka Server start failed and reports "unsafe memory access operation"
- Reopened