[KAFKA-2560] Fatal error during KafkaServer startup because of Map failed error.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.8.2.1
• Component/s: log
• Environment: Linux
• Assignee: Jay Kreps
• Reporter: Bo Wang
• Created: 21/Sep/15 09:16
• Updated: 24/Aug/17 15:53
• Resolved: 24/Aug/17 15:53

Description

I have 3 kafka nodes,
create 30 topics ,every topic has 100 pations, and replica factor is 2.

Kafka server start failed,
2015-09-21 10:28:35,668 | INFO | pool-2-thread-1 | Recovering unflushed segment 0 in log testTopic_14-34. | kafka.utils.Logging$class.info(Logging.scala:68)
2015-09-21 10:28:35,942 | ERROR | main | There was an error in one of the threads during logs loading: java.io.IOException: Map failed | kafka.utils.Logging$class.error(Logging.scala:97)
2015-09-21 10:28:35,943 | INFO | pool-2-thread-5 | Recovering unflushed segment 0 in log testTopic_17-23. | kafka.utils.Logging$class.info(Logging.scala:68)
2015-09-21 10:28:35,944 | INFO | pool-2-thread-5 | Completed load of log testTopic_17-23 with log end offset 0 | kafka.utils.Logging$class.info(Logging.scala:68)
2015-09-21 10:28:35,945 | FATAL | main | [Kafka Server 54], Fatal error during KafkaServer startup. Prepare to shutdown | kafka.utils.Logging$class.fatal(Logging.scala:116)
java.io.IOException: Map failed
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:907)
at kafka.log.OffsetIndex$$anonfun$resize$1.apply(OffsetIndex.scala:286)
at kafka.log.OffsetIndex$$anonfun$resize$1.apply(OffsetIndex.scala:276)
at kafka.utils.Utils$.inLock(Utils.scala:535)
at kafka.log.OffsetIndex.resize(OffsetIndex.scala:276)
at kafka.log.Log.loadSegments(Log.scala:179)
at kafka.log.Log.<init>(Log.scala:67)
at kafka.log.LogManager$$anonfun$loadLogs$2$$anonfun$3$$anonfun$apply$7$$anonfun$apply$1.apply$mcV$sp(LogManager.scala:142)
at kafka.utils.Utils$$anon$1.run(Utils.scala:54)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.OutOfMemoryError: Map failed
at sun.nio.ch.FileChannelImpl.map0(Native Method)
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:904)
... 13 more
2015-09-21 10:28:35,946 | INFO | pool-2-thread-5 | Recovering unflushed segment 0 in log testTopic_25-77. | kafka.utils.Logging$class.info(Logging.scala:68)
2015-09-21 10:28:35,949 | INFO | main | [Kafka Server 54], shutting down | kafka.utils.Logging$class.info(Logging.scala:68)

Kafka server host's top infomation below:
top - 17:16:23 up 53 min, 6 users, load average: 0.42, 0.99, 1.19
Tasks: 215 total, 1 running, 214 sleeping, 0 stopped, 0 zombie
Cpu(s): 4.5%us, 2.4%sy, 0.0%ni, 92.9%id, 0.1%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 40169M total, 6118M used, 34050M free, 9M buffers
Swap: 0M total, 0M used, 0M free, 431M cached