Kafka
  1. Kafka
  2. KAFKA-1065

IO exception on windows when high throughput of messages

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Duplicate
    • Affects Version/s: 0.8.0
    • Fix Version/s: None
    • Component/s: log
    • Labels:
      None
    • Environment:
      Windows 7 64bit

      Description

      When a large number of messages are sent per second to a broker on Windows a memory mapping exception occurs and kills Kafka. The exception follows :

      kafka.common.KafkaStorageException: I/O exception in append to log 'test-0'
      	at kafka.log.Log.append(Log.scala:349)
      	at kafka.cluster.Partition.appendMessagesToLeader(Partition.scala:340)
      	at kafka.server.KafkaApis$$anonfun$appendToLocalLog$2.apply(KafkaApis.scala:236)
      	at kafka.server.KafkaApis$$anonfun$appendToLocalLog$2.apply(KafkaApis.scala:228)
      	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206)
      	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206)
      	at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:80)
      	at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:80)
      	at scala.collection.Iterator$class.foreach(Iterator.scala:631)
      	at scala.collection.mutable.HashTable$$anon$1.foreach(HashTable.scala:161)
      	at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:194)
      	at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39)
      	at scala.collection.mutable.HashMap.foreach(HashMap.scala:80)
      	at scala.collection.TraversableLike$class.map(TraversableLike.scala:206)
      	at scala.collection.mutable.HashMap.map(HashMap.scala:39)
      	at kafka.server.KafkaApis.appendToLocalLog(KafkaApis.scala:228)
      	at kafka.server.KafkaApis.handleProducerRequest(KafkaApis.scala:162)
      	at kafka.server.KafkaApis.handle(KafkaApis.scala:66)
      	at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:42)
      	at java.lang.Thread.run(Thread.java:662)
      Caused by: java.io.IOException: The requested operation cannot be performed on a file with a user-mapped section open
      	at java.io.RandomAccessFile.setLength(Native Method)
      	at kafka.log.OffsetIndex.liftedTree2$1(OffsetIndex.scala:263)
      	at kafka.log.OffsetIndex.resize(OffsetIndex.scala:262)
      	at kafka.log.OffsetIndex.trimToValidSize(OffsetIndex.scala:247)
      	at kafka.log.Log.rollToOffset(Log.scala:518)
      	at kafka.log.Log.roll(Log.scala:502)
      	at kafka.log.Log.maybeRoll(Log.scala:484)
      	at kafka.log.Log.append(Log.scala:297)
      	... 19 more

      This seems to have been mentioned in the past on the Kafka mail list and is an issue related to http://bugs.sun.com/view_bug.do?bug_id=4724038.

      Unfortunately this means that we cannot use Kafka on Windows.

        Issue Links

          Activity

          Hide
          Jun Rao added a comment -

          Could you try the patch in https://issues.apache.org/jira/browse/KAFKA-1008? I think it's fixing the same issue on windows.

          Show
          Jun Rao added a comment - Could you try the patch in https://issues.apache.org/jira/browse/KAFKA-1008? I think it's fixing the same issue on windows.
          Hide
          Carl Austin added a comment -

          Sorry for the slow reply, I've only just got around to testing this out, and it works a treat! Thanks for the help.

          Show
          Carl Austin added a comment - Sorry for the slow reply, I've only just got around to testing this out, and it works a treat! Thanks for the help.
          Hide
          Jun Rao added a comment -

          Close this since it duplicates kafka-1008.

          Show
          Jun Rao added a comment - Close this since it duplicates kafka-1008.

            People

            • Assignee:
              Jun Rao
              Reporter:
              Carl Austin
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development