Kafka
  1. Kafka
  2. KAFKA-1065

IO exception on windows when high throughput of messages

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Duplicate
    • Affects Version/s: 0.8.0
    • Fix Version/s: None
    • Component/s: log
    • Labels:
      None
    • Environment:
      Windows 7 64bit

      Description

      When a large number of messages are sent per second to a broker on Windows a memory mapping exception occurs and kills Kafka. The exception follows :

      kafka.common.KafkaStorageException: I/O exception in append to log 'test-0'
      	at kafka.log.Log.append(Log.scala:349)
      	at kafka.cluster.Partition.appendMessagesToLeader(Partition.scala:340)
      	at kafka.server.KafkaApis$$anonfun$appendToLocalLog$2.apply(KafkaApis.scala:236)
      	at kafka.server.KafkaApis$$anonfun$appendToLocalLog$2.apply(KafkaApis.scala:228)
      	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206)
      	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206)
      	at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:80)
      	at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:80)
      	at scala.collection.Iterator$class.foreach(Iterator.scala:631)
      	at scala.collection.mutable.HashTable$$anon$1.foreach(HashTable.scala:161)
      	at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:194)
      	at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39)
      	at scala.collection.mutable.HashMap.foreach(HashMap.scala:80)
      	at scala.collection.TraversableLike$class.map(TraversableLike.scala:206)
      	at scala.collection.mutable.HashMap.map(HashMap.scala:39)
      	at kafka.server.KafkaApis.appendToLocalLog(KafkaApis.scala:228)
      	at kafka.server.KafkaApis.handleProducerRequest(KafkaApis.scala:162)
      	at kafka.server.KafkaApis.handle(KafkaApis.scala:66)
      	at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:42)
      	at java.lang.Thread.run(Thread.java:662)
      Caused by: java.io.IOException: The requested operation cannot be performed on a file with a user-mapped section open
      	at java.io.RandomAccessFile.setLength(Native Method)
      	at kafka.log.OffsetIndex.liftedTree2$1(OffsetIndex.scala:263)
      	at kafka.log.OffsetIndex.resize(OffsetIndex.scala:262)
      	at kafka.log.OffsetIndex.trimToValidSize(OffsetIndex.scala:247)
      	at kafka.log.Log.rollToOffset(Log.scala:518)
      	at kafka.log.Log.roll(Log.scala:502)
      	at kafka.log.Log.maybeRoll(Log.scala:484)
      	at kafka.log.Log.append(Log.scala:297)
      	... 19 more

      This seems to have been mentioned in the past on the Kafka mail list and is an issue related to http://bugs.sun.com/view_bug.do?bug_id=4724038.

      Unfortunately this means that we cannot use Kafka on Windows.

        Issue Links

          Activity

          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open In Progress In Progress
          6d 5h 15m 1 Jun Rao 30/Sep/13 15:42
          In Progress In Progress Resolved Resolved
          45s 1 Jun Rao 30/Sep/13 15:42
          Jun Rao made changes -
          Status In Progress [ 3 ] Resolved [ 5 ]
          Resolution Duplicate [ 3 ]
          Jun Rao made changes -
          Link This issue is duplicated by KAFKA-1008 [ KAFKA-1008 ]
          Hide
          Jun Rao added a comment -

          Close this since it duplicates kafka-1008.

          Show
          Jun Rao added a comment - Close this since it duplicates kafka-1008.
          Jun Rao made changes -
          Status Open [ 1 ] In Progress [ 3 ]
          Hide
          Carl Austin added a comment -

          Sorry for the slow reply, I've only just got around to testing this out, and it works a treat! Thanks for the help.

          Show
          Carl Austin added a comment - Sorry for the slow reply, I've only just got around to testing this out, and it works a treat! Thanks for the help.
          Hide
          Jun Rao added a comment -

          Could you try the patch in https://issues.apache.org/jira/browse/KAFKA-1008? I think it's fixing the same issue on windows.

          Show
          Jun Rao added a comment - Could you try the patch in https://issues.apache.org/jira/browse/KAFKA-1008? I think it's fixing the same issue on windows.
          Neha Narkhede made changes -
          Field Original Value New Value
          Component/s log [ 12320320 ]
          Component/s producer [ 12320325 ]
          Carl Austin created issue -

            People

            • Assignee:
              Jun Rao
              Reporter:
              Carl Austin
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development