Kafka
  1. Kafka
  2. KAFKA-695

Broker shuts down due to attempt to read a closed index file

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.8.0
    • Fix Version/s: 0.8.0
    • Component/s: log
    • Labels:

      Description

      Broker shuts down with the following error message -

      013/01/11 01:43:51.320 ERROR [KafkaApis] [request-expiration-task] [kafka] [] [KafkaApi-277] error when processing request (service_metrics,2,39192,2000000)
      java.nio.channels.ClosedChannelException
      at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:88)
      at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:613)
      at kafka.log.FileMessageSet.searchFor(FileMessageSet.scala:82)
      at kafka.log.LogSegment.translateOffset(LogSegment.scala:76)
      at kafka.log.LogSegment.read(LogSegment.scala:106)
      at kafka.log.Log.read(Log.scala:386)
      at kafka.server.KafkaApis.kafka$server$KafkaApis$$readMessageSet(KafkaApis.scala:369)
      at kafka.server.KafkaApis$$anonfun$kafka$server$KafkaApis$$readMessageSets$1.apply(KafkaApis.scala:327)
      at kafka.server.KafkaApis$$anonfun$kafka$server$KafkaApis$$readMessageSets$1.apply(KafkaApis.scala:323)
      at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206)
      at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206)
      at scala.collection.immutable.Map$Map1.foreach(Map.scala:105)
      at scala.collection.TraversableLike$class.map(TraversableLike.scala:206)
      at scala.collection.immutable.Map$Map1.map(Map.scala:93)
      at kafka.server.KafkaApis.kafka$server$KafkaApis$$readMessageSets(KafkaApis.scala:323)
      at kafka.server.KafkaApis$FetchRequestPurgatory.expire(KafkaApis.scala:519)
      at kafka.server.KafkaApis$FetchRequestPurgatory.expire(KafkaApis.scala:501)
      at kafka.server.RequestPurgatory$ExpiredRequestReaper.run(RequestPurgatory.scala:222)
      at java.lang.Thread.run(Thread.java:619)
      2013/01/11 01:43:52.815 INFO [Processor] [kafka-processor-10251-2] [kafka] [] Closing socket connection to /172.20.72.244.
      2013/01/11 01:43:54.286 INFO [Processor] [kafka-processor-10251-3] [kafka] [] Closing socket connection to /172.20.72.243.
      2013/01/11 01:43:54.385 ERROR [LogManager] [kafka-logflusher-1] [kafka] [] [Log Manager on Broker 277] Error flushing topic service_metrics
      java.nio.channels.ClosedChannelException
      at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:88)
      at sun.nio.ch.FileChannelImpl.force(FileChannelImpl.java:349)
      at kafka.log.FileMessageSet$$anonfun$flush$1.apply$mcV$sp(FileMessageSet.scala:154)
      at kafka.log.FileMessageSet$$anonfun$flush$1.apply(FileMessageSet.scala:154)
      at kafka.log.FileMessageSet$$anonfun$flush$1.apply(FileMessageSet.scala:154)
      at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
      at kafka.log.FileMessageSet.flush(FileMessageSet.scala:153)
      at kafka.log.LogSegment.flush(LogSegment.scala:151)
      at kafka.log.Log.flush(Log.scala:493)
      at kafka.log.LogManager$$anonfun$kafka$log$LogManager$$flushDirtyLogs$2.apply(LogManager.scala:319)
      at kafka.log.LogManager$$anonfun$kafka$log$LogManager$$flushDirtyLogs$2.apply(LogManager.scala:310)
      at scala.collection.Iterator$class.foreach(Iterator.scala:631)
      at scala.collection.JavaConversions$JIteratorWrapper.foreach(JavaConversions.scala:474)
      at scala.collection.IterableLike$class.foreach(IterableLike.scala:79)
      at scala.collection.JavaConversions$JCollectionWrapper.foreach(JavaConversions.scala:495)
      at kafka.log.LogManager.kafka$log$LogManager$$flushDirtyLogs(LogManager.scala:310)
      at kafka.log.LogManager$$anonfun$startup$2.apply$mcV$sp(LogManager.scala:144)
      at kafka.utils.Utils$$anon$2.run(Utils.scala:66)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
      at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
      at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      at java.lang.Thread.run(Thread.java:619)
      2013/01/11 01:43:54.447 FATAL [LogManager] [kafka-logflusher-1] [kafka] [] [Log Manager on Broker 277] Halting due to unrecoverable I/O error while flushing logs: null
      java.nio.channels.ClosedChannelException
      at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:88)
      at sun.nio.ch.FileChannelImpl.force(FileChannelImpl.java:349)
      at kafka.log.FileMessageSet$$anonfun$flush$1.apply$mcV$sp(FileMessageSet.scala:154)
      at kafka.log.FileMessageSet$$anonfun$flush$1.apply(FileMessageSet.scala:154)
      at kafka.log.FileMessageSet$$anonfun$flush$1.apply(FileMessageSet.scala:154)
      at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
      at kafka.log.FileMessageSet.flush(FileMessageSet.scala:153)
      at kafka.log.LogSegment.flush(LogSegment.scala:151)
      at kafka.log.Log.flush(Log.scala:493)

      at kafka.log.LogSegment.flush(LogSegment.scala:151)
      at kafka.log.Log.flush(Log.scala:493)
      at kafka.log.LogManager$$anonfun$kafka$log$LogManager$$flushDirtyLogs$2.apply(LogManager.scala:319)
      at kafka.log.LogManager$$anonfun$kafka$log$LogManager$$flushDirtyLogs$2.apply(LogManager.scala:310)
      at scala.collection.Iterator$class.foreach(Iterator.scala:631)
      at scala.collection.JavaConversions$JIteratorWrapper.foreach(JavaConversions.scala:474)
      at scala.collection.IterableLike$class.foreach(IterableLike.scala:79)
      at scala.collection.JavaConversions$JCollectionWrapper.foreach(JavaConversions.scala:495)
      at kafka.log.LogManager.kafka$log$LogManager$$flushDirtyLogs(LogManager.scala:310)
      at kafka.log.LogManager$$anonfun$startup$2.apply$mcV$sp(LogManager.scala:144)
      at kafka.utils.Utils$$anon$2.run(Utils.scala:66)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
      at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
      at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      at java.lang.Thread.run(Thread.java:619)
      2013/01/11 01:43:54.512 INFO [ComponentsContextLoaderListener] [Thread-2] [kafka] [] Shutting down...

      1. kafka-695_followup_v2.patch
        10 kB
        Jun Rao
      2. kafka-695_followup.patch
        8 kB
        Jun Rao
      3. kafka-695_v2.patch
        3 kB
        Jun Rao
      4. kafka-695.patch
        2 kB
        Jun Rao

        Issue Links

          Activity

          Neha Narkhede created issue -
          Neha Narkhede made changes -
          Field Original Value New Value
          Component/s log [ 12320320 ]
          Neha Narkhede made changes -
          Labels p2
          Neha Narkhede made changes -
          Labels p2 p1
          Neha Narkhede made changes -
          Assignee Jun Rao [ junrao ]
          Neha Narkhede made changes -
          Link This issue is duplicated by KAFKA-719 [ KAFKA-719 ]
          Jun Rao made changes -
          Attachment kafka-695.patch [ 12566696 ]
          Jun Rao made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Fix Version/s 0.8 [ 12317244 ]
          Jun Rao made changes -
          Attachment kafka-695_v2.patch [ 12566836 ]
          Neha Narkhede made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Jun Rao made changes -
          Attachment kafka-695_followup.patch [ 12567187 ]
          Jun Rao made changes -
          Resolution Fixed [ 1 ]
          Status Resolved [ 5 ] Reopened [ 4 ]
          Jun Rao made changes -
          Attachment kafka-695_followup_v2.patch [ 12567980 ]
          Neha Narkhede made changes -
          Status Reopened [ 4 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Neha Narkhede made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

            People

            • Assignee:
              Jun Rao
              Reporter:
              Neha Narkhede
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development