Uploaded image for project: 'Bookkeeper'
  1. Bookkeeper
  2. BOOKKEEPER-893

bookie exited with status 0 on journal I/O exception

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 4.3.2
    • Fix Version/s: 4.4.0
    • Component/s: bookkeeper-server
    • Labels:
      None

      Description

      06:13:32.041 [BookieJournal-3181] ERROR org.apache.bookkeeper.bookie.Journal - I/O exception in Journal thread!
      java.nio.channels.ClosedChannelException: null
              at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:110) ~[na:1.8.0_60]
              at sun.nio.ch.FileChannelImpl.position(FileChannelImpl.java:253) ~[na:1.8.0_60]
              at org.apache.bookkeeper.bookie.Journal.run(Journal.java:964) ~[bookkeeper-server-4.3.1.33.jar:4.3.1]
      06:13:32.041 [BookieJournal-3181] INFO  org.apache.bookkeeper.bookie.Journal - Journal exited loop!
      06:13:32.041 [Bookie-3181] ERROR org.apache.bookkeeper.bookie.Bookie  - Journal manager quits unexpectedly.
      06:13:32.042 [Bookie-3181] INFO  org.apache.bookkeeper.bookie.Bookie  - Triggering shutdown of Bookie-3181 with exitCode 5
      06:13:32.042 [BookieShutdownTrigger] INFO  org.apache.bookkeeper.bookie.Bookie  - Shutting down Bookie-3181 with exitCode 5
      06:13:32.044 [main] INFO  o.a.bookkeeper.proto.BookieServer    - Stop stats provider
      06:13:32.045 [bookie-reg-0] INFO  org.apache.zookeeper.ZooKeeper       - Session: 0x14ff63dc02cbd02 closed
      06:13:32.045 [bookie-reg-0-EventThread] INFO  org.apache.zookeeper.ClientCnxn      - EventThread shut down
      06:13:32.045 [BookieShutdownTrigger] INFO  org.apache.bookkeeper.bookie.Journal - Shutting down Journal
      06:13:32.046 [Thread-2] INFO  o.a.bookkeeper.proto.BookieServer    - Shutting down BookieServer
      06:13:32.046 [Thread-2] INFO  o.a.b.proto.BookieNettyServer        - Shutting down BookieNettyServer
      06:13:32.046 [ForceWriteThread] ERROR org.apache.bookkeeper.bookie.Journal - ForceWrite thread interrupted
      java.lang.InterruptedException: null
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) ~[na:1.8.0_60]
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048) ~[na:1.8.0_60]
              at org.apache.bookkeeper.util.UnboundArrayBlockingQueue.take(UnboundArrayBlockingQueue.java:312) ~[bookkeeper-server-4.3.1.33.jar:4.3.1]
              at org.apache.bookkeeper.bookie.Journal$ForceWriteThread.run(Journal.java:448) ~[bookkeeper-server-4.3.1.33.jar:4.3.1]
      ...
      06:13:32.184 [bookie-io-0] WARN  i.n.channel.DefaultChannelPipeline   - An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
      java.nio.channels.ClosedChannelException: null
      

      When the bookie shuts down due to some exception, it exits with exitCode 0.
      We try to get the exitCode when calling System.exit() here:
      https://github.com/apache/bookkeeper/blob/master/bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/BookieServer.java#L400

      But the exitCode is set by the shutdown hook which is triggered by System.exit(). Thus, we will always exit with code 0.

      Solution:
      We explicitly call bookie shutdown to set the exit code.

        Attachments

          Activity

            People

            • Assignee:
              sboobna Siddharth Sunil Boobna
              Reporter:
              sboobna Siddharth Sunil Boobna
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Due:
                Created:
                Updated:
                Resolved: