Uploaded image for project: 'ZooKeeper'
  1. ZooKeeper
  2. ZOOKEEPER-2247

Zookeeper service becomes unavailable when leader fails to write transaction log

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 3.5.0
    • Fix Version/s: 3.4.9, 3.5.3, 3.6.0
    • Component/s: None
    • Labels:
      None

      Description

      Zookeeper service becomes unavailable when leader fails to write transaction log. Bellow are the exceptions

      2015-08-14 15:41:18,556 [myid:100] - ERROR [SyncThread:100:ZooKeeperCriticalThread@48] - Severe unrecoverable error, from thread : SyncThread:100
      java.io.IOException: Input/output error
      	at sun.nio.ch.FileDispatcherImpl.force0(Native Method)
      	at sun.nio.ch.FileDispatcherImpl.force(FileDispatcherImpl.java:76)
      	at sun.nio.ch.FileChannelImpl.force(FileChannelImpl.java:376)
      	at org.apache.zookeeper.server.persistence.FileTxnLog.commit(FileTxnLog.java:331)
      	at org.apache.zookeeper.server.persistence.FileTxnSnapLog.commit(FileTxnSnapLog.java:380)
      	at org.apache.zookeeper.server.ZKDatabase.commit(ZKDatabase.java:563)
      	at org.apache.zookeeper.server.SyncRequestProcessor.flush(SyncRequestProcessor.java:178)
      	at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:113)
      2015-08-14 15:41:18,559 [myid:100] - INFO  [SyncThread:100:ZooKeeperServer$ZooKeeperServerListenerImpl@500] - Thread SyncThread:100 exits, error code 1
      2015-08-14 15:41:18,559 [myid:100] - INFO  [SyncThread:100:ZooKeeperServer@523] - shutting down
      2015-08-14 15:41:18,560 [myid:100] - INFO  [SyncThread:100:SessionTrackerImpl@232] - Shutting down
      2015-08-14 15:41:18,560 [myid:100] - INFO  [SyncThread:100:LeaderRequestProcessor@77] - Shutting down
      2015-08-14 15:41:18,560 [myid:100] - INFO  [SyncThread:100:PrepRequestProcessor@1035] - Shutting down
      2015-08-14 15:41:18,560 [myid:100] - INFO  [SyncThread:100:ProposalRequestProcessor@88] - Shutting down
      2015-08-14 15:41:18,561 [myid:100] - INFO  [SyncThread:100:CommitProcessor@356] - Shutting down
      2015-08-14 15:41:18,561 [myid:100] - INFO  [CommitProcessor:100:CommitProcessor@191] - CommitProcessor exited loop!
      2015-08-14 15:41:18,562 [myid:100] - INFO  [SyncThread:100:Leader$ToBeAppliedRequestProcessor@915] - Shutting down
      2015-08-14 15:41:18,562 [myid:100] - INFO  [SyncThread:100:FinalRequestProcessor@646] - shutdown of request processor complete
      2015-08-14 15:41:18,562 [myid:100] - INFO  [SyncThread:100:SyncRequestProcessor@191] - Shutting down
      2015-08-14 15:41:18,563 [myid:100] - INFO  [ProcessThread(sid:100 cport:-1)::PrepRequestProcessor@159] - PrepRequestProcessor exited loop!
      

      After this exception Leader server still remains leader. After this non recoverable exception the leader should go down and let other followers become leader.

        Attachments

        1. ZOOKEEPER-2247-br-3.4.patch
          24 kB
          Rakesh R
        2. ZOOKEEPER-2247-br-3.4.patch
          30 kB
          Rakesh R
        3. ZOOKEEPER-2247-b3.5.patch
          12 kB
          Flavio Junqueira
        4. ZOOKEEPER-2247-23.patch
          29 kB
          Rakesh R
        5. ZOOKEEPER-2247-22.patch
          29 kB
          Rakesh R
        6. ZOOKEEPER-2247-21.patch
          29 kB
          Rakesh R
        7. ZOOKEEPER-2247-20.patch
          28 kB
          Rakesh R
        8. ZOOKEEPER-2247-19.patch
          28 kB
          Rakesh R
        9. ZOOKEEPER-2247-18.patch
          28 kB
          Rakesh R
        10. ZOOKEEPER-2247-17.patch
          28 kB
          Rakesh R
        11. ZOOKEEPER-2247-16.patch
          30 kB
          Rakesh R
        12. ZOOKEEPER-2247-15.patch
          24 kB
          Rakesh R
        13. ZOOKEEPER-2247-14.patch
          25 kB
          Rakesh R
        14. ZOOKEEPER-2247-13.patch
          24 kB
          Rakesh R
        15. ZOOKEEPER-2247-12.patch
          22 kB
          Rakesh R
        16. ZOOKEEPER-2247-11.patch
          21 kB
          Rakesh R
        17. ZOOKEEPER-2247-10.patch
          21 kB
          Rakesh R
        18. ZOOKEEPER-2247-09.patch
          21 kB
          Rakesh R
        19. ZOOKEEPER-2247-07.patch
          21 kB
          Rakesh R
        20. ZOOKEEPER-2247-06.patch
          19 kB
          Mohammad Arshad
        21. ZOOKEEPER-2247-05.patch
          35 kB
          Mohammad Arshad
        22. ZOOKEEPER-2247-04.patch
          45 kB
          Mohammad Arshad
        23. ZOOKEEPER-2247-03.patch
          45 kB
          Mohammad Arshad
        24. ZOOKEEPER-2247-02.patch
          37 kB
          Mohammad Arshad
        25. ZOOKEEPER-2247-01.patch
          37 kB
          Mohammad Arshad

          Issue Links

            Activity

              People

              • Assignee:
                rakeshr Rakesh R
                Reporter:
                arshad.mohammad Mohammad Arshad
              • Votes:
                0 Vote for this issue
                Watchers:
                11 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: