Uploaded image for project: 'ZooKeeper'
  1. ZooKeeper
  2. ZOOKEEPER-713

zookeeper fails to start - broken snapshot?

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Invalid
    • Affects Version/s: 3.2.2
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Environment:

      debian lenny; ia64; xen virtualization

      Description

      Hi guys,

      The following is not a bug report but rather a question - but as I am attaching large files I am posting it here rather than on mailinglist.

      Today we had major failure in our production environment. Machines in zookeeper cluster gone wild and all clients got disconnected.
      We tried to restart whole zookeeper cluster but cluster got stuck in leader election phase.

      Calling stat command on any machine in the cluster resulted in 'ZooKeeperServer not running' message
      In one of logs I noticed 'Invalid snapshot' message which disturbed me a bit.

      We did not manage to make cluster work again with data. We deleted all version-2 directories on all nodes and then cluster started up without problems.
      Is it possible that snapshot/log data got corrupted in a way which made cluster unable to start?
      Fortunately we could rebuild data we store in zookeeper as we use it only for locks and most of nodes is ephemeral.

      I am attaching contents of version-2 directory from all nodes and server logs.
      Source problem occurred some time before 15. First cluster restart happened at 15:03.
      At some point later we experimented with deleting version-2 directory so I would not look at following restart because they can be misleading due to our actions.

      I am also attaching zoo.cfg. Maybe something is wrong at this place.
      As I know look into logs i see read timeout during initialization phase after 20secs (initLimit=10, tickTime=2000).
      Maybe all I have to do is increase one or other. which one? Are there any downsides of increasing tickTime.

      Best regards, Ɓukasz Osipiuk

      PS. due to attachment size limit I used split. to untar use
      cat nodeX-version-2.tgz-* |tar -xz

        Attachments

        1. zoo.cfg
          0.5 kB
          Lukasz Osipiuk
        2. node3-zookeeper.log.gz
          192 kB
          Lukasz Osipiuk
        3. node3-version-2.tgz-ac
          1.75 MB
          Lukasz Osipiuk
        4. node3-version-2.tgz-ab
          9.00 MB
          Lukasz Osipiuk
        5. node3-version-2.tgz-aa
          9.00 MB
          Lukasz Osipiuk
        6. node2-zookeeper.log.gz
          237 kB
          Lukasz Osipiuk
        7. node2-version-2.tgz-ac
          5.41 MB
          Lukasz Osipiuk
        8. node2-version-2.tgz-ab
          9.00 MB
          Lukasz Osipiuk
        9. node2-version-2.tgz-aa
          9.00 MB
          Lukasz Osipiuk
        10. node1-zookeeper.log.gz
          213 kB
          Lukasz Osipiuk
        11. node1-version-2.tgz-ab
          6.21 MB
          Lukasz Osipiuk
        12. node1-version-2.tgz-aa
          9.00 MB
          Lukasz Osipiuk

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                losipiuk Lukasz Osipiuk
              • Votes:
                0 Vote for this issue
                Watchers:
                0 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: