ZooKeeper
  1. ZooKeeper
  2. ZOOKEEPER-158

Leader and followers increase cpu utilization upon loss of a follower

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.3.0
    • Component/s: quorum
    • Labels:
      None

      Description

      In a set of ZooKeeper servers, when there is a leader operation and supported by a quorum of servers, we have observed that cpu utilization increases substantially once a follower fails or disconnects. Stu Hood provided logs showing this behavior.

      1. dead-follower.tar.gz
        3 kB
        Flavio Junqueira

        Activity

        Hide
        Flavio Junqueira added a comment -

        Attaching logs provided by Stu Hood (copying from JIRA 157).

        Show
        Flavio Junqueira added a comment - Attaching logs provided by Stu Hood (copying from JIRA 157).
        Hide
        Mahadev konar added a comment -

        i dont see anything in the logs that would help us identify the problem. how much was the substantial increase in the cpu utilitization ? the logs of the leader should help....

        Show
        Mahadev konar added a comment - i dont see anything in the logs that would help us identify the problem. how much was the substantial increase in the cpu utilitization ? the logs of the leader should help....
        Hide
        Flavio Junqueira added a comment -

        I also observed this issue using top. To recreate it, I have started 3 servers, killed one follower after a leader had been elected, and use top to observe cpu utilization.I'm not sure how we can pinpoint the problem from the logs.

        Show
        Flavio Junqueira added a comment - I also observed this issue using top. To recreate it, I have started 3 servers, killed one follower after a leader had been elected, and use top to observe cpu utilization.I'm not sure how we can pinpoint the problem from the logs.
        Hide
        Mahadev konar added a comment -

        well if its easy to reproduce then we should be able to just attach a profiler to the nodes.... i will try this out with a profiler and post the results ...

        Show
        Mahadev konar added a comment - well if its easy to reproduce then we should be able to just attach a profiler to the nodes.... i will try this out with a profiler and post the results ...
        Hide
        Patrick Hunt added a comment -

        re logs remember that you can increase the logging level from the default INFO to DEBUG or TRACE by editing the log4j config file.

        Show
        Patrick Hunt added a comment - re logs remember that you can increase the logging level from the default INFO to DEBUG or TRACE by editing the log4j config file.
        Hide
        Flavio Junqueira added a comment -

        I think that profiling is a great idea in this case.

        Show
        Flavio Junqueira added a comment - I think that profiling is a great idea in this case.
        Hide
        Patrick Hunt added a comment -

        another interesting thing to try is run with JMX enabled and you can use jconsole to attach to the VM and get some detail about the VM/ZK during the election.

        Show
        Patrick Hunt added a comment - another interesting thing to try is run with JMX enabled and you can use jconsole to attach to the VM and get some detail about the VM/ZK during the election.
        Hide
        Flavio Junqueira added a comment -

        Is this the same issue as ZOOKEEPER-427?

        Show
        Flavio Junqueira added a comment - Is this the same issue as ZOOKEEPER-427 ?
        Hide
        Mahadev konar added a comment -

        dont know... could be... ill take a look at the logs and see if thats the problem...

        Show
        Mahadev konar added a comment - dont know... could be... ill take a look at the logs and see if thats the problem...
        Hide
        Mahadev konar added a comment -

        this issue was same as ZOOKEEPER-427. closing it as resolved.

        Show
        Mahadev konar added a comment - this issue was same as ZOOKEEPER-427 . closing it as resolved.

          People

          • Assignee:
            Mahadev konar
            Reporter:
            Flavio Junqueira
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development