Uploaded image for project: 'ZooKeeper'
  1. ZooKeeper
  2. ZOOKEEPER-885

Zookeeper drops connections under moderate IO load

    Details

    • Type: Bug
    • Status: Open
    • Priority: Critical
    • Resolution: Unresolved
    • Affects Version/s: 3.2.2, 3.3.1
    • Fix Version/s: 3.6.0, 3.5.5
    • Component/s: server
    • Labels:
      None
    • Environment:

      Debian (Lenny)
      1Gb RAM
      swap disabled
      100Mb heap for zookeeper

      Description

      A zookeeper server under minimum load, with a number of clients watching exactly one node will fail to maintain the connection when the machine is subjected to moderate IO load.

      In a specific test example we had three zookeeper servers running on dedicated machines with 45 clients connected, watching exactly one node. The clients would disconnect after moderate load was added to each of the zookeeper servers with the command:

      dd if=/dev/urandom of=/dev/mapper/nimbula-test
      

      The dd command transferred data at a rate of about 4Mb/s.

      The same thing happens with

      dd if=/dev/zero of=/dev/mapper/nimbula-test
      

      It seems strange that such a moderate load should cause instability in the connection.

      Very few other processes were running, the machines were setup to test the connection instability we have experienced. Clients performed no other read or mutation operations.

      Although the documents state that minimal competing IO load should present on the zookeeper server, it seems reasonable that moderate IO should not cause problems in this case.

        Attachments

        1. benchmark.csv
          24 kB
          Alexandre Hardy
        2. tracezklogs.tar.gz
          98 kB
          Alexandre Hardy
        3. tracezklogs.tar.gz
          18 kB
          Alexandre Hardy
        4. WatcherTest.java
          2 kB
          Alexandre Hardy
        5. zklogs.tar.gz
          64 kB
          Alexandre Hardy

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              ahardy Alexandre Hardy
            • Votes:
              3 Vote for this issue
              Watchers:
              15 Start watching this issue

              Dates

              • Created:
                Updated: