Uploaded image for project: 'Apache Curator'
  1. Apache Curator
  2. CURATOR-328

PathChildrenCache fails silently if server is unavailable for sufficient time when client starts

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.1.0, 2.10.0
    • 2.11.0, 3.2.0
    • Recipes
    • None

    Description

      When initializing the PathChildrenCache, if the curator client is not yet connected to the ZooKeeper server (e.g. the server is down or the network connection is unavailable), then the internal initialization of the cache will eventually fail silently and the cache stays empty even after the client finally connects to the server and the path is populated with znodes.

      The following unit test demonstrates the problem (the unit test is ugly as the problem depends on timing, but it suffices to demonstrate the issue):

          @Test
          public void pathChildrenCacheTest() throws Exception
          {
              TestingServer server = new TestingServer(false);
      
              Timing timing = new Timing();
              CuratorFramework client = CuratorFrameworkFactory.newClient(
                      server.getConnectString(), timing.session(), timing.connection(), new ExponentialBackoffRetry(1000, 3));
              try {
                  new Thread() {
                      @Override
                      public void run()
                      {
                          try {
                              Thread.sleep(60000);
                              server.start();
                          } catch (Exception e) {
                              e.printStackTrace();
                          }
                      }
                  }.start();
      
                  client.start();
      
                  PathChildrenCache cache = new PathChildrenCache(client, "/", true);
                  cache.start();
      
                  client.blockUntilConnected();
      
                  client.create().creatingParentContainersIfNeeded().forPath("/baz", new byte[] {1,2,3});
      
                  assertNotNull("/baz does not exist", client.checkExists().forPath("/baz"));
      
                  /* Ugly hack for this test to ensure the cache got time to update itself. */
                  Thread.sleep(1000);
      
                  assertNotNull("cache doesn't see /baz", cache.getCurrentData("/baz"));
              } finally {
                  client.close();
                  server.stop();
              }
          }
      

      Here the server startup is delayed until some point after the curator client was started and after the recipe has been created. Eventually the server starts and the path is populated with data - some time is given for the cache to update itself, yet no data is visible: The second assertion fails.

      If the startup time is reduced to - say - 20 seconds, the test passes.

      If the client is allowed to first connect to the server before creating the recipe and then disconnect and reconnect after creating the recipe, then the test passes too.

      I tracked down the problem to the state change listener of the recipe: If the connection to the server is down for long enough, the refresh call during the background initialization will eventually fail (ensurePath throws an exception). This isn't a problem as the recipe has a state change listener, so it gets notified when the client eventually connects to the server. The handleStateChange method however doesn't react to a CONNECTED event - only to a RECONNECTED event. Thus if the client has been connected to the server in the past, everything works, however if this is the first time it connects, the recipe will not react to the event and thus not refresh itself.

      Attachments

        Issue Links

          Activity

            People

              randgalt Jordan Zimmerman
              behrmann Gerd Behrmann
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: