[CURATOR-326] createContainers fails silently if client is not connected

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1.0, 2.10.0
• Fix Version/s: 2.11.0, 3.2.0
• Component/s: Framework
• Assignee: Jordan Zimmerman
• Reporter: Gerd Behrmann
• Created: 20/May/16 20:41
• Updated: 25/May/16 01:21
• Resolved: 23/May/16 03:32

Description

Calling CuratorFramework#createContainers on a non-existing path while the client is not yet connected to the server (e.g. server is down while client is starting) fails silently if enough time goes by before the server is started.

The following unit test demonstrates the issue:

package dmg.cells.zookeeper;

import org.apache.curator.framework.CuratorFramework;
import org.apache.curator.framework.CuratorFrameworkFactory;
import org.apache.curator.retry.RetryForever;
import org.apache.curator.test.TestingServer;
import org.apache.curator.test.Timing;
import org.junit.Test;

import static junit.framework.TestCase.assertNotNull;

public class CuratorTest
{
    @Test
    public void createContainersTest() throws Exception
    {
        TestingServer server = new TestingServer(false);

        Timing timing = new Timing();
        CuratorFramework client = CuratorFrameworkFactory.newClient(server.getConnectString(), timing.session(), timing.connection(), new RetryForever(100));
        try {
            new Thread() {
                @Override
                public void run()
                {
                    try {
                        Thread.sleep(30000);
                        server.start();
                    } catch (Exception e) {
                        e.printStackTrace();
                    }
                }
            }.start();

            client.start();
            client.createContainers("/this/does/not/exist");
            assertNotNull(client.checkExists().forPath("/this/does/not/exist"));
        } finally {
            client.close();
            server.stop();
        }
    }
}

The delay before starting the server is significant. If only sleeping for 10 seconds, the unit test passes. Sleeping for 30 seconds triggers a code path in Curator that will cause CuratorFramework#createContainers to wait until the server is started, yet it returns without exception and without creating the path. The assertion fails.

I tracked down the issue to ExistingBuilderImpl#pathInForeground in which the call to ZKPath#mkdirs is wrapped with a try-catch that ignores the exception. Thus the failed operation is neither retried nor propagated.

Specifically this causes silent problems with PathAndChildren cache as it uses EnsureContainer#ensure during startup to ensure that the path exists. This internally calls the above createContainers. When it fails silently, the recipe fails to register a watcher on the non-existing path and consequently the cache stays empty even when the server finally is started and the path is populated by another client.

Issue Links

• links to: GitHub Pull Request #151