[CURATOR-595] InterProcessSemaphoreV2 LOST isn't releasing permits for other clients

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Information Provided
• Affects Version/s: 5.1.0
• Component/s: Recipes
• Assignee: Jordan Zimmerman
• Reporter: Francesco Nigro
• Created: 22/Apr/21 16:07
• Updated: 16/Jun/23 07:07
• Resolved: 16/Jun/23 07:07

Description

I'm not sure this is the right place to raise this, but I've added this test on TestInterProcessSemaphore:

    @Test
    public void testAcquireAfterLostServerOnRestart() throws Exception {
        final int sessionTimout = 4000;
        final int connectionTimout = 2000;
        try (CuratorFramework client = CuratorFrameworkFactory.newClient(server.getConnectString(), sessionTimout, connectionTimout, new RetryNTimes(0, 1))) {
            client.start();
            client.blockUntilConnected();
            final InterProcessSemaphoreV2 semaphore = new InterProcessSemaphoreV2(client, "/1", 1);
            assertNotNull(semaphore.acquire());
            CountDownLatch lost = new CountDownLatch(1);
            client.getConnectionStateListenable().addListener((client1, newState) -> {
                if (newState == ConnectionState.LOST) {
                    lost.countDown();
                }
            });
            server.stop();
            lost.await();
        }
        server.restart();
        try (CuratorFramework client = CuratorFrameworkFactory.newClient(server.getConnectString(), sessionTimout, connectionTimout, new RetryNTimes(0, 1))) {
            client.start();
            client.blockUntilConnected();
            final InterProcessSemaphoreV2 semaphore = new InterProcessSemaphoreV2(client, "/1", 1);
            final int serverTick = ZooKeeperServer.DEFAULT_TICK_TIME;
            Thread.sleep(sessionTimout + serverTick);
            assertNotNull(semaphore.acquire(0, TimeUnit.SECONDS));
        }
    }

And this is not passing: the doc of InterProcessSemaphoreV2 state that

"However, if the client session drops (crash, etc.), any leases held by the client are automatically closed and made available to other clients."

maybe I'm missing something obvious on the ZK server config instead.

Just checked out that by running on separated processes the same test:

1. start server on process A
2. start lease acquire on process B, listening for LOST events before suicide
3. restart server on Process A cause process B to suicide (as expected)
4. start lease acquire on process C, now succeed

It seems that there is something going on in the intra-process case that's not working as expected (to me, at least).

NOTE: as written in newer comments, raising the timeout doesn't seems to work too and different boxes are getting different outcomes (making this an intermittent failure).