[CURATOR-659] Error log message when closing LeaderSelector

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 5.1.0
• Component/s: Recipes
• Assignee: Unassigned
• Reporter: Christophe Bornet
• Created: 16/Dec/22 15:45
• Updated: 16/Dec/22 15:45

Description

When closing the client quickly after closing a LeaderSelector, we get an annoying error log message because the client is in state STOPPED when mutex.release() is called in LeaderSelector:

16:29:50.451 [Curator-LeaderSelector-0:org.apache.curator.framework.recipes.leader.LeaderSelector@464] ERROR org.apache.curator.framework.recipes.leader.LeaderSelector - The leader threw an exception
java.lang.IllegalStateException: Expected state [STARTED] was [STOPPED]
	at org.apache.curator.shaded.com.google.common.base.Preconditions.checkState(Preconditions.java:823) ~[curator-client-5.1.0.jar:?]
	at org.apache.curator.framework.imps.CuratorFrameworkImpl.checkState(CuratorFrameworkImpl.java:423) ~[curator-framework-5.1.0.jar:5.1.0]
	at org.apache.curator.framework.imps.CuratorFrameworkImpl.delete(CuratorFrameworkImpl.java:443) ~[curator-framework-5.1.0.jar:5.1.0]
	at org.apache.curator.framework.recipes.locks.LockInternals.deleteOurPath(LockInternals.java:347) ~[curator-recipes-5.1.0.jar:5.1.0]
	at org.apache.curator.framework.recipes.locks.LockInternals.releaseLock(LockInternals.java:124) ~[curator-recipes-5.1.0.jar:5.1.0]
	at org.apache.curator.framework.recipes.locks.InterProcessMutex.release(InterProcessMutex.java:154) ~[curator-recipes-5.1.0.jar:5.1.0]
	at org.apache.curator.framework.recipes.leader.LeaderSelector.doWork(LeaderSelector.java:454) [curator-recipes-5.1.0.jar:5.1.0]
	at org.apache.curator.framework.recipes.leader.LeaderSelector.doWorkLoop(LeaderSelector.java:483) [curator-recipes-5.1.0.jar:5.1.0]
	at org.apache.curator.framework.recipes.leader.LeaderSelector.access$100(LeaderSelector.java:66) [curator-recipes-5.1.0.jar:5.1.0]
	at org.apache.curator.framework.recipes.leader.LeaderSelector$2.call(LeaderSelector.java:247) [curator-recipes-5.1.0.jar:5.1.0]
	at org.apache.curator.framework.recipes.leader.LeaderSelector$2.call(LeaderSelector.java:241) [curator-recipes-5.1.0.jar:5.1.0]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]

Here is a short code sample to reproduce the issue (change the connectString to your ZK cluster):

  @Test
  void curatorTest() throws Exception {
    RetryPolicy retryPolicy = new ExponentialBackoffRetry(1000, 3);
    CuratorFramework client =
        CuratorFrameworkFactory.builder()
            .connectString(config.getConfigurationMetadataStoreUrl())
            .sessionTimeoutMs(5000)
            .connectionTimeoutMs(5000)
            .retryPolicy(retryPolicy)
            .build();
    client.start();

    TestLeaderSelectorListener selectorListenerAdapter = new TestLeaderSelectorListener(client);

    Thread.sleep(1000);
    selectorListenerAdapter.close();
    client.close();
  }

  public static class TestLeaderSelectorListener extends LeaderSelectorListenerAdapter implements Closeable {

    private final LeaderSelector leaderSelector;

    public TestLeaderSelectorListener(CuratorFramework client) {
      leaderSelector = new LeaderSelector(client, "/test", this);
      leaderSelector.autoRequeue();
      leaderSelector.start();
    }

    @Override
    public void takeLeadership(CuratorFramework client) {
      try {
        Thread.sleep(5000);
      } catch (InterruptedException e) {
        Thread.currentThread().interrupt();
      }
    }

    @Override
    public void close() throws IOException {
      leaderSelector.close();
    }
  }

Can this be detected and the log not produce ? It makes the shutting down unclean.