[ZOOKEEPER-2383] Startup race in ZooKeeperServer

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 3.4.8
• Fix Version/s: 3.4.10, 3.5.3, 3.6.0
• Component/s: jmx, server
• Assignee: Rakesh Radhakrishnan
• Reporter: Steven Rowe
• Created: 08/Mar/16 17:00
• Updated: 31/Mar/17 09:01
• Resolved: 03/Jan/17 17:54

Description

In attempting to upgrade Solr's ZooKeeper dependency from 3.4.6 to 3.4.8 (SOLR-8724) I ran into test failures where attempts to create a node in a newly started standalone ZooKeeperServer were failing because of an assertion in MBeanRegistry.

ZooKeeperServer.startup() first sets up its request processor chain then registers itself in JMX, but if a connection comes in before the server's JMX registration happens, registration of the connection will fail because it trips the assertion that (effectively) its parent (the server) has already registered itself.

ZooKeeperServer.java

    public synchronized void startup() {
        if (sessionTracker == null) {
            createSessionTracker();
        }
        startSessionTracker();
        setupRequestProcessors();

        registerJMX();

        state = State.RUNNING;
        notifyAll();
    }

MBeanRegistry.java

    public void register(ZKMBeanInfo bean, ZKMBeanInfo parent)
        throws JMException
    {
        assert bean != null;
        String path = null;
        if (parent != null) {
            path = mapBean2Path.get(parent);
            assert path != null;
        }

This problem appears to be new with ZK 3.4.8 - AFAIK Solr never had this issue with ZK 3.4.6.

Issue Links

• is related to: ARIES-1684 Upgrade ZooKeeper to 3.4.10
• is related to: SOLR-9386 Upgrade Zookeeper to 3.4.10
• relates to: SOLR-8724 Upgrade Zookeeper to 3.4.8
• relates to: ZOOKEEPER-2655 Improve NIOServerCnxn#isZKServerRunning to reflect the semantics correctly
• links to: GitHub Pull Request #101
• links to: GitHub Pull Request #135