When enabling Daemon-Daemon SSL with Kerberos, I noticed that the impalad is able to connect to the statestore without any problems, however, the catalog always failed in authenticating the statestore with the error message:
This caused the hang mentioned in
Our current theory is that the Catalog server tries to subscribe to the Statestore before it calls EnableSsl()(done in catalogd-main.cc) and sets up it's SSL Server socket (which is done in server->Start() where 'server' is of type 'ThriftServer').
Theoretically, since Impala does only one-way authentication in SSL/TLS, this shouldn't matter (because the Catalog connects to the Statestore as a client). However, moving the statestore subscribe code to after the SSL Server Socket is set up seems to have fixed the problem. This might be because of some internal openSSL state that's set. I'm still looking into that.
The fix currently just involves moving catalog_server.Start() after server->Start().