We need to wait for ZK-1437 upstream

Todd afaik this is not the case, you just need to wait until you see SaslAuthenticated. ZOOKEEPER-1437 fixes things in the case where you don't wait.

Todd afaik this is not the case, you just need to wait until you see SaslAuthenticated

How do we know if we need to wait, though? Some users might not be using sasl auth, in which case we don't need to wait for SaslAuthenticated.. but if we wait for it, then it may never come, right?

Good point. I've filed ZOOKEEPER-1455 and ZOOKEEPER-1456 based on your feedback.

Only workaround I could see would be to make a request to verify whether you get an auth failure or not, if you do then wait for the event. It's hacky but could be replaced with ZOOKEEPER-1455 when it becomes available. Depends on how quickly you need the fix.

I don't think there's a big rush for this – for the HA feature, shared-secret (digest) auth is a pretty reasonable.

Attached patch upgrades to ZK 3.4.5 (which includes ZOOKEEPER-1437) and adds a case to the ZK callback to avoid a FATAL error on SaslAuthenticated.

I read through the notes on ZK-1437 but didn't follow every last bit. Pat – from your perspective, the above two steps should be sufficient, right? ie we don't need to actually wait on SaslAuthenticated before performing other actions, since the patch makes the other actions automatically delay until auth is complete?

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12577300/hadoop-8315.txt
against trunk revision .

+1 @author. The patch does not contain any @author tags.

-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.

-1 javac. The patch appears to cause the build to fail.

Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2423//console

This message is automatically generated.

ZK 3.4.5's test jar is missing from the upstream maven repo. Filed ZOOKEEPER-1686

Todd Lipcon that's correct, ZOOKEEPER-1437 will re-order the messages between client and server such that the original issue will no longer happen - any client operations will be delayed until the SASL auth completes. Your patch should be sufficient.

I'll ping Mahadev konar about that test jar issues. ZOOKEEPER-1430 seems to have been updated recently to include some test jar changes are part of deploy. So it might be new (next rel?)

Refreshed the patch.

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12600266/hadoop-8315_v2.txt
against trunk revision .

+1 @author. The patch does not contain any @author tags.

-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.

-1 javac. The patch appears to cause the build to fail.

Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3025//console

This message is automatically generated.

Thanks for this fix it's possible to deploy a secured HA cluster with SASL support for the connection with Zookeeper, with a work around to configure the JAAS login for the Zookeeper client initialization like follows.

In hadoop-env.sh,

export HADOOP_ZKFC_OPTS="$HADOOP_ZKFC_OPTS -Djava.security.auth.login.config=/etc/hadoop/conf/hazk-jaas.conf"

To avoid such redundancy and the unnecessary extra login in Zookeeper, opened HDFS-5152 to address this.

Correction: the issue HDFS-5152 mentioned above was transferred to HADOOP-9938.

Patrick Hunt suggested depending on ZK 3.4.5 but with the ZK 3.4.2 test jar. Unfortunately I tried that, and the maven enforcer plugin wouldn't let me do it... and apparently the ZK folks are having trouble getting 3.4.5 test jar published in the ASF repository.

I can get the 3.4.5 artifacts including the test jar published on either Cloudera's public maven repository or in my Apache home directory. Does anyone have any preference? I'd like to close this issue out.

Todd Lipcon the 3.4.5 test jars are now being published on nexus (mirrors should be updated soon). LMK if you need anything more:
https://repository.apache.org/content/repositories/releases/org/apache/zookeeper/zookeeper/3.4.5/

Thanks Pat. I re-triggered the Jenkins build for Robert's patch above.

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12600266/hadoop-8315_v2.txt
against trunk revision .

+1 @author. The patch does not contain any @author tags.

-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

+1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

+1 contrib tests. The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3145//testReport/
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3145//console

This message is automatically generated.

+1, the latest patch looks good to me.

Committed to branch-2, branch-2.1, and trunk. I'm not sure if I set the Fix Versions right, since there's no "2.2.0" available to choose from. Feel free to update if it I got it wrong.

SUCCESS: Integrated in Hadoop-trunk-Commit #4511 (See https://builds.apache.org/job/Hadoop-trunk-Commit/4511/)
HADOOP-8315. Support SASL-authenticated ZooKeeper in ActiveStandbyElector. Contributed by Todd Lipcon (todd: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1528293)

• /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
• /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/ActiveStandbyElector.java
• /hadoop/common/trunk/hadoop-project/pom.xml

FAILURE: Integrated in Hadoop-Yarn-trunk #350 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/350/)
HADOOP-8315. Support SASL-authenticated ZooKeeper in ActiveStandbyElector. Contributed by Todd Lipcon (todd: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1528293)

• /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
• /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/ActiveStandbyElector.java
• /hadoop/common/trunk/hadoop-project/pom.xml