Hadoop Common
  1. Hadoop Common
  2. HADOOP-8315

Support SASL-authenticated ZooKeeper in ActiveStandbyElector

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: Auto Failover (HDFS-3042)
    • Fix Version/s: 3.0.0, 2.2.0
    • Component/s: auto-failover, ha
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Currently, if you try to use SASL-authenticated ZK with the ActiveStandbyElector, you run into a couple issues:
      1) We hit ZOOKEEPER-1437 - we need to wait until we see SaslAuthenticated before we can make any requests
      2) We currently throw a fatalError when we see the SaslAuthenticated callback on the connection watcher

      We need to wait for ZK-1437 upstream, and then upgrade to the fixed version for #1. For #2 we just need to add a case there and ignore it.

      1. hadoop-8315_v2.txt
        2 kB
        Robert Parker
      2. hadoop-8315.txt
        3 kB
        Todd Lipcon

        Issue Links

          Activity

          Hide
          Patrick Hunt added a comment -

          We need to wait for ZK-1437 upstream

          Todd afaik this is not the case, you just need to wait until you see SaslAuthenticated. ZOOKEEPER-1437 fixes things in the case where you don't wait.

          Show
          Patrick Hunt added a comment - We need to wait for ZK-1437 upstream Todd afaik this is not the case, you just need to wait until you see SaslAuthenticated. ZOOKEEPER-1437 fixes things in the case where you don't wait.
          Hide
          Todd Lipcon added a comment -

          Todd afaik this is not the case, you just need to wait until you see SaslAuthenticated

          How do we know if we need to wait, though? Some users might not be using sasl auth, in which case we don't need to wait for SaslAuthenticated.. but if we wait for it, then it may never come, right?

          Show
          Todd Lipcon added a comment - Todd afaik this is not the case, you just need to wait until you see SaslAuthenticated How do we know if we need to wait, though? Some users might not be using sasl auth, in which case we don't need to wait for SaslAuthenticated.. but if we wait for it, then it may never come, right?
          Hide
          Patrick Hunt added a comment -

          Good point. I've filed ZOOKEEPER-1455 and ZOOKEEPER-1456 based on your feedback.

          Only workaround I could see would be to make a request to verify whether you get an auth failure or not, if you do then wait for the event. It's hacky but could be replaced with ZOOKEEPER-1455 when it becomes available. Depends on how quickly you need the fix.

          Show
          Patrick Hunt added a comment - Good point. I've filed ZOOKEEPER-1455 and ZOOKEEPER-1456 based on your feedback. Only workaround I could see would be to make a request to verify whether you get an auth failure or not, if you do then wait for the event. It's hacky but could be replaced with ZOOKEEPER-1455 when it becomes available. Depends on how quickly you need the fix.
          Hide
          Todd Lipcon added a comment -

          I don't think there's a big rush for this – for the HA feature, shared-secret (digest) auth is a pretty reasonable.

          Show
          Todd Lipcon added a comment - I don't think there's a big rush for this – for the HA feature, shared-secret (digest) auth is a pretty reasonable.
          Hide
          Todd Lipcon added a comment -

          Attached patch upgrades to ZK 3.4.5 (which includes ZOOKEEPER-1437) and adds a case to the ZK callback to avoid a FATAL error on SaslAuthenticated.

          I read through the notes on ZK-1437 but didn't follow every last bit. Pat – from your perspective, the above two steps should be sufficient, right? ie we don't need to actually wait on SaslAuthenticated before performing other actions, since the patch makes the other actions automatically delay until auth is complete?

          Show
          Todd Lipcon added a comment - Attached patch upgrades to ZK 3.4.5 (which includes ZOOKEEPER-1437 ) and adds a case to the ZK callback to avoid a FATAL error on SaslAuthenticated. I read through the notes on ZK-1437 but didn't follow every last bit. Pat – from your perspective, the above two steps should be sufficient, right? ie we don't need to actually wait on SaslAuthenticated before performing other actions, since the patch makes the other actions automatically delay until auth is complete?
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12577300/hadoop-8315.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          -1 javac. The patch appears to cause the build to fail.

          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2423//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12577300/hadoop-8315.txt against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javac . The patch appears to cause the build to fail. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2423//console This message is automatically generated.
          Hide
          Todd Lipcon added a comment -

          ZK 3.4.5's test jar is missing from the upstream maven repo. Filed ZOOKEEPER-1686

          Show
          Todd Lipcon added a comment - ZK 3.4.5's test jar is missing from the upstream maven repo. Filed ZOOKEEPER-1686
          Hide
          Patrick Hunt added a comment -

          Todd Lipcon that's correct, ZOOKEEPER-1437 will re-order the messages between client and server such that the original issue will no longer happen - any client operations will be delayed until the SASL auth completes. Your patch should be sufficient.

          Show
          Patrick Hunt added a comment - Todd Lipcon that's correct, ZOOKEEPER-1437 will re-order the messages between client and server such that the original issue will no longer happen - any client operations will be delayed until the SASL auth completes. Your patch should be sufficient.
          Hide
          Patrick Hunt added a comment -

          I'll ping Mahadev konar about that test jar issues. ZOOKEEPER-1430 seems to have been updated recently to include some test jar changes are part of deploy. So it might be new (next rel?)

          Show
          Patrick Hunt added a comment - I'll ping Mahadev konar about that test jar issues. ZOOKEEPER-1430 seems to have been updated recently to include some test jar changes are part of deploy. So it might be new (next rel?)
          Hide
          Robert Parker added a comment -

          Refreshed the patch.

          Show
          Robert Parker added a comment - Refreshed the patch.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12600266/hadoop-8315_v2.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          -1 javac. The patch appears to cause the build to fail.

          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3025//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12600266/hadoop-8315_v2.txt against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javac . The patch appears to cause the build to fail. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3025//console This message is automatically generated.
          Hide
          Kai Zheng added a comment -

          Thanks for this fix it's possible to deploy a secured HA cluster with SASL support for the connection with Zookeeper, with a work around to configure the JAAS login for the Zookeeper client initialization like follows.

          In hadoop-env.sh,

          export HADOOP_ZKFC_OPTS="$HADOOP_ZKFC_OPTS -Djava.security.auth.login.config=/etc/hadoop/conf/hazk-jaas.conf"
          

          To avoid such redundancy and the unnecessary extra login in Zookeeper, opened HDFS-5152 to address this.

          Show
          Kai Zheng added a comment - Thanks for this fix it's possible to deploy a secured HA cluster with SASL support for the connection with Zookeeper, with a work around to configure the JAAS login for the Zookeeper client initialization like follows. In hadoop-env.sh, export HADOOP_ZKFC_OPTS= "$HADOOP_ZKFC_OPTS -Djava.security.auth.login.config=/etc/hadoop/conf/hazk-jaas.conf" To avoid such redundancy and the unnecessary extra login in Zookeeper, opened HDFS-5152 to address this.
          Hide
          Kai Zheng added a comment -

          Correction: the issue HDFS-5152 mentioned above was transferred to HADOOP-9938.

          Show
          Kai Zheng added a comment - Correction: the issue HDFS-5152 mentioned above was transferred to HADOOP-9938 .
          Hide
          Todd Lipcon added a comment -

          Patrick Hunt suggested depending on ZK 3.4.5 but with the ZK 3.4.2 test jar. Unfortunately I tried that, and the maven enforcer plugin wouldn't let me do it... and apparently the ZK folks are having trouble getting 3.4.5 test jar published in the ASF repository.

          I can get the 3.4.5 artifacts including the test jar published on either Cloudera's public maven repository or in my Apache home directory. Does anyone have any preference? I'd like to close this issue out.

          Show
          Todd Lipcon added a comment - Patrick Hunt suggested depending on ZK 3.4.5 but with the ZK 3.4.2 test jar. Unfortunately I tried that, and the maven enforcer plugin wouldn't let me do it... and apparently the ZK folks are having trouble getting 3.4.5 test jar published in the ASF repository. I can get the 3.4.5 artifacts including the test jar published on either Cloudera's public maven repository or in my Apache home directory. Does anyone have any preference? I'd like to close this issue out.
          Hide
          Patrick Hunt added a comment -

          Todd Lipcon the 3.4.5 test jars are now being published on nexus (mirrors should be updated soon). LMK if you need anything more:
          https://repository.apache.org/content/repositories/releases/org/apache/zookeeper/zookeeper/3.4.5/

          Show
          Patrick Hunt added a comment - Todd Lipcon the 3.4.5 test jars are now being published on nexus (mirrors should be updated soon). LMK if you need anything more: https://repository.apache.org/content/repositories/releases/org/apache/zookeeper/zookeeper/3.4.5/
          Hide
          Todd Lipcon added a comment -

          Thanks Pat. I re-triggered the Jenkins build for Robert's patch above.

          Show
          Todd Lipcon added a comment - Thanks Pat. I re-triggered the Jenkins build for Robert's patch above.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12600266/hadoop-8315_v2.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3145//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3145//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12600266/hadoop-8315_v2.txt against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3145//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3145//console This message is automatically generated.
          Hide
          Aaron T. Myers added a comment -

          +1, the latest patch looks good to me.

          Show
          Aaron T. Myers added a comment - +1, the latest patch looks good to me.
          Hide
          Todd Lipcon added a comment -

          Committed to branch-2, branch-2.1, and trunk. I'm not sure if I set the Fix Versions right, since there's no "2.2.0" available to choose from. Feel free to update if it I got it wrong.

          Show
          Todd Lipcon added a comment - Committed to branch-2, branch-2.1, and trunk. I'm not sure if I set the Fix Versions right, since there's no "2.2.0" available to choose from. Feel free to update if it I got it wrong.
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #4511 (See https://builds.apache.org/job/Hadoop-trunk-Commit/4511/)
          HADOOP-8315. Support SASL-authenticated ZooKeeper in ActiveStandbyElector. Contributed by Todd Lipcon (todd: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1528293)

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/ActiveStandbyElector.java
          • /hadoop/common/trunk/hadoop-project/pom.xml
          Show
          Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #4511 (See https://builds.apache.org/job/Hadoop-trunk-Commit/4511/ ) HADOOP-8315 . Support SASL-authenticated ZooKeeper in ActiveStandbyElector. Contributed by Todd Lipcon (todd: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1528293 ) /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/ActiveStandbyElector.java /hadoop/common/trunk/hadoop-project/pom.xml
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #350 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/350/)
          HADOOP-8315. Support SASL-authenticated ZooKeeper in ActiveStandbyElector. Contributed by Todd Lipcon (todd: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1528293)

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/ActiveStandbyElector.java
          • /hadoop/common/trunk/hadoop-project/pom.xml
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #350 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/350/ ) HADOOP-8315 . Support SASL-authenticated ZooKeeper in ActiveStandbyElector. Contributed by Todd Lipcon (todd: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1528293 ) /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/ActiveStandbyElector.java /hadoop/common/trunk/hadoop-project/pom.xml

            People

            • Assignee:
              Todd Lipcon
              Reporter:
              Todd Lipcon
            • Votes:
              0 Vote for this issue
              Watchers:
              17 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development