Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13190

Mention LoadBalancingKMSClientProvider in KMS HA documentation

    Details

    • Hadoop Flags:
      Reviewed

      Description

      Currently, there are two ways to achieve KMS HA.

      The first one, and the only documented one, is running multiple KMS instances behind a load balancer. https://hadoop.apache.org/docs/stable/hadoop-kms/index.html

      The other way, is make use of LoadBalancingKMSClientProvider which is added in HADOOP-11620. However the usage is undocumented.

      I think we should update the KMS document to introduce LoadBalancingKMSClientProvider, provide examples, and also update kms-site.xml to explain it.

      1. HADOOP-13190.001.patch
        2 kB
        Wei-Chiu Chuang
      2. HADOOP-13190.002.patch
        4 kB
        Wei-Chiu Chuang
      3. HADOOP-13190.003.patch
        4 kB
        Wei-Chiu Chuang
      4. HADOOP-13190.004.patch
        4 kB
        Wei-Chiu Chuang

        Issue Links

          Activity

          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          v01: added a section to describe configuration for HA in this alternative set up.

          Show
          jojochuang Wei-Chiu Chuang added a comment - v01: added a section to describe configuration for HA in this alternative set up.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 10s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 mvninstall 7m 10s trunk passed
          +1 mvnsite 0m 20s trunk passed
          +1 mvnsite 0m 20s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 asflicense 0m 14s The patch does not generate ASF License warnings.
          8m 27s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:2c91fd8
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12805701/HADOOP-13190.001.patch
          JIRA Issue HADOOP-13190
          Optional Tests asflicense mvnsite
          uname Linux 6bcf3f462e30 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / ac95448
          modules C: hadoop-common-project/hadoop-kms U: hadoop-common-project/hadoop-kms
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9559/console
          Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 10s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 mvninstall 7m 10s trunk passed +1 mvnsite 0m 20s trunk passed +1 mvnsite 0m 20s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 asflicense 0m 14s The patch does not generate ASF License warnings. 8m 27s Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12805701/HADOOP-13190.001.patch JIRA Issue HADOOP-13190 Optional Tests asflicense mvnsite uname Linux 6bcf3f462e30 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / ac95448 modules C: hadoop-common-project/hadoop-kms U: hadoop-common-project/hadoop-kms Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9559/console Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xiaochen Xiao Chen added a comment -

          Thank you Wei-Chiu Chuang for creating this jira and posting a patch!

          We should definitely document this, and overall looks good. Comments:

          • Maybe we can change the current $H3 level title to Using Multiple Instances of KMS, and list the current LB/VIP and the new LBKMSCP under it? The other sub-sections (kerberos, secret-sharing) applies to multiple instances in general.
          • In the new LBKMSCP section, please also add the failure-handling behavior. If a request to a KMSCP failed, LBKMSCP will retry the next KMSCP. The request is returned as failure only if all KMSCPs failed.
          • In the sample xml, maybe also list an http example?
          Show
          xiaochen Xiao Chen added a comment - Thank you Wei-Chiu Chuang for creating this jira and posting a patch! We should definitely document this, and overall looks good. Comments: Maybe we can change the current $H3 level title to Using Multiple Instances of KMS , and list the current LB/VIP and the new LBKMSCP under it? The other sub-sections (kerberos, secret-sharing) applies to multiple instances in general. In the new LBKMSCP section, please also add the failure-handling behavior. If a request to a KMSCP failed, LBKMSCP will retry the next KMSCP. The request is returned as failure only if all KMSCPs failed. In the sample xml, maybe also list an http example?
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 9s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 mvninstall 7m 43s trunk passed
          +1 mvnsite 0m 18s trunk passed
          +1 mvnsite 0m 15s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 asflicense 0m 17s The patch does not generate ASF License warnings.
          9m 1s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12805701/HADOOP-13190.001.patch
          JIRA Issue HADOOP-13190
          Optional Tests asflicense mvnsite
          uname Linux fbf55e6bf39e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 6255859
          modules C: hadoop-common-project/hadoop-kms U: hadoop-common-project/hadoop-kms
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10203/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 9s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 mvninstall 7m 43s trunk passed +1 mvnsite 0m 18s trunk passed +1 mvnsite 0m 15s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 9m 1s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12805701/HADOOP-13190.001.patch JIRA Issue HADOOP-13190 Optional Tests asflicense mvnsite uname Linux fbf55e6bf39e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 6255859 modules C: hadoop-common-project/hadoop-kms U: hadoop-common-project/hadoop-kms Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10203/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          Thanks Xiao Chen for reviewing the patch.

          Maybe we can change the current $H3 level title to Using Multiple Instances of KMS, and list the current LB/VIP and the new LBKMSCP under it? The other sub-sections (kerberos, secret-sharing) applies to multiple instances in general.

          Good idea.

          In the new LBKMSCP section, please also add the failure-handling behavior. If a request to a KMSCP failed, LBKMSCP will retry the next KMSCP. The request is returned as failure only if all KMSCPs failed.

          Done.

          In the sample xml, maybe also list an http example?

          Not sure how to best capture this. I found the section KMS Client Configuration is vague, so I put up an example of configuring NameNode as a KMS client here.

          Show
          jojochuang Wei-Chiu Chuang added a comment - Thanks Xiao Chen for reviewing the patch. Maybe we can change the current $H3 level title to Using Multiple Instances of KMS, and list the current LB/VIP and the new LBKMSCP under it? The other sub-sections (kerberos, secret-sharing) applies to multiple instances in general. Good idea. In the new LBKMSCP section, please also add the failure-handling behavior. If a request to a KMSCP failed, LBKMSCP will retry the next KMSCP. The request is returned as failure only if all KMSCPs failed. Done. In the sample xml, maybe also list an http example? Not sure how to best capture this. I found the section KMS Client Configuration is vague, so I put up an example of configuring NameNode as a KMS client here.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 mvninstall 9m 10s trunk passed
          +1 mvnsite 0m 20s trunk passed
          +1 mvnsite 0m 17s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 asflicense 0m 22s The patch does not generate ASF License warnings.
          10m 48s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12822700/HADOOP-13190.002.patch
          JIRA Issue HADOOP-13190
          Optional Tests asflicense mvnsite
          uname Linux 0b0eecd6fd49 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 0705489
          modules C: hadoop-common-project/hadoop-kms U: hadoop-common-project/hadoop-kms
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10206/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 mvninstall 9m 10s trunk passed +1 mvnsite 0m 20s trunk passed +1 mvnsite 0m 17s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 asflicense 0m 22s The patch does not generate ASF License warnings. 10m 48s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12822700/HADOOP-13190.002.patch JIRA Issue HADOOP-13190 Optional Tests asflicense mvnsite uname Linux 0b0eecd6fd49 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 0705489 modules C: hadoop-common-project/hadoop-kms U: hadoop-common-project/hadoop-kms Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10206/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks Wei-Chiu Chuang for revving! Looks pretty good, and I think we're close. Nice catch on the macro at the beginning BTW.

          Nits:

          • Seems a typo in KMS Client Configuration session. 'mustbe'.
          • {{...(for example, a NameNode) }} -> {{(for example, HDFS NameNode) }}
          • The host names -> hostnames
          • For example, the following configuration in hdfs-site.xml sets up two KMS instances. Technically they don't 'set up' the 2, since they're client side. How about we s/sets up/configures/g ?
          • Suggest we add 1 sentence to describe how LBKMSCP is used. Something like: When more than one key provider is configured in the uri, a LBKMSCP is automatically created. We can combine this with the intro about round-robin.
          Show
          xiaochen Xiao Chen added a comment - Thanks Wei-Chiu Chuang for revving! Looks pretty good, and I think we're close. Nice catch on the macro at the beginning BTW. Nits: Seems a typo in KMS Client Configuration session. 'mustbe'. {{...(for example, a NameNode) }} -> {{(for example, HDFS NameNode) }} The host names -> hostnames For example, the following configuration in hdfs-site.xml sets up two KMS instances . Technically they don't 'set up' the 2, since they're client side. How about we s/sets up/configures/g ? Suggest we add 1 sentence to describe how LBKMSCP is used. Something like: When more than one key provider is configured in the uri, a LBKMSCP is automatically created. We can combine this with the intro about round-robin.
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          Thanks again Xiao Chen!

          Attach v03 patch to address typos, rephrases, and etc.

          Show
          jojochuang Wei-Chiu Chuang added a comment - Thanks again Xiao Chen ! Attach v03 patch to address typos, rephrases, and etc.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 13s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 mvninstall 6m 32s trunk passed
          +1 mvnsite 0m 17s trunk passed
          +1 mvnsite 0m 15s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 asflicense 0m 15s The patch does not generate ASF License warnings.
          7m 52s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12822808/HADOOP-13190.003.patch
          JIRA Issue HADOOP-13190
          Optional Tests asflicense mvnsite
          uname Linux 090104aeffc4 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 522ddbd
          modules C: hadoop-common-project/hadoop-kms U: hadoop-common-project/hadoop-kms
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10211/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 mvninstall 6m 32s trunk passed +1 mvnsite 0m 17s trunk passed +1 mvnsite 0m 15s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 asflicense 0m 15s The patch does not generate ASF License warnings. 7m 52s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12822808/HADOOP-13190.003.patch JIRA Issue HADOOP-13190 Optional Tests asflicense mvnsite uname Linux 090104aeffc4 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 522ddbd modules C: hadoop-common-project/hadoop-kms U: hadoop-common-project/hadoop-kms Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10211/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks Wei-Chiu. Patch 3 looks great, +1 pending the following nitter nits:

          • requests from the same user may be handled by different KMS instances s/user/client/g
          • clients are unaware there are multiple KMS instances. s/there are multiple KMS instances/of multiple KMS instances at the server-side/g
          • In the LBKMSCP example, please use 9600 as the port instead of 16000.
          Show
          xiaochen Xiao Chen added a comment - Thanks Wei-Chiu. Patch 3 looks great, +1 pending the following nitter nits: requests from the same user may be handled by different KMS instances s/user/client/g clients are unaware there are multiple KMS instances. s/there are multiple KMS instances/of multiple KMS instances at the server-side/g In the LBKMSCP example, please use 9600 as the port instead of 16000.
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          Thanks Xiao Chen again for the follow-up review. Attaching v04 patch for review to address them.

          Show
          jojochuang Wei-Chiu Chuang added a comment - Thanks Xiao Chen again for the follow-up review. Attaching v04 patch for review to address them.
          Hide
          xiaochen Xiao Chen added a comment -

          +1 pending jenkins, thanks Wei-Chiu!

          Show
          xiaochen Xiao Chen added a comment - +1 pending jenkins, thanks Wei-Chiu!
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 13s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 mvninstall 8m 12s trunk passed
          +1 mvnsite 0m 20s trunk passed
          +1 mvnsite 0m 18s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 asflicense 0m 16s The patch does not generate ASF License warnings.
          9m 37s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12822900/HADOOP-13190.004.patch
          JIRA Issue HADOOP-13190
          Optional Tests asflicense mvnsite
          uname Linux 5387d532910d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / cc48251
          modules C: hadoop-common-project/hadoop-kms U: hadoop-common-project/hadoop-kms
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10214/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 mvninstall 8m 12s trunk passed +1 mvnsite 0m 20s trunk passed +1 mvnsite 0m 18s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 asflicense 0m 16s The patch does not generate ASF License warnings. 9m 37s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12822900/HADOOP-13190.004.patch JIRA Issue HADOOP-13190 Optional Tests asflicense mvnsite uname Linux 5387d532910d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / cc48251 modules C: hadoop-common-project/hadoop-kms U: hadoop-common-project/hadoop-kms Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10214/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          Committing this.

          Show
          jojochuang Wei-Chiu Chuang added a comment - Committing this.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #10264 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10264/)
          HADOOP-13190. Mention LoadBalancingKMSClientProvider in KMS HA (weichiu: rev db719ef125b11b01eab3353e2dc4b48992bf88d5)

          • hadoop-common-project/hadoop-kms/src/site/markdown/index.md.vm
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #10264 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10264/ ) HADOOP-13190 . Mention LoadBalancingKMSClientProvider in KMS HA (weichiu: rev db719ef125b11b01eab3353e2dc4b48992bf88d5) hadoop-common-project/hadoop-kms/src/site/markdown/index.md.vm
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          Committed to trunk, branch-2 and branch-2.8. Thanks Xiao Chen for review!

          Show
          jojochuang Wei-Chiu Chuang added a comment - Committed to trunk, branch-2 and branch-2.8. Thanks Xiao Chen for review!

            People

            • Assignee:
              jojochuang Wei-Chiu Chuang
              Reporter:
              jojochuang Wei-Chiu Chuang
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development