Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-12811

Change kms server port number which conflicts with HMaster port number

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 2.7.2
    • Fix Version/s: 3.0.0-alpha1
    • Component/s: kms
    • Target Version/s:
    • Hadoop Flags:
      Incompatible change
    • Release Note:
      The default port for KMS service is now 9600. This is to avoid conflicts on the previous port 16000, which is also used by HMaster as the default port.
    • Flags:
      Patch

      Description

      The HBase's HMaster port number conflicts with Hadoop kms port number. Both uses 16000.

      There might be use cases user need kms and HBase present on the same cluster. The HBase is able to encrypt its HFiles but user might need KMS to encrypt other HDFS directories.

      Users would have to manually override the default port of either application on their cluster. It would be nice to have different default ports so kms and HBase could naturally coexist.

        Issue Links

          Activity

          Hide
          xiaochen Xiao Chen added a comment -

          Thanks a lot Andrew Wang!
          I updated the release note, please let me know if you have any suggestions.

          Show
          xiaochen Xiao Chen added a comment - Thanks a lot Andrew Wang ! I updated the release note, please let me know if you have any suggestions.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #9614 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9614/)
          HADOOP-12811. Change kms server port number which conflicts with HMaster (wang: rev a74580a4d3039ff95e7744f1d7a386b2bc7a7484)

          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/crypto/key/kms/TestLoadBalancingKMSClientProvider.java
          • hadoop-common-project/hadoop-kms/src/main/libexec/kms-config.sh
          • hadoop-common-project/hadoop-kms/src/site/markdown/index.md.vm
          • hadoop-common-project/hadoop-kms/src/main/conf/kms-env.sh
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9614 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9614/ ) HADOOP-12811 . Change kms server port number which conflicts with HMaster (wang: rev a74580a4d3039ff95e7744f1d7a386b2bc7a7484) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/crypto/key/kms/TestLoadBalancingKMSClientProvider.java hadoop-common-project/hadoop-kms/src/main/libexec/kms-config.sh hadoop-common-project/hadoop-kms/src/site/markdown/index.md.vm hadoop-common-project/hadoop-kms/src/main/conf/kms-env.sh
          Hide
          andrew.wang Andrew Wang added a comment -

          Committed to trunk, thanks for working on this Xiao!

          Show
          andrew.wang Andrew Wang added a comment - Committed to trunk, thanks for working on this Xiao!
          Hide
          andrew.wang Andrew Wang added a comment -

          LGTM +1, will commit shortly.

          Xiao Chen do you mind adding a release note? Good to have for incompatible changes.

          Show
          andrew.wang Andrew Wang added a comment - LGTM +1, will commit shortly. Xiao Chen do you mind adding a release note? Good to have for incompatible changes.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 20s Docker mode activated.
          0 shelldocs 0m 0s Shelldocs was not available.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 11s Maven dependency ordering for branch
          +1 mvninstall 8m 53s trunk passed
          +1 compile 7m 51s trunk passed with JDK v1.8.0_77
          +1 compile 8m 10s trunk passed with JDK v1.7.0_95
          +1 checkstyle 0m 30s trunk passed
          +1 mvnsite 1m 33s trunk passed
          +1 mvneclipse 0m 31s trunk passed
          +1 findbugs 2m 20s trunk passed
          +1 javadoc 1m 16s trunk passed with JDK v1.8.0_77
          +1 javadoc 1m 31s trunk passed with JDK v1.7.0_95
          0 mvndep 0m 10s Maven dependency ordering for patch
          +1 mvninstall 1m 11s the patch passed
          +1 compile 8m 10s the patch passed with JDK v1.8.0_77
          +1 javac 8m 10s the patch passed
          +1 compile 8m 33s the patch passed with JDK v1.7.0_95
          +1 javac 8m 33s the patch passed
          +1 checkstyle 0m 29s the patch passed
          +1 mvnsite 1m 37s the patch passed
          +1 mvneclipse 0m 33s the patch passed
          +1 shellcheck 0m 13s There were no new shellcheck issues.
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 3m 3s the patch passed
          +1 javadoc 1m 19s the patch passed with JDK v1.8.0_77
          +1 javadoc 1m 36s the patch passed with JDK v1.7.0_95
          +1 unit 8m 58s hadoop-common in the patch passed with JDK v1.8.0_77.
          +1 unit 2m 13s hadoop-kms in the patch passed with JDK v1.8.0_77.
          +1 unit 9m 48s hadoop-common in the patch passed with JDK v1.7.0_95.
          +1 unit 2m 27s hadoop-kms in the patch passed with JDK v1.7.0_95.
          +1 asflicense 0m 31s Patch does not generate ASF License warnings.
          85m 31s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:fbe3e86
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12798544/HADOOP-12811.01.patch
          JIRA Issue HADOOP-12811
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle shellcheck shelldocs
          uname Linux f067ad5f97c8 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 27b131e
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          shellcheck v0.4.3
          findbugs v3.0.0
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9089/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms U: hadoop-common-project
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9089/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 20s Docker mode activated. 0 shelldocs 0m 0s Shelldocs was not available. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 11s Maven dependency ordering for branch +1 mvninstall 8m 53s trunk passed +1 compile 7m 51s trunk passed with JDK v1.8.0_77 +1 compile 8m 10s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 30s trunk passed +1 mvnsite 1m 33s trunk passed +1 mvneclipse 0m 31s trunk passed +1 findbugs 2m 20s trunk passed +1 javadoc 1m 16s trunk passed with JDK v1.8.0_77 +1 javadoc 1m 31s trunk passed with JDK v1.7.0_95 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 1m 11s the patch passed +1 compile 8m 10s the patch passed with JDK v1.8.0_77 +1 javac 8m 10s the patch passed +1 compile 8m 33s the patch passed with JDK v1.7.0_95 +1 javac 8m 33s the patch passed +1 checkstyle 0m 29s the patch passed +1 mvnsite 1m 37s the patch passed +1 mvneclipse 0m 33s the patch passed +1 shellcheck 0m 13s There were no new shellcheck issues. +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 3m 3s the patch passed +1 javadoc 1m 19s the patch passed with JDK v1.8.0_77 +1 javadoc 1m 36s the patch passed with JDK v1.7.0_95 +1 unit 8m 58s hadoop-common in the patch passed with JDK v1.8.0_77. +1 unit 2m 13s hadoop-kms in the patch passed with JDK v1.8.0_77. +1 unit 9m 48s hadoop-common in the patch passed with JDK v1.7.0_95. +1 unit 2m 27s hadoop-kms in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 31s Patch does not generate ASF License warnings. 85m 31s Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12798544/HADOOP-12811.01.patch JIRA Issue HADOOP-12811 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle shellcheck shelldocs uname Linux f067ad5f97c8 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 27b131e Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 shellcheck v0.4.3 findbugs v3.0.0 JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9089/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-common-project/hadoop-kms U: hadoop-common-project Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9089/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          xiaochen Xiao Chen added a comment -

          HDFS-9427 seems to be close to commit. I'm attaching a patch for review. KMS port is changed from 16000 to 9600. Thanks.

          I picked 9600 because it's in the same range as HDFS-9427. I also searched apache github, 9600 seems to not being used by other applications. (except juddi, which I feel should be Okay.)

          Also searched across projects for 16000, and found impala should update some files due to this change. But that should be implicit by the 'incompatible' flag anyway.

          Show
          xiaochen Xiao Chen added a comment - HDFS-9427 seems to be close to commit. I'm attaching a patch for review. KMS port is changed from 16000 to 9600. Thanks. I picked 9600 because it's in the same range as HDFS-9427 . I also searched apache github, 9600 seems to not being used by other applications. (except juddi , which I feel should be Okay.) Also searched across projects for 16000, and found impala should update some files due to this change. But that should be implicit by the 'incompatible' flag anyway.
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks Yufeng for creating this. I've linked HDFS-9427 which will change HDFS default ports.

          I'd like to work on this, but I'm thinking to wait for HDFS-9427 to be agreed and committed to avoid conflicts. Then I plan to change the default kms port to be in the same range (e,g, 9170 if HDFS-9427 goes with 9070).

          The only problem I can think of this change is backwards compatibility. Added a label to this jira.

          Show
          xiaochen Xiao Chen added a comment - Thanks Yufeng for creating this. I've linked HDFS-9427 which will change HDFS default ports. I'd like to work on this, but I'm thinking to wait for HDFS-9427 to be agreed and committed to avoid conflicts. Then I plan to change the default kms port to be in the same range (e,g, 9170 if HDFS-9427 goes with 9070). The only problem I can think of this change is backwards compatibility. Added a label to this jira.
          Hide
          jmhsieh Jonathan Hsieh added a comment -

          I'll suggest moving this into the 9xxx range where the rest of the hadoop services are being moved to by HDFS-9427.

          Show
          jmhsieh Jonathan Hsieh added a comment - I'll suggest moving this into the 9xxx range where the rest of the hadoop services are being moved to by HDFS-9427 .
          Hide
          yufengj Yufeng Jiang added a comment -

          Stating from HBase 0.99, HMaster's port number was set to 16000 to avoid the conflict with linux ephemeral port. At roughly same time, hadoop kms was released. As pointed out by jmhsieh@

          "Timeline wise, the changes were released at roughly same time – This change was in a released by hbase in 30/oct/2014 [1] and in hadoop 18/nov/2014 [2].
          [1] https://archive.apache.org/dist/hbase/hbase-0.99.0/
          [2] http://hadoop.apache.org/releases.html
          "

          Show
          yufengj Yufeng Jiang added a comment - Stating from HBase 0.99, HMaster's port number was set to 16000 to avoid the conflict with linux ephemeral port. At roughly same time, hadoop kms was released. As pointed out by jmhsieh@ "Timeline wise, the changes were released at roughly same time – This change was in a released by hbase in 30/oct/2014 [1] and in hadoop 18/nov/2014 [2] . [1] https://archive.apache.org/dist/hbase/hbase-0.99.0/ [2] http://hadoop.apache.org/releases.html "

            People

            • Assignee:
              xiaochen Xiao Chen
              Reporter:
              yufengj Yufeng Jiang
            • Votes:
              1 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development