Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-12962

KMS key names are incorrectly encoded when creating key

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.6.0
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Target Version/s:

      Description

      Creating a key that contains special character(s) in its name will result in failure when creating, while that key is in fact created ok on the underlying key provider.

      E.g.

      $hadoop key create "key name"
      key name has not been created. java.io.IOException: HTTP status [500], exception [java.net.URISyntaxException], message [Illegal character in path at index 11: /v1/key/key name] 
      java.io.IOException: HTTP status [500], exception [java.net.URISyntaxException], message [Illegal character in path at index 11: /v1/key/key name] 
      	at org.apache.hadoop.util.HttpExceptionUtils.validateResponse(HttpExceptionUtils.java:159)
      	at org.apache.hadoop.crypto.key.kms.KMSClientProvider.call(KMSClientProvider.java:548)
      	at org.apache.hadoop.crypto.key.kms.KMSClientProvider.call(KMSClientProvider.java:506)
      	at org.apache.hadoop.crypto.key.kms.KMSClientProvider.createKeyInternal(KMSClientProvider.java:672)
      	at org.apache.hadoop.crypto.key.kms.KMSClientProvider.createKey(KMSClientProvider.java:680)
      	at org.apache.hadoop.crypto.key.KeyShell$CreateCommand.execute(KeyShell.java:483)
      	at org.apache.hadoop.crypto.key.KeyShell.run(KeyShell.java:79)
      	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
      	at org.apache.hadoop.crypto.key.KeyShell.main(KeyShell.java:515)
      
      

      but

      $ hadoop key list
      Listing keys for KeyProvider: KMSClientProvider[https://hostname:16000/kms/v1/]
      key name
      
      1. HADOOP-12962.01.patch
        5 kB
        Xiao Chen
      2. HADOOP-12962.02.patch
        5 kB
        Xiao Chen

        Activity

        Hide
        xiaochen Xiao Chen added a comment -

        Patch 1 to fix the encoding.
        Previously when creating URI, the key name string is directly concatenated, leaving the special characters unescaped. This patch uses jersey UriBuilder to do this.
        Also trivially refactored the getKeyURI so that the 2 places currently building URI share the method.

        Show
        xiaochen Xiao Chen added a comment - Patch 1 to fix the encoding. Previously when creating URI, the key name string is directly concatenated, leaving the special characters unescaped. This patch uses jersey UriBuilder to do this. Also trivially refactored the getKeyURI so that the 2 places currently building URI share the method.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 20s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 7m 59s trunk passed
        +1 compile 8m 38s trunk passed with JDK v1.8.0_74
        +1 compile 7m 43s trunk passed with JDK v1.7.0_95
        +1 checkstyle 0m 13s trunk passed
        +1 mvnsite 0m 22s trunk passed
        +1 mvneclipse 0m 15s trunk passed
        +1 findbugs 0m 27s trunk passed
        +1 javadoc 0m 14s trunk passed with JDK v1.8.0_74
        +1 javadoc 0m 14s trunk passed with JDK v1.7.0_95
        +1 mvninstall 0m 19s the patch passed
        +1 compile 8m 20s the patch passed with JDK v1.8.0_74
        +1 javac 8m 20s the patch passed
        +1 compile 7m 40s the patch passed with JDK v1.7.0_95
        +1 javac 7m 40s the patch passed
        -1 checkstyle 0m 12s hadoop-common-project/hadoop-kms: patch generated 1 new + 6 unchanged - 0 fixed = 7 total (was 6)
        +1 mvnsite 0m 20s the patch passed
        +1 mvneclipse 0m 14s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 0m 40s the patch passed
        +1 javadoc 0m 14s the patch passed with JDK v1.8.0_74
        +1 javadoc 0m 13s the patch passed with JDK v1.7.0_95
        +1 unit 1m 59s hadoop-kms in the patch passed with JDK v1.8.0_74.
        +1 unit 2m 3s hadoop-kms in the patch passed with JDK v1.7.0_95.
        +1 asflicense 0m 27s Patch does not generate ASF License warnings.
        50m 18s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:fbe3e86
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12795337/HADOOP-12962.01.patch
        JIRA Issue HADOOP-12962
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 7509811ed95b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 2e1d0ff
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/8923/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-kms.txt
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8923/testReport/
        modules C: hadoop-common-project/hadoop-kms U: hadoop-common-project/hadoop-kms
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8923/console
        Powered by Apache Yetus 0.2.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 20s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 59s trunk passed +1 compile 8m 38s trunk passed with JDK v1.8.0_74 +1 compile 7m 43s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 13s trunk passed +1 mvnsite 0m 22s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 0m 27s trunk passed +1 javadoc 0m 14s trunk passed with JDK v1.8.0_74 +1 javadoc 0m 14s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 19s the patch passed +1 compile 8m 20s the patch passed with JDK v1.8.0_74 +1 javac 8m 20s the patch passed +1 compile 7m 40s the patch passed with JDK v1.7.0_95 +1 javac 7m 40s the patch passed -1 checkstyle 0m 12s hadoop-common-project/hadoop-kms: patch generated 1 new + 6 unchanged - 0 fixed = 7 total (was 6) +1 mvnsite 0m 20s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 0m 40s the patch passed +1 javadoc 0m 14s the patch passed with JDK v1.8.0_74 +1 javadoc 0m 13s the patch passed with JDK v1.7.0_95 +1 unit 1m 59s hadoop-kms in the patch passed with JDK v1.8.0_74. +1 unit 2m 3s hadoop-kms in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 27s Patch does not generate ASF License warnings. 50m 18s Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12795337/HADOOP-12962.01.patch JIRA Issue HADOOP-12962 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 7509811ed95b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 2e1d0ff Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/8923/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-kms.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8923/testReport/ modules C: hadoop-common-project/hadoop-kms U: hadoop-common-project/hadoop-kms Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8923/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
        Hide
        xiaochen Xiao Chen added a comment -

        Patch 2 removes an unused import.
        I should also mention that other methods doesn't need to be changed, since only KMS#createKey contains the key name in the response; client-side is fine since it's via KMSClientProvider#createURL.

        Show
        xiaochen Xiao Chen added a comment - Patch 2 removes an unused import. I should also mention that other methods doesn't need to be changed, since only KMS#createKey contains the key name in the response; client-side is fine since it's via KMSClientProvider#createURL .
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 10s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 6m 53s trunk passed
        +1 compile 6m 22s trunk passed with JDK v1.8.0_74
        +1 compile 7m 3s trunk passed with JDK v1.7.0_95
        +1 checkstyle 0m 13s trunk passed
        +1 mvnsite 0m 21s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 findbugs 0m 28s trunk passed
        +1 javadoc 0m 12s trunk passed with JDK v1.8.0_74
        +1 javadoc 0m 15s trunk passed with JDK v1.7.0_95
        +1 mvninstall 0m 20s the patch passed
        +1 compile 6m 7s the patch passed with JDK v1.8.0_74
        +1 javac 6m 7s the patch passed
        +1 compile 6m 57s the patch passed with JDK v1.7.0_95
        +1 javac 6m 57s the patch passed
        +1 checkstyle 0m 14s the patch passed
        +1 mvnsite 0m 21s the patch passed
        +1 mvneclipse 0m 15s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 0m 41s the patch passed
        +1 javadoc 0m 12s the patch passed with JDK v1.8.0_74
        +1 javadoc 0m 14s the patch passed with JDK v1.7.0_95
        +1 unit 1m 30s hadoop-kms in the patch passed with JDK v1.8.0_74.
        +1 unit 1m 38s hadoop-kms in the patch passed with JDK v1.7.0_95.
        +1 asflicense 0m 22s Patch does not generate ASF License warnings.
        42m 9s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:fbe3e86
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12795368/HADOOP-12962.02.patch
        JIRA Issue HADOOP-12962
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 02886c924266 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 2c268cc
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8925/testReport/
        modules C: hadoop-common-project/hadoop-kms U: hadoop-common-project/hadoop-kms
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8925/console
        Powered by Apache Yetus 0.2.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 10s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 53s trunk passed +1 compile 6m 22s trunk passed with JDK v1.8.0_74 +1 compile 7m 3s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 13s trunk passed +1 mvnsite 0m 21s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 0m 28s trunk passed +1 javadoc 0m 12s trunk passed with JDK v1.8.0_74 +1 javadoc 0m 15s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 20s the patch passed +1 compile 6m 7s the patch passed with JDK v1.8.0_74 +1 javac 6m 7s the patch passed +1 compile 6m 57s the patch passed with JDK v1.7.0_95 +1 javac 6m 57s the patch passed +1 checkstyle 0m 14s the patch passed +1 mvnsite 0m 21s the patch passed +1 mvneclipse 0m 15s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 0m 41s the patch passed +1 javadoc 0m 12s the patch passed with JDK v1.8.0_74 +1 javadoc 0m 14s the patch passed with JDK v1.7.0_95 +1 unit 1m 30s hadoop-kms in the patch passed with JDK v1.8.0_74. +1 unit 1m 38s hadoop-kms in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 22s Patch does not generate ASF License warnings. 42m 9s Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12795368/HADOOP-12962.02.patch JIRA Issue HADOOP-12962 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 02886c924266 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 2c268cc Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8925/testReport/ modules C: hadoop-common-project/hadoop-kms U: hadoop-common-project/hadoop-kms Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8925/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
        Hide
        andrew.wang Andrew Wang added a comment -

        LGTM +1, do you mind setting the target/affects versions too? Will commit afterwards.

        I was slightly worried about special chars because of a previous key name issue at HADOOP-11311, but some googling didn't turn up anything besides the HADOOP-11311 case-sensitivity issue. Good to see JCEKS test coverage.

        Show
        andrew.wang Andrew Wang added a comment - LGTM +1, do you mind setting the target/affects versions too? Will commit afterwards. I was slightly worried about special chars because of a previous key name issue at HADOOP-11311 , but some googling didn't turn up anything besides the HADOOP-11311 case-sensitivity issue. Good to see JCEKS test coverage.
        Hide
        xiaochen Xiao Chen added a comment -

        Thanks Andrew for the comment and proactively checking!
        I agree it's recommended to not use tricky key names. If the need arises, we can have a jira similar to HADOOP-11311 to restrict it.

        I set the affects version to 2.6.0 since this bug was from HADOOP-10433 which is committed to 2.6.0. Target version I think next minor would be ok, so 2.8. Also fixed typo in the jira title s/correctly/incorrectly/g

        Show
        xiaochen Xiao Chen added a comment - Thanks Andrew for the comment and proactively checking! I agree it's recommended to not use tricky key names. If the need arises, we can have a jira similar to HADOOP-11311 to restrict it. I set the affects version to 2.6.0 since this bug was from HADOOP-10433 which is committed to 2.6.0. Target version I think next minor would be ok, so 2.8. Also fixed typo in the jira title s/correctly/incorrectly/g
        Hide
        andrew.wang Andrew Wang added a comment -

        Committed down to 2.8, thanks Xiao for the contribution!

        Show
        andrew.wang Andrew Wang added a comment - Committed down to 2.8, thanks Xiao for the contribution!
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-trunk-Commit #9501 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9501/)
        HADOOP-12962. KMS key names are incorrectly encoded when creating key. (wang: rev d4df7849a5caf749403bd89d29652f69c9c3f5a8)

        • hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMS.java
        • hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server/TestKMS.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #9501 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9501/ ) HADOOP-12962 . KMS key names are incorrectly encoded when creating key. (wang: rev d4df7849a5caf749403bd89d29652f69c9c3f5a8) hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMS.java hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server/TestKMS.java

          People

          • Assignee:
            xiaochen Xiao Chen
            Reporter:
            xiaochen Xiao Chen
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development