Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13030

Handle special characters in passwords in KMS startup script

    Details

    • Target Version/s:

      Description

      kms.sh currently cannot handle special characters.

       sed -e 's/_kms_ssl_keystore_pass_/'${KMS_SSL_KEYSTORE_PASS}'/g' \
              -e 's/_kms_ssl_truststore_pass_/'${KMS_SSL_TRUSTSTORE_PASS}'/g' \
              "${HADOOP_CATALINA_HOME}/conf/ssl-server.xml.conf" \
              > "${HADOOP_CATALINA_HOME}/conf/ssl-server.xml"
      
      1. HADOOP-13030.01.patch
        2 kB
        Xiao Chen
      2. HADOOP-13030.02.patch
        2 kB
        Xiao Chen
      3. HADOOP-13030.03.patch
        2 kB
        Xiao Chen
      4. HADOOP-13030.b28.patch
        2 kB
        Xiao Chen
      5. HADOOP-13030-repro.tar.gz
        1 kB
        Xiao Chen

        Issue Links

          Activity

          Hide
          xiaochen Xiao Chen added a comment -

          Thanks Allen Wittenauer for pointing this out, I wasn't aware of that..
          I created HADOOP-13077 to fix it and attached a patch, let's move the following work there.

          Show
          xiaochen Xiao Chen added a comment - Thanks Allen Wittenauer for pointing this out, I wasn't aware of that.. I created HADOOP-13077 to fix it and attached a patch, let's move the following work there.
          Hide
          aw Allen Wittenauer added a comment -

          Pretty much this exact same code exists in httpfs.sh ....

          Show
          aw Allen Wittenauer added a comment - Pretty much this exact same code exists in httpfs.sh ....
          Hide
          andrew.wang Andrew Wang added a comment -

          Backport patches look good, committed to branch-2 and branch-2.8. Thank you Xiao for the contribution!

          Show
          andrew.wang Andrew Wang added a comment - Backport patches look good, committed to branch-2 and branch-2.8. Thank you Xiao for the contribution!
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          -1 patch 0m 5s HADOOP-13030 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



          Subsystem Report/Notes
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12801154/HADOOP-13030.b28.patch
          JIRA Issue HADOOP-13030
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9200/console
          Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 5s HADOOP-13030 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12801154/HADOOP-13030.b28.patch JIRA Issue HADOOP-13030 Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9200/console Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xiaochen Xiao Chen added a comment -

          Thank you Andrew Wang! Attached a patch based on branch-2.8.

          Verified the same way, works. (had to add export CATALINA_BASE=$HADOOP_CATALINA_HOME to test-source.sh due to kms.sh diff).

          Show
          xiaochen Xiao Chen added a comment - Thank you Andrew Wang ! Attached a patch based on branch-2.8. Verified the same way, works. (had to add export CATALINA_BASE=$HADOOP_CATALINA_HOME to test-source.sh due to kms.sh diff).
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #9686 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9686/)
          HADOOP-13030. Handle special characters in passwords in KMS startup (wang: rev 6f26b665874f923d50087f68357ac822fa9fe709)

          • hadoop-common-project/hadoop-kms/src/main/sbin/kms.sh
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9686 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9686/ ) HADOOP-13030 . Handle special characters in passwords in KMS startup (wang: rev 6f26b665874f923d50087f68357ac822fa9fe709) hadoop-common-project/hadoop-kms/src/main/sbin/kms.sh
          Hide
          andrew.wang Andrew Wang added a comment -

          +1 LGTM, thanks Xiao for working on this. I've committed this to trunk.

          branch-2 and before don't have the shell script rewrite, so are somewhat different. Xiao, do you mind preparing a branch-2 patch too?

          Show
          andrew.wang Andrew Wang added a comment - +1 LGTM, thanks Xiao for working on this. I've committed this to trunk. branch-2 and before don't have the shell script rewrite, so are somewhat different. Xiao, do you mind preparing a branch-2 patch too?
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 10s Docker mode activated.
          0 shelldocs 0m 4s Shelldocs was not available.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 6m 46s trunk passed
          +1 mvnsite 0m 20s trunk passed
          +1 mvnsite 0m 19s the patch passed
          +1 shellcheck 0m 9s The applied patch generated 0 new + 90 unchanged - 3 fixed = 90 total (was 93)
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 unit 0m 11s hadoop-kms in the patch passed with JDK v1.8.0_77.
          +1 unit 0m 14s hadoop-kms in the patch passed with JDK v1.7.0_95.
          +1 asflicense 0m 17s Patch does not generate ASF License warnings.
          8m 41s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:fbe3e86
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12799297/HADOOP-13030.03.patch
          JIRA Issue HADOOP-13030
          Optional Tests asflicense mvnsite unit shellcheck shelldocs
          uname Linux 2f630927a439 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / d8b729e
          shellcheck v0.4.3
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9112/testReport/
          modules C: hadoop-common-project/hadoop-kms U: hadoop-common-project/hadoop-kms
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9112/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 10s Docker mode activated. 0 shelldocs 0m 4s Shelldocs was not available. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 6m 46s trunk passed +1 mvnsite 0m 20s trunk passed +1 mvnsite 0m 19s the patch passed +1 shellcheck 0m 9s The applied patch generated 0 new + 90 unchanged - 3 fixed = 90 total (was 93) +1 whitespace 0m 0s Patch has no whitespace issues. +1 unit 0m 11s hadoop-kms in the patch passed with JDK v1.8.0_77. +1 unit 0m 14s hadoop-kms in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 17s Patch does not generate ASF License warnings. 8m 41s Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12799297/HADOOP-13030.03.patch JIRA Issue HADOOP-13030 Optional Tests asflicense mvnsite unit shellcheck shelldocs uname Linux 2f630927a439 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / d8b729e shellcheck v0.4.3 JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9112/testReport/ modules C: hadoop-common-project/hadoop-kms U: hadoop-common-project/hadoop-kms Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9112/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          xiaochen Xiao Chen added a comment -

          Patch 3 adds my first part of comment to the function, for better readability.

          Show
          xiaochen Xiao Chen added a comment - Patch 3 adds my first part of comment to the function, for better readability.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 21s Docker mode activated.
          0 shelldocs 0m 4s Shelldocs was not available.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 10m 0s trunk passed
          +1 mvnsite 0m 28s trunk passed
          +1 mvnsite 0m 22s the patch passed
          +1 shellcheck 0m 9s The applied patch generated 0 new + 90 unchanged - 3 fixed = 90 total (was 93)
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 unit 0m 17s hadoop-kms in the patch passed with JDK v1.8.0_77.
          +1 unit 0m 14s hadoop-kms in the patch passed with JDK v1.7.0_95.
          +1 asflicense 0m 19s Patch does not generate ASF License warnings.
          12m 30s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:fbe3e86
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12799198/HADOOP-13030.02.patch
          JIRA Issue HADOOP-13030
          Optional Tests asflicense mvnsite unit shellcheck shelldocs
          uname Linux 99f867c796bc 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 67523ff
          shellcheck v0.4.3
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9107/testReport/
          modules C: hadoop-common-project/hadoop-kms U: hadoop-common-project/hadoop-kms
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9107/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 21s Docker mode activated. 0 shelldocs 0m 4s Shelldocs was not available. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 10m 0s trunk passed +1 mvnsite 0m 28s trunk passed +1 mvnsite 0m 22s the patch passed +1 shellcheck 0m 9s The applied patch generated 0 new + 90 unchanged - 3 fixed = 90 total (was 93) +1 whitespace 0m 0s Patch has no whitespace issues. +1 unit 0m 17s hadoop-kms in the patch passed with JDK v1.8.0_77. +1 unit 0m 14s hadoop-kms in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 19s Patch does not generate ASF License warnings. 12m 30s Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12799198/HADOOP-13030.02.patch JIRA Issue HADOOP-13030 Optional Tests asflicense mvnsite unit shellcheck shelldocs uname Linux 99f867c796bc 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 67523ff shellcheck v0.4.3 JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9107/testReport/ modules C: hadoop-common-project/hadoop-kms U: hadoop-common-project/hadoop-kms Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9107/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          xiaochen Xiao Chen added a comment -

          Patch 2 fixes the shellcheck warnings.

          Show
          xiaochen Xiao Chen added a comment - Patch 2 fixes the shellcheck warnings.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 14s Docker mode activated.
          0 shelldocs 0m 4s Shelldocs was not available.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 7m 17s trunk passed
          +1 mvnsite 0m 23s trunk passed
          +1 mvnsite 0m 17s the patch passed
          -1 shellcheck 0m 9s The applied patch generated 1 new + 90 unchanged - 3 fixed = 91 total (was 93)
          +1 whitespace 0m 1s Patch has no whitespace issues.
          +1 unit 0m 10s hadoop-kms in the patch passed with JDK v1.8.0_77.
          +1 unit 0m 12s hadoop-kms in the patch passed with JDK v1.7.0_95.
          +1 asflicense 0m 18s Patch does not generate ASF License warnings.
          9m 16s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:fbe3e86
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12799174/HADOOP-13030.01.patch
          JIRA Issue HADOOP-13030
          Optional Tests asflicense mvnsite unit shellcheck shelldocs
          uname Linux fa1dd24ff186 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 67523ff
          shellcheck v0.4.3
          shellcheck https://builds.apache.org/job/PreCommit-HADOOP-Build/9104/artifact/patchprocess/diff-patch-shellcheck.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9104/testReport/
          modules C: hadoop-common-project/hadoop-kms U: hadoop-common-project/hadoop-kms
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9104/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. 0 shelldocs 0m 4s Shelldocs was not available. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 17s trunk passed +1 mvnsite 0m 23s trunk passed +1 mvnsite 0m 17s the patch passed -1 shellcheck 0m 9s The applied patch generated 1 new + 90 unchanged - 3 fixed = 91 total (was 93) +1 whitespace 0m 1s Patch has no whitespace issues. +1 unit 0m 10s hadoop-kms in the patch passed with JDK v1.8.0_77. +1 unit 0m 12s hadoop-kms in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 18s Patch does not generate ASF License warnings. 9m 16s Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12799174/HADOOP-13030.01.patch JIRA Issue HADOOP-13030 Optional Tests asflicense mvnsite unit shellcheck shelldocs uname Linux fa1dd24ff186 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 67523ff shellcheck v0.4.3 shellcheck https://builds.apache.org/job/PreCommit-HADOOP-Build/9104/artifact/patchprocess/diff-patch-shellcheck.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9104/testReport/ modules C: hadoop-common-project/hadoop-kms U: hadoop-common-project/hadoop-kms Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9104/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          xiaochen Xiao Chen added a comment -

          Attached patch 01 to fix the issue.
          Generally the problem is two fold:

          • The sed line need to escape its sed-special chars(\/&).
          • Since the output is an XML, xml-special chars also need to be escaped(&'"<>).

          The tarball attached is how I verified the fix. (test.sh contains only the relevant part of the kms.sh script in patch 01)
          Step I used to verify:

          • extract tarball and cd to the dir
          • source ./test-source.sh
          • ./test.sh
          • Open up conf/ssl-server.xml in a browser and compare that to the passwords given in test-source.sh. They're identical.

          Special thanks to Mike Yoder for helping explain the issue and provide test data + fun reproduction.

          Show
          xiaochen Xiao Chen added a comment - Attached patch 01 to fix the issue. Generally the problem is two fold: The sed line need to escape its sed-special chars( \/& ). Since the output is an XML, xml-special chars also need to be escaped( &'"<> ). The tarball attached is how I verified the fix. ( test.sh contains only the relevant part of the kms.sh script in patch 01) Step I used to verify: extract tarball and cd to the dir source ./test-source.sh ./test.sh Open up conf/ssl-server.xml in a browser and compare that to the passwords given in test-source.sh. They're identical. Special thanks to Mike Yoder for helping explain the issue and provide test data + fun reproduction.

            People

            • Assignee:
              xiaochen Xiao Chen
              Reporter:
              xiaochen Xiao Chen
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development