Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-14260

Configuration.dumpConfiguration should redact sensitive information

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.6.0
    • Fix Version/s: 2.9.0, 3.0.0-beta1, 2.8.3
    • Component/s: conf, security
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Incompatible change
    • Release Note:
      Hide
      <!-- markdown -->
      Configuration.dumpConfiguration no longer prints out the clear text values for the sensitive keys listed in `hadoop.security.sensitive-config-keys`. Callers can override the default list of sensitive keys either to redact more keys or print the clear text values for a few extra keys for debugging purpose.
      Show
      <!-- markdown --> Configuration.dumpConfiguration no longer prints out the clear text values for the sensitive keys listed in `hadoop.security.sensitive-config-keys`. Callers can override the default list of sensitive keys either to redact more keys or print the clear text values for a few extra keys for debugging purpose.

      Description

      Configuration.dumpConfiguration dumps all the configuration values without redacting the sensitive configurations stored in the Configuration object.

      1. HADOOP-14260.001.patch
        6 kB
        John Zhuge
      2. HADOOP-14260.002.patch
        6 kB
        John Zhuge

        Issue Links

          Activity

          Hide
          jzhuge John Zhuge added a comment -

          Patch 001

          • React sensitive keys in 2 dumpConfiguration methods
          • Please note callers of dumpConfiguration can override the default list of sensitive keys by setting HADOOP_SECURITY_SENSITIVE_CONFIG_KEYS
          Show
          jzhuge John Zhuge added a comment - Patch 001 React sensitive keys in 2 dumpConfiguration methods Please note callers of dumpConfiguration can override the default list of sensitive keys by setting HADOOP_SECURITY_SENSITIVE_CONFIG_KEYS
          Hide
          mackrorysd Sean Mackrory added a comment - - edited

          +1, pending a clean Yetus run.

          As a side-note, seeing the redactor applied to this tool made me wonder if redaction should be able to be turned off via a config, if someone was debugging something involving redacted properties. But then I remembered that the list of regexes that get disabled is configurable, so you could always temporarily exclude one if you really needed to. I don't think we should make that more accessible - but if anyone is wondering how to debug redacted properties, that's what I'd recommend.

          edit: which of course I just saw you called out in your 2nd bullet point. Good one

          Show
          mackrorysd Sean Mackrory added a comment - - edited +1, pending a clean Yetus run. As a side-note, seeing the redactor applied to this tool made me wonder if redaction should be able to be turned off via a config, if someone was debugging something involving redacted properties. But then I remembered that the list of regexes that get disabled is configurable, so you could always temporarily exclude one if you really needed to. I don't think we should make that more accessible - but if anyone is wondering how to debug redacted properties, that's what I'd recommend. edit: which of course I just saw you called out in your 2nd bullet point. Good one
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 21m 27s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
                trunk Compile Tests
          +1 mvninstall 14m 57s trunk passed
          +1 compile 15m 20s trunk passed
          +1 checkstyle 0m 41s trunk passed
          +1 mvnsite 1m 31s trunk passed
          +1 findbugs 1m 31s trunk passed
          +1 javadoc 0m 54s trunk passed
                Patch Compile Tests
          +1 mvninstall 0m 44s the patch passed
          +1 compile 15m 33s the patch passed
          +1 javac 15m 33s the patch passed
          -0 checkstyle 0m 51s hadoop-common-project/hadoop-common: The patch generated 3 new + 263 unchanged - 1 fixed = 266 total (was 264)
          +1 mvnsite 1m 43s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 0s the patch passed
          +1 javadoc 1m 0s the patch passed
                Other Tests
          -1 unit 9m 19s hadoop-common in the patch failed.
          +1 asflicense 0m 39s The patch does not generate ASF License warnings.
          90m 19s



          Reason Tests
          Failed junit tests hadoop.conf.TestCommonConfigurationFields



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:14b5c93
          JIRA Issue HADOOP-14260
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12881356/HADOOP-14260.001.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux ff01c376f66f 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / a32e013
          Default Java 1.8.0_144
          findbugs v3.1.0-RC1
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/13004/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/13004/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/13004/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/13004/console
          Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 21m 27s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       trunk Compile Tests +1 mvninstall 14m 57s trunk passed +1 compile 15m 20s trunk passed +1 checkstyle 0m 41s trunk passed +1 mvnsite 1m 31s trunk passed +1 findbugs 1m 31s trunk passed +1 javadoc 0m 54s trunk passed       Patch Compile Tests +1 mvninstall 0m 44s the patch passed +1 compile 15m 33s the patch passed +1 javac 15m 33s the patch passed -0 checkstyle 0m 51s hadoop-common-project/hadoop-common: The patch generated 3 new + 263 unchanged - 1 fixed = 266 total (was 264) +1 mvnsite 1m 43s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 0s the patch passed +1 javadoc 1m 0s the patch passed       Other Tests -1 unit 9m 19s hadoop-common in the patch failed. +1 asflicense 0m 39s The patch does not generate ASF License warnings. 90m 19s Reason Tests Failed junit tests hadoop.conf.TestCommonConfigurationFields Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HADOOP-14260 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12881356/HADOOP-14260.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux ff01c376f66f 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / a32e013 Default Java 1.8.0_144 findbugs v3.1.0-RC1 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/13004/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/13004/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/13004/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/13004/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          jzhuge John Zhuge added a comment -

          TestCommonConfigurationFields failed is caused by HADOOP-14754. Will upload another patch to fix checkstyle.

          Show
          jzhuge John Zhuge added a comment - TestCommonConfigurationFields failed is caused by HADOOP-14754 . Will upload another patch to fix checkstyle.
          Hide
          jzhuge John Zhuge added a comment -

          Patch 002

          • Fix checkstyle
          Show
          jzhuge John Zhuge added a comment - Patch 002 Fix checkstyle
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 17s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
                trunk Compile Tests
          +1 mvninstall 14m 10s trunk passed
          +1 compile 14m 34s trunk passed
          +1 checkstyle 0m 39s trunk passed
          +1 mvnsite 1m 32s trunk passed
          +1 findbugs 1m 33s trunk passed
          +1 javadoc 0m 55s trunk passed
                Patch Compile Tests
          +1 mvninstall 0m 43s the patch passed
          +1 compile 10m 34s the patch passed
          +1 javac 10m 34s the patch passed
          +1 checkstyle 0m 39s hadoop-common-project/hadoop-common: The patch generated 0 new + 263 unchanged - 1 fixed = 263 total (was 264)
          +1 mvnsite 1m 25s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 33s the patch passed
          +1 javadoc 0m 49s the patch passed
                Other Tests
          -1 unit 8m 6s hadoop-common in the patch failed.
          +1 asflicense 0m 29s The patch does not generate ASF License warnings.
          59m 50s



          Reason Tests
          Failed junit tests hadoop.security.TestKDiag
            hadoop.conf.TestCommonConfigurationFields



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:14b5c93
          JIRA Issue HADOOP-14260
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12881379/HADOOP-14260.002.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux f61cf0ee12e5 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / a32e013
          Default Java 1.8.0_144
          findbugs v3.1.0-RC1
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/13005/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/13005/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/13005/console
          Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       trunk Compile Tests +1 mvninstall 14m 10s trunk passed +1 compile 14m 34s trunk passed +1 checkstyle 0m 39s trunk passed +1 mvnsite 1m 32s trunk passed +1 findbugs 1m 33s trunk passed +1 javadoc 0m 55s trunk passed       Patch Compile Tests +1 mvninstall 0m 43s the patch passed +1 compile 10m 34s the patch passed +1 javac 10m 34s the patch passed +1 checkstyle 0m 39s hadoop-common-project/hadoop-common: The patch generated 0 new + 263 unchanged - 1 fixed = 263 total (was 264) +1 mvnsite 1m 25s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 33s the patch passed +1 javadoc 0m 49s the patch passed       Other Tests -1 unit 8m 6s hadoop-common in the patch failed. +1 asflicense 0m 29s The patch does not generate ASF License warnings. 59m 50s Reason Tests Failed junit tests hadoop.security.TestKDiag   hadoop.conf.TestCommonConfigurationFields Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HADOOP-14260 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12881379/HADOOP-14260.002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux f61cf0ee12e5 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / a32e013 Default Java 1.8.0_144 findbugs v3.1.0-RC1 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/13005/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/13005/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/13005/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          jzhuge John Zhuge added a comment -

          Again TestCommonConfigurationFields failure is caused by HADOOP-14754 where I already submitted a patch.

          Committing this tomorrow if there is no objection.

          Show
          jzhuge John Zhuge added a comment - Again TestCommonConfigurationFields failure is caused by HADOOP-14754 where I already submitted a patch. Committing this tomorrow if there is no objection.
          Hide
          jzhuge John Zhuge added a comment -

          Committed to trunk, branch-2, and branch-2.8.

          Thanks Vihang Karajgaonkar for reporting the issue and Sean Mackrory for the review!

          Show
          jzhuge John Zhuge added a comment - Committed to trunk, branch-2, and branch-2.8. Thanks Vihang Karajgaonkar for reporting the issue and Sean Mackrory for the review!
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12166 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12166/)
          HADOOP-14260. Configuration.dumpConfiguration should redact sensitive (jzhuge: rev 582648befaf9908159f937d2cc8f549583a3483e)

          • (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java
          • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12166 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12166/ ) HADOOP-14260 . Configuration.dumpConfiguration should redact sensitive (jzhuge: rev 582648befaf9908159f937d2cc8f549583a3483e) (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java

            People

            • Assignee:
              jzhuge John Zhuge
              Reporter:
              vihangk1 Vihang Karajgaonkar
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development