Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-8451

DFSClient probe for encryption testing interprets empty URI property for "enabled"

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 2.7.1
    • Fix Version/s: 2.8.0, 2.7.1, 3.0.0-alpha1
    • Component/s: encryption
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      HDFS-7931 added a check in DFSClient for encryption isHDFSEncryptionEnabled(), looking for the property {{"dfs.encryption.key.provider.uri"}.

      This probe returns true even if the property is empty.

      If there is an empty provider.uri property, you get an NPE when a YARN client tries to set up the tokens to deploy an AM.

        Issue Links

          Activity

          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2151 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2151/)
          HDFS-8451. DFSClient probe for encryption testing interprets empty URI property for enabled. Contributed by Steve Loughran. (xyao: rev 05e04f34f27149537fdb89f46af26bee14531ca4)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/KeyProviderCache.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestEncryptionZones.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUtil.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2151 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2151/ ) HDFS-8451 . DFSClient probe for encryption testing interprets empty URI property for enabled. Contributed by Steve Loughran. (xyao: rev 05e04f34f27149537fdb89f46af26bee14531ca4) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/KeyProviderCache.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestEncryptionZones.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUtil.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #203 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/203/)
          HDFS-8451. DFSClient probe for encryption testing interprets empty URI property for enabled. Contributed by Steve Loughran. (xyao: rev 05e04f34f27149537fdb89f46af26bee14531ca4)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestEncryptionZones.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/KeyProviderCache.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUtil.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #203 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/203/ ) HDFS-8451 . DFSClient probe for encryption testing interprets empty URI property for enabled. Contributed by Steve Loughran. (xyao: rev 05e04f34f27149537fdb89f46af26bee14531ca4) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestEncryptionZones.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/KeyProviderCache.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUtil.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #193 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/193/)
          HDFS-8451. DFSClient probe for encryption testing interprets empty URI property for enabled. Contributed by Steve Loughran. (xyao: rev 05e04f34f27149537fdb89f46af26bee14531ca4)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUtil.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/KeyProviderCache.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestEncryptionZones.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #193 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/193/ ) HDFS-8451 . DFSClient probe for encryption testing interprets empty URI property for enabled. Contributed by Steve Loughran. (xyao: rev 05e04f34f27149537fdb89f46af26bee14531ca4) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUtil.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/KeyProviderCache.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestEncryptionZones.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2133 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2133/)
          HDFS-8451. DFSClient probe for encryption testing interprets empty URI property for enabled. Contributed by Steve Loughran. (xyao: rev 05e04f34f27149537fdb89f46af26bee14531ca4)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/KeyProviderCache.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestEncryptionZones.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUtil.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2133 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2133/ ) HDFS-8451 . DFSClient probe for encryption testing interprets empty URI property for enabled. Contributed by Steve Loughran. (xyao: rev 05e04f34f27149537fdb89f46af26bee14531ca4) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/KeyProviderCache.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestEncryptionZones.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUtil.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #204 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/204/)
          HDFS-8451. DFSClient probe for encryption testing interprets empty URI property for enabled. Contributed by Steve Loughran. (xyao: rev 05e04f34f27149537fdb89f46af26bee14531ca4)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestEncryptionZones.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUtil.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/KeyProviderCache.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #204 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/204/ ) HDFS-8451 . DFSClient probe for encryption testing interprets empty URI property for enabled. Contributed by Steve Loughran. (xyao: rev 05e04f34f27149537fdb89f46af26bee14531ca4) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestEncryptionZones.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUtil.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/KeyProviderCache.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk #935 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/935/)
          HDFS-8451. DFSClient probe for encryption testing interprets empty URI property for enabled. Contributed by Steve Loughran. (xyao: rev 05e04f34f27149537fdb89f46af26bee14531ca4)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUtil.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestEncryptionZones.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/KeyProviderCache.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #935 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/935/ ) HDFS-8451 . DFSClient probe for encryption testing interprets empty URI property for enabled. Contributed by Steve Loughran. (xyao: rev 05e04f34f27149537fdb89f46af26bee14531ca4) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUtil.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestEncryptionZones.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/KeyProviderCache.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
          Hide
          xyao Xiaoyu Yao added a comment -

          Thanks Steve Loughran for the contribution! I'm committed the fix to trunk, branch-2 and branch-2.7.

          Show
          xyao Xiaoyu Yao added a comment - Thanks Steve Loughran for the contribution! I'm committed the fix to trunk, branch-2 and branch-2.7.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #7884 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7884/)
          HDFS-8451. DFSClient probe for encryption testing interprets empty URI property for enabled. Contributed by Steve Loughran. (xyao: rev 05e04f34f27149537fdb89f46af26bee14531ca4)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/KeyProviderCache.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestEncryptionZones.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUtil.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7884 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7884/ ) HDFS-8451 . DFSClient probe for encryption testing interprets empty URI property for enabled. Contributed by Steve Loughran. (xyao: rev 05e04f34f27149537fdb89f46af26bee14531ca4) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/KeyProviderCache.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestEncryptionZones.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUtil.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          xyao Xiaoyu Yao added a comment -

          I will commit it shortly.

          Show
          xyao Xiaoyu Yao added a comment - I will commit it shortly.
          Hide
          xyao Xiaoyu Yao added a comment -

          +1 for the fix.

          Show
          xyao Xiaoyu Yao added a comment - +1 for the fix.
          Hide
          xyao Xiaoyu Yao added a comment -

          Thanks Steve Loughran for working on this. The patch looks good to me.
          We could also fix this by checking if (dfs.getKeyProvider != null) instead of using if (dfs.isHDFSEncryptionEnabled()) in DistributedFileSystem#addDelegationTokens(). This way, invalid key provider URIs (such as null or empty) could naturally be handled in KeyProviderCache#get(). And we could also remove isHDFSEncryptionEnabled() which is only used in DistributedFileSystem#addDelegationTokens.

          Show
          xyao Xiaoyu Yao added a comment - Thanks Steve Loughran for working on this. The patch looks good to me. We could also fix this by checking if (dfs.getKeyProvider != null) instead of using if (dfs.isHDFSEncryptionEnabled()) in DistributedFileSystem#addDelegationTokens(). This way, invalid key provider URIs (such as null or empty) could naturally be handled in KeyProviderCache#get(). And we could also remove isHDFSEncryptionEnabled() which is only used in DistributedFileSystem#addDelegationTokens.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Test failure is about a zip exception reading core-default.xml. That looks to me like one build has just stamped on the maven artifacts of another. Unrelated to this JIRA at all. Cancelling and restarting just to demonstrate.

          Show
          stevel@apache.org Steve Loughran added a comment - Test failure is about a zip exception reading core-default.xml. That looks to me like one build has just stamped on the maven artifacts of another. Unrelated to this JIRA at all. Cancelling and restarting just to demonstrate.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 17m 8s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 2 new or modified test files.
          +1 javac 7m 35s There were no new javac warning messages.
          +1 javadoc 9m 52s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 2m 14s The applied patch generated 1 new checkstyle issues (total was 177, now 177).
          +1 whitespace 0m 1s The patch has no lines that end in whitespace.
          +1 install 1m 34s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          +1 findbugs 3m 9s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 native 3m 21s Pre-build of native portion
          -1 hdfs tests 162m 58s Tests failed in hadoop-hdfs.
              208m 55s  



          Reason Tests
          Failed unit tests hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12734418/HDFS-8451-001.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 0e4f108
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/11083/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11083/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11083/testReport/
          Java 1.7.0_55
          uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11083/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 8s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 2 new or modified test files. +1 javac 7m 35s There were no new javac warning messages. +1 javadoc 9m 52s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 2m 14s The applied patch generated 1 new checkstyle issues (total was 177, now 177). +1 whitespace 0m 1s The patch has no lines that end in whitespace. +1 install 1m 34s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 3m 9s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 21s Pre-build of native portion -1 hdfs tests 162m 58s Tests failed in hadoop-hdfs.     208m 55s   Reason Tests Failed unit tests hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12734418/HDFS-8451-001.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 0e4f108 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/11083/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11083/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11083/testReport/ Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11083/console This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Patch 001

          1. moves the probe to a static method in DFSUtils
          2. changes the logic so that if the trimmed value is "" then there's no encryption. This includes changing the default from null to "" so the getTrimmed() never returns null.
          3. adds a test which validates the logic.
          4. went through all references to the key DFSConfigKeys.DFS_ENCRYPTION_KEY_PROVIDER_URI to enforce the same logic: getTrimmed( KEY, "") with "" meaning "no provider"

          I'm tagging this as critical as in its current state you can't include an empty property in the configuration.

          Show
          stevel@apache.org Steve Loughran added a comment - Patch 001 moves the probe to a static method in DFSUtils changes the logic so that if the trimmed value is "" then there's no encryption. This includes changing the default from null to "" so the getTrimmed() never returns null. adds a test which validates the logic. went through all references to the key DFSConfigKeys.DFS_ENCRYPTION_KEY_PROVIDER_URI to enforce the same logic: getTrimmed( KEY, "") with "" meaning "no provider" I'm tagging this as critical as in its current state you can't include an empty property in the configuration.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Here's the check for HDFS encryption

            public boolean isHDFSEncryptionEnabled() {
              return conf.get(
                  DFSConfigKeys.DFS_ENCRYPTION_KEY_PROVIDER_URI, null) != null;
            }
          

          The presence of an empty <dfs.encryption.key.provider.uri> is enough to fail this test, because the result isn't null, it is "".

          It's not enough to have an empty property (as ramya verified) —the property must be completely deleted.

          So: I think it's a bug in the DFSClient check; it should be something like:

            public boolean isHDFSEncryptionEnabled() {
             String provider = conf.get(DFSConfigKeys.DFS_ENCRYPTION_KEY_PROVIDER_URI, "") 
              return !provider.isEmpty();
            }
          

          That is: if the provider URI is "" then there's no key provider, hence no encryption.

          Fix is trivial, writing tests to verify that everything work will take slightly longer.

          Show
          stevel@apache.org Steve Loughran added a comment - Here's the check for HDFS encryption public boolean isHDFSEncryptionEnabled() { return conf.get( DFSConfigKeys.DFS_ENCRYPTION_KEY_PROVIDER_URI, null ) != null ; } The presence of an empty <dfs.encryption.key.provider.uri> is enough to fail this test, because the result isn't null, it is "". It's not enough to have an empty property (as ramya verified) —the property must be completely deleted. So: I think it's a bug in the DFSClient check; it should be something like: public boolean isHDFSEncryptionEnabled() { String provider = conf.get(DFSConfigKeys.DFS_ENCRYPTION_KEY_PROVIDER_URI, "") return !provider.isEmpty(); } That is: if the provider URI is "" then there's no key provider, hence no encryption. Fix is trivial, writing tests to verify that everything work will take slightly longer.

            People

            • Assignee:
              stevel@apache.org Steve Loughran
              Reporter:
              stevel@apache.org Steve Loughran
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - 1h
                1h
                Remaining:
                Time Spent - 0.5h Remaining Estimate - 0.5h
                0.5h
                Logged:
                Time Spent - 0.5h Remaining Estimate - 0.5h
                0.5h

                  Development