Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-14524

Make CryptoCodec Closeable so it can be cleaned up proactively

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-alpha4
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      See HADOOP-14523 for motivation. Credit to Misha Dmitriev for reporting initially there.

      Basically, the CryptoCodec class is not a closeable, but the OpensslAesCtrCryptoCodec implementation of it contains a closeable member (the Random object). Currently it is left for finalize() to clean up, this depends on when a FGC is run, and would create problems if OpensslAesCtrCryptoCodec is used with OsSecureRandom, which could let OS run out of FDs on /dev/urandom if too many codecs created.

      1. HADOOP-14524.01.patch
        11 kB
        Xiao Chen
      2. HADOOP-14524.branch-2.01.patch
        11 kB
        Xiao Chen

        Issue Links

          Activity

          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11875 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11875/)
          HADOOP-14524. Make CryptoCodec Closeable so it can be cleaned up (xiao: rev 4ebc23ba7b16c7b9acf38b5a864682a6c8890690)

          • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/CryptoCodec.java
          • (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/crypto/TestCryptoStreamsWithOpensslAesCtrCryptoCodec.java
          • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/KeyProviderCryptoExtension.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/sasl/DataTransferSaslUtil.java
          • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/OpensslAesCtrCryptoCodec.java
          • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/AesCtrCryptoCodec.java
          • (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/CryptoUtils.java
          • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/CryptoOutputStream.java
          • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/CryptoInputStream.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11875 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11875/ ) HADOOP-14524 . Make CryptoCodec Closeable so it can be cleaned up (xiao: rev 4ebc23ba7b16c7b9acf38b5a864682a6c8890690) (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/CryptoCodec.java (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/crypto/TestCryptoStreamsWithOpensslAesCtrCryptoCodec.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/KeyProviderCryptoExtension.java (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/sasl/DataTransferSaslUtil.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/OpensslAesCtrCryptoCodec.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/AesCtrCryptoCodec.java (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/CryptoUtils.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/CryptoOutputStream.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/CryptoInputStream.java
          Hide
          xiaochen Xiao Chen added a comment -

          Committed this to trunk and branch-2. Thanks again Andrew!

          Show
          xiaochen Xiao Chen added a comment - Committed this to trunk and branch-2. Thanks again Andrew!
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 12m 42s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 1m 18s Maven dependency ordering for branch
          +1 mvninstall 6m 37s branch-2 passed
          +1 compile 5m 37s branch-2 passed with JDK v1.8.0_131
          +1 compile 6m 36s branch-2 passed with JDK v1.7.0_131
          +1 checkstyle 1m 28s branch-2 passed
          +1 mvnsite 2m 6s branch-2 passed
          +1 findbugs 4m 25s branch-2 passed
          +1 javadoc 1m 26s branch-2 passed with JDK v1.8.0_131
          +1 javadoc 1m 42s branch-2 passed with JDK v1.7.0_131
          0 mvndep 0m 16s Maven dependency ordering for patch
          +1 mvninstall 1m 41s the patch passed
          +1 compile 5m 35s the patch passed with JDK v1.8.0_131
          +1 javac 5m 35s the patch passed
          +1 compile 6m 39s the patch passed with JDK v1.7.0_131
          +1 javac 6m 39s the patch passed
          +1 checkstyle 1m 30s root: The patch generated 0 new + 58 unchanged - 1 fixed = 58 total (was 59)
          +1 mvnsite 2m 13s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 5m 18s the patch passed
          +1 javadoc 1m 39s the patch passed with JDK v1.8.0_131
          +1 javadoc 1m 58s the patch passed with JDK v1.7.0_131
          +1 unit 8m 30s hadoop-common in the patch passed with JDK v1.7.0_131.
          +1 unit 1m 19s hadoop-hdfs-client in the patch passed with JDK v1.7.0_131.
          +1 unit 2m 42s hadoop-mapreduce-client-core in the patch passed with JDK v1.7.0_131.
          +1 asflicense 0m 29s The patch does not generate ASF License warnings.
          119m 7s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:5e40efe
          JIRA Issue HADOOP-14524
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12873235/HADOOP-14524.branch-2.01.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 34ca9d9bd5c0 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2 / 12cb345
          Default Java 1.7.0_131
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_131 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_131
          findbugs v3.0.0
          JDK v1.7.0_131 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12541/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12541/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 12m 42s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 1m 18s Maven dependency ordering for branch +1 mvninstall 6m 37s branch-2 passed +1 compile 5m 37s branch-2 passed with JDK v1.8.0_131 +1 compile 6m 36s branch-2 passed with JDK v1.7.0_131 +1 checkstyle 1m 28s branch-2 passed +1 mvnsite 2m 6s branch-2 passed +1 findbugs 4m 25s branch-2 passed +1 javadoc 1m 26s branch-2 passed with JDK v1.8.0_131 +1 javadoc 1m 42s branch-2 passed with JDK v1.7.0_131 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvninstall 1m 41s the patch passed +1 compile 5m 35s the patch passed with JDK v1.8.0_131 +1 javac 5m 35s the patch passed +1 compile 6m 39s the patch passed with JDK v1.7.0_131 +1 javac 6m 39s the patch passed +1 checkstyle 1m 30s root: The patch generated 0 new + 58 unchanged - 1 fixed = 58 total (was 59) +1 mvnsite 2m 13s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 5m 18s the patch passed +1 javadoc 1m 39s the patch passed with JDK v1.8.0_131 +1 javadoc 1m 58s the patch passed with JDK v1.7.0_131 +1 unit 8m 30s hadoop-common in the patch passed with JDK v1.7.0_131. +1 unit 1m 19s hadoop-hdfs-client in the patch passed with JDK v1.7.0_131. +1 unit 2m 42s hadoop-mapreduce-client-core in the patch passed with JDK v1.7.0_131. +1 asflicense 0m 29s The patch does not generate ASF License warnings. 119m 7s Subsystem Report/Notes Docker Image:yetus/hadoop:5e40efe JIRA Issue HADOOP-14524 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12873235/HADOOP-14524.branch-2.01.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 34ca9d9bd5c0 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / 12cb345 Default Java 1.7.0_131 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_131 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_131 findbugs v3.0.0 JDK v1.7.0_131 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12541/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12541/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks for the review Andrew! Attaching a branch-2 patch, since there were some minor conflicts in KPCE.
          Also added a import that's removed by HADOOP-14523...

          Show
          xiaochen Xiao Chen added a comment - Thanks for the review Andrew! Attaching a branch-2 patch, since there were some minor conflicts in KPCE. Also added a import that's removed by HADOOP-14523 ...
          Hide
          andrew.wang Andrew Wang added a comment -

          +1 LGTM

          Show
          andrew.wang Andrew Wang added a comment - +1 LGTM
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 1m 32s Maven dependency ordering for branch
          +1 mvninstall 13m 31s trunk passed
          +1 compile 13m 36s trunk passed
          +1 checkstyle 1m 53s trunk passed
          +1 mvnsite 2m 21s trunk passed
          -1 findbugs 1m 33s hadoop-common-project/hadoop-common in trunk has 19 extant Findbugs warnings.
          +1 javadoc 1m 52s trunk passed
          0 mvndep 0m 18s Maven dependency ordering for patch
          +1 mvninstall 1m 46s the patch passed
          +1 compile 10m 34s the patch passed
          +1 javac 10m 34s the patch passed
          +1 checkstyle 1m 54s root: The patch generated 0 new + 60 unchanged - 1 fixed = 60 total (was 61)
          +1 mvnsite 2m 24s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 4m 23s the patch passed
          +1 javadoc 2m 1s the patch passed
          +1 unit 8m 5s hadoop-common in the patch passed.
          +1 unit 1m 24s hadoop-hdfs-client in the patch passed.
          +1 unit 2m 50s hadoop-mapreduce-client-core in the patch passed.
          +1 asflicense 0m 41s The patch does not generate ASF License warnings.
          97m 42s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:14b5c93
          JIRA Issue HADOOP-14524
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12872927/HADOOP-14524.01.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 670e326e4421 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 6ed54f3
          Default Java 1.8.0_131
          findbugs v3.1.0-RC1
          findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/12529/artifact/patchprocess/branch-findbugs-hadoop-common-project_hadoop-common-warnings.html
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12529/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12529/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 1m 32s Maven dependency ordering for branch +1 mvninstall 13m 31s trunk passed +1 compile 13m 36s trunk passed +1 checkstyle 1m 53s trunk passed +1 mvnsite 2m 21s trunk passed -1 findbugs 1m 33s hadoop-common-project/hadoop-common in trunk has 19 extant Findbugs warnings. +1 javadoc 1m 52s trunk passed 0 mvndep 0m 18s Maven dependency ordering for patch +1 mvninstall 1m 46s the patch passed +1 compile 10m 34s the patch passed +1 javac 10m 34s the patch passed +1 checkstyle 1m 54s root: The patch generated 0 new + 60 unchanged - 1 fixed = 60 total (was 61) +1 mvnsite 2m 24s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 4m 23s the patch passed +1 javadoc 2m 1s the patch passed +1 unit 8m 5s hadoop-common in the patch passed. +1 unit 1m 24s hadoop-hdfs-client in the patch passed. +1 unit 2m 50s hadoop-mapreduce-client-core in the patch passed. +1 asflicense 0m 41s The patch does not generate ASF License warnings. 97m 42s Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HADOOP-14524 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12872927/HADOOP-14524.01.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 670e326e4421 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 6ed54f3 Default Java 1.8.0_131 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/12529/artifact/patchprocess/branch-findbugs-hadoop-common-project_hadoop-common-warnings.html Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12529/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12529/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.

            People

            • Assignee:
              xiaochen Xiao Chen
              Reporter:
              xiaochen Xiao Chen
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development