Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-14524

Make CryptoCodec Closeable so it can be cleaned up proactively

        Details

        • Type: Improvement
        • Status: Resolved
        • Priority: Major
        • Resolution: Fixed
        • Affects Version/s: None
        • Fix Version/s: 2.9.0, 3.0.0-alpha4
        • Component/s: None
        • Labels:
          None
        • Hadoop Flags:
          Reviewed

          Description

          See HADOOP-14523 for motivation. Credit to Misha Dmitriev for reporting initially there.

          Basically, the CryptoCodec class is not a closeable, but the OpensslAesCtrCryptoCodec implementation of it contains a closeable member (the Random object). Currently it is left for finalize() to clean up, this depends on when a FGC is run, and would create problems if OpensslAesCtrCryptoCodec is used with OsSecureRandom, which could let OS run out of FDs on /dev/urandom if too many codecs created.

          1. Text File
            HADOOP-14524.01.patch
            11 kB
            Xiao Chen
          2. Text File
            HADOOP-14524.branch-2.01.patch
            11 kB
            Xiao Chen

            Issue Links

            relates to

            Improvement - An improvement or enhancement to an existing feature or task. HADOOP-14523 OpensslAesCtrCryptoCodec.finalize() holds excessive amounts of memory

            • Major - Major loss of function.
            • Resolved

              Activity

              Ascending order - Click to sort in descending order
              Hide
              Permalink
              hadoopqa Hadoop QA added a comment -
              -1 overall



              Vote Subsystem Runtime Comment
              0 reexec 0m 18s Docker mode activated.
              +1 @author 0m 0s The patch does not contain any @author tags.
              +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
              0 mvndep 1m 32s Maven dependency ordering for branch
              +1 mvninstall 13m 31s trunk passed
              +1 compile 13m 36s trunk passed
              +1 checkstyle 1m 53s trunk passed
              +1 mvnsite 2m 21s trunk passed
              -1 findbugs 1m 33s hadoop-common-project/hadoop-common in trunk has 19 extant Findbugs warnings.
              +1 javadoc 1m 52s trunk passed
              0 mvndep 0m 18s Maven dependency ordering for patch
              +1 mvninstall 1m 46s the patch passed
              +1 compile 10m 34s the patch passed
              +1 javac 10m 34s the patch passed
              +1 checkstyle 1m 54s root: The patch generated 0 new + 60 unchanged - 1 fixed = 60 total (was 61)
              +1 mvnsite 2m 24s the patch passed
              +1 whitespace 0m 0s The patch has no whitespace issues.
              +1 findbugs 4m 23s the patch passed
              +1 javadoc 2m 1s the patch passed
              +1 unit 8m 5s hadoop-common in the patch passed.
              +1 unit 1m 24s hadoop-hdfs-client in the patch passed.
              +1 unit 2m 50s hadoop-mapreduce-client-core in the patch passed.
              +1 asflicense 0m 41s The patch does not generate ASF License warnings.
              97m 42s



              Subsystem Report/Notes
              Docker Image:yetus/hadoop:14b5c93
              JIRA Issue HADOOP-14524
              JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12872927/HADOOP-14524.01.patch
              Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
              uname Linux 670e326e4421 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
              Build tool maven
              Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
              git revision trunk / 6ed54f3
              Default Java 1.8.0_131
              findbugs v3.1.0-RC1
              findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/12529/artifact/patchprocess/branch-findbugs-hadoop-common-project_hadoop-common-warnings.html
              Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12529/testReport/
              modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: .
              Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12529/console
              Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

              This message was automatically generated.

              Show
              hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 1m 32s Maven dependency ordering for branch +1 mvninstall 13m 31s trunk passed +1 compile 13m 36s trunk passed +1 checkstyle 1m 53s trunk passed +1 mvnsite 2m 21s trunk passed -1 findbugs 1m 33s hadoop-common-project/hadoop-common in trunk has 19 extant Findbugs warnings. +1 javadoc 1m 52s trunk passed 0 mvndep 0m 18s Maven dependency ordering for patch +1 mvninstall 1m 46s the patch passed +1 compile 10m 34s the patch passed +1 javac 10m 34s the patch passed +1 checkstyle 1m 54s root: The patch generated 0 new + 60 unchanged - 1 fixed = 60 total (was 61) +1 mvnsite 2m 24s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 4m 23s the patch passed +1 javadoc 2m 1s the patch passed +1 unit 8m 5s hadoop-common in the patch passed. +1 unit 1m 24s hadoop-hdfs-client in the patch passed. +1 unit 2m 50s hadoop-mapreduce-client-core in the patch passed. +1 asflicense 0m 41s The patch does not generate ASF License warnings. 97m 42s Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HADOOP-14524 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12872927/HADOOP-14524.01.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 670e326e4421 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 6ed54f3 Default Java 1.8.0_131 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/12529/artifact/patchprocess/branch-findbugs-hadoop-common-project_hadoop-common-warnings.html Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12529/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12529/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
              Hide
              Permalink
              andrew.wang Andrew Wang added a comment -

              +1 LGTM

              Show
              andrew.wang Andrew Wang added a comment - +1 LGTM
              Hide
              Permalink
              xiaochen Xiao Chen added a comment -

              Thanks for the review Andrew! Attaching a branch-2 patch, since there were some minor conflicts in KPCE.
              Also added a import that's removed by HADOOP-14523...

              Show
              xiaochen Xiao Chen added a comment - Thanks for the review Andrew! Attaching a branch-2 patch, since there were some minor conflicts in KPCE. Also added a import that's removed by HADOOP-14523 ...
              Hide
              Permalink
              hadoopqa Hadoop QA added a comment -
              +1 overall



              Vote Subsystem Runtime Comment
              0 reexec 12m 42s Docker mode activated.
              +1 @author 0m 0s The patch does not contain any @author tags.
              +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
              0 mvndep 1m 18s Maven dependency ordering for branch
              +1 mvninstall 6m 37s branch-2 passed
              +1 compile 5m 37s branch-2 passed with JDK v1.8.0_131
              +1 compile 6m 36s branch-2 passed with JDK v1.7.0_131
              +1 checkstyle 1m 28s branch-2 passed
              +1 mvnsite 2m 6s branch-2 passed
              +1 findbugs 4m 25s branch-2 passed
              +1 javadoc 1m 26s branch-2 passed with JDK v1.8.0_131
              +1 javadoc 1m 42s branch-2 passed with JDK v1.7.0_131
              0 mvndep 0m 16s Maven dependency ordering for patch
              +1 mvninstall 1m 41s the patch passed
              +1 compile 5m 35s the patch passed with JDK v1.8.0_131
              +1 javac 5m 35s the patch passed
              +1 compile 6m 39s the patch passed with JDK v1.7.0_131
              +1 javac 6m 39s the patch passed
              +1 checkstyle 1m 30s root: The patch generated 0 new + 58 unchanged - 1 fixed = 58 total (was 59)
              +1 mvnsite 2m 13s the patch passed
              +1 whitespace 0m 0s The patch has no whitespace issues.
              +1 findbugs 5m 18s the patch passed
              +1 javadoc 1m 39s the patch passed with JDK v1.8.0_131
              +1 javadoc 1m 58s the patch passed with JDK v1.7.0_131
              +1 unit 8m 30s hadoop-common in the patch passed with JDK v1.7.0_131.
              +1 unit 1m 19s hadoop-hdfs-client in the patch passed with JDK v1.7.0_131.
              +1 unit 2m 42s hadoop-mapreduce-client-core in the patch passed with JDK v1.7.0_131.
              +1 asflicense 0m 29s The patch does not generate ASF License warnings.
              119m 7s



              Subsystem Report/Notes
              Docker Image:yetus/hadoop:5e40efe
              JIRA Issue HADOOP-14524
              JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12873235/HADOOP-14524.branch-2.01.patch
              Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
              uname Linux 34ca9d9bd5c0 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
              Build tool maven
              Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
              git revision branch-2 / 12cb345
              Default Java 1.7.0_131
              Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_131 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_131
              findbugs v3.0.0
              JDK v1.7.0_131 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12541/testReport/
              modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: .
              Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12541/console
              Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

              This message was automatically generated.

              Show
              hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 12m 42s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 1m 18s Maven dependency ordering for branch +1 mvninstall 6m 37s branch-2 passed +1 compile 5m 37s branch-2 passed with JDK v1.8.0_131 +1 compile 6m 36s branch-2 passed with JDK v1.7.0_131 +1 checkstyle 1m 28s branch-2 passed +1 mvnsite 2m 6s branch-2 passed +1 findbugs 4m 25s branch-2 passed +1 javadoc 1m 26s branch-2 passed with JDK v1.8.0_131 +1 javadoc 1m 42s branch-2 passed with JDK v1.7.0_131 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvninstall 1m 41s the patch passed +1 compile 5m 35s the patch passed with JDK v1.8.0_131 +1 javac 5m 35s the patch passed +1 compile 6m 39s the patch passed with JDK v1.7.0_131 +1 javac 6m 39s the patch passed +1 checkstyle 1m 30s root: The patch generated 0 new + 58 unchanged - 1 fixed = 58 total (was 59) +1 mvnsite 2m 13s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 5m 18s the patch passed +1 javadoc 1m 39s the patch passed with JDK v1.8.0_131 +1 javadoc 1m 58s the patch passed with JDK v1.7.0_131 +1 unit 8m 30s hadoop-common in the patch passed with JDK v1.7.0_131. +1 unit 1m 19s hadoop-hdfs-client in the patch passed with JDK v1.7.0_131. +1 unit 2m 42s hadoop-mapreduce-client-core in the patch passed with JDK v1.7.0_131. +1 asflicense 0m 29s The patch does not generate ASF License warnings. 119m 7s Subsystem Report/Notes Docker Image:yetus/hadoop:5e40efe JIRA Issue HADOOP-14524 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12873235/HADOOP-14524.branch-2.01.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 34ca9d9bd5c0 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / 12cb345 Default Java 1.7.0_131 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_131 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_131 findbugs v3.0.0 JDK v1.7.0_131 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12541/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12541/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
              Hide
              Permalink
              xiaochen Xiao Chen added a comment -

              Committed this to trunk and branch-2. Thanks again Andrew!

              Show
              xiaochen Xiao Chen added a comment - Committed this to trunk and branch-2. Thanks again Andrew!
              Hide
              Permalink
              hudson Hudson added a comment -

              SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11875 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11875/)
              HADOOP-14524. Make CryptoCodec Closeable so it can be cleaned up (xiao: rev 4ebc23ba7b16c7b9acf38b5a864682a6c8890690)

              • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/CryptoCodec.java
              • (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/crypto/TestCryptoStreamsWithOpensslAesCtrCryptoCodec.java
              • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/KeyProviderCryptoExtension.java
              • (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/sasl/DataTransferSaslUtil.java
              • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/OpensslAesCtrCryptoCodec.java
              • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/AesCtrCryptoCodec.java
              • (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/CryptoUtils.java
              • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/CryptoOutputStream.java
              • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/CryptoInputStream.java
              Show
              hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11875 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11875/ ) HADOOP-14524 . Make CryptoCodec Closeable so it can be cleaned up (xiao: rev 4ebc23ba7b16c7b9acf38b5a864682a6c8890690) (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/CryptoCodec.java (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/crypto/TestCryptoStreamsWithOpensslAesCtrCryptoCodec.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/KeyProviderCryptoExtension.java (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/sasl/DataTransferSaslUtil.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/OpensslAesCtrCryptoCodec.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/AesCtrCryptoCodec.java (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/CryptoUtils.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/CryptoOutputStream.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/CryptoInputStream.java

                People

                • Assignee:
                  xiaochen Xiao Chen
                  Reporter:
                  xiaochen Xiao Chen
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  4 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved:

                    Development