Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-11891

OsSecureRandom should lazily fill its reservoir

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.7.0
    • Fix Version/s: 2.8.0, 2.7.1, 3.0.0-alpha1
    • Component/s: security
    • Labels:
      None
    • Target Version/s:

      Description

      The OsSecureRandom class that is initialized by on the the CryptoCodec implementations eagerly fills its reservoir of random bits. This will result in too many file descriptors being opened simply and is not really required if the client already has a random IV and key needed for decryption.

      1. HADOOP-11891.1.patch
        0.7 kB
        Arun Suresh
      2. HADOOP-11891.2.patch
        1 kB
        Arun Suresh

        Activity

        Hide
        asuresh Arun Suresh added a comment -

        Ataching trivial patch..

        Show
        asuresh Arun Suresh added a comment - Ataching trivial patch..
        Hide
        andrew.wang Andrew Wang added a comment -

        Hey Arun, looks good, I think we can improve it a bit more by lazily opening the FileInputStream too. That way we save the open and the FD if the Random is never used.

        Show
        andrew.wang Andrew Wang added a comment - Hey Arun, looks good, I think we can improve it a bit more by lazily opening the FileInputStream too. That way we save the open and the FD if the Random is never used.
        Hide
        asuresh Arun Suresh added a comment -

        Good catch Andrew Wang, updating patch..

        Show
        asuresh Arun Suresh added a comment - Good catch Andrew Wang , updating patch..
        Hide
        andrew.wang Andrew Wang added a comment -

        +1 pending Jenkins, thanks Arun

        Show
        andrew.wang Andrew Wang added a comment - +1 pending Jenkins, thanks Arun
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 14m 36s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 javac 7m 30s There were no new javac warning messages.
        +1 javadoc 9m 34s There were no new javadoc warning messages.
        +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
        +1 checkstyle 5m 29s There were no new checkstyle issues.
        +1 install 1m 33s mvn install still works.
        +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
        +1 findbugs 1m 38s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
        +1 common tests 23m 2s Tests passed in hadoop-common.
            64m 20s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12729295/HADOOP-11891.2.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 3dd6395
        hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6218/artifact/patchprocess/testrun_hadoop-common.txt
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6218/testReport/
        Java 1.7.0_55
        uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6218/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 36s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 javac 7m 30s There were no new javac warning messages. +1 javadoc 9m 34s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 5m 29s There were no new checkstyle issues. +1 install 1m 33s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 1m 38s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 common tests 23m 2s Tests passed in hadoop-common.     64m 20s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12729295/HADOOP-11891.2.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 3dd6395 hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6218/artifact/patchprocess/testrun_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6218/testReport/ Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6218/console This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 15m 2s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 javac 7m 45s There were no new javac warning messages.
        +1 javadoc 9m 47s There were no new javadoc warning messages.
        +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
        +1 checkstyle 5m 28s There were no new checkstyle issues.
        +1 install 1m 34s mvn install still works.
        +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
        +1 findbugs 1m 40s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
        +1 common tests 22m 43s Tests passed in hadoop-common.
            65m 2s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12729295/HADOOP-11891.2.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 3dd6395
        hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6219/artifact/patchprocess/testrun_hadoop-common.txt
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6219/testReport/
        Java 1.7.0_55
        uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6219/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 15m 2s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 javac 7m 45s There were no new javac warning messages. +1 javadoc 9m 47s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 5m 28s There were no new checkstyle issues. +1 install 1m 34s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 1m 40s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 common tests 22m 43s Tests passed in hadoop-common.     65m 2s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12729295/HADOOP-11891.2.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 3dd6395 hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6219/artifact/patchprocess/testrun_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6219/testReport/ Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6219/console This message was automatically generated.
        Hide
        asuresh Arun Suresh added a comment -

        Andrew Wang, thanks for the review
        Manually tested and verified using lsof that no new file descriptors are created for /dev/urandom when OsSecureRandom is initialized and setConf() is called.. only when a random bytes are requested.

        Show
        asuresh Arun Suresh added a comment - Andrew Wang , thanks for the review Manually tested and verified using lsof that no new file descriptors are created for /dev/urandom when OsSecureRandom is initialized and setConf() is called.. only when a random bytes are requested.
        Hide
        asuresh Arun Suresh added a comment -

        committed to trunk, branch-2 and branch-2.7

        Show
        asuresh Arun Suresh added a comment - committed to trunk, branch-2 and branch-2.7
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #7708 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7708/)
        HADOOP-11891. OsSecureRandom should lazily fill its reservoir (asuresh) (Arun Suresh: rev f0db797be28ca221d540c6a3accd6bff9a7996fa)

        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/random/OsSecureRandom.java
        • hadoop-common-project/hadoop-common/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7708 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7708/ ) HADOOP-11891 . OsSecureRandom should lazily fill its reservoir (asuresh) (Arun Suresh: rev f0db797be28ca221d540c6a3accd6bff9a7996fa) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/random/OsSecureRandom.java hadoop-common-project/hadoop-common/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #180 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/180/)
        HADOOP-11891. OsSecureRandom should lazily fill its reservoir (asuresh) (Arun Suresh: rev f0db797be28ca221d540c6a3accd6bff9a7996fa)

        • hadoop-common-project/hadoop-common/CHANGES.txt
        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/random/OsSecureRandom.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #180 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/180/ ) HADOOP-11891 . OsSecureRandom should lazily fill its reservoir (asuresh) (Arun Suresh: rev f0db797be28ca221d540c6a3accd6bff9a7996fa) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/random/OsSecureRandom.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk #914 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/914/)
        HADOOP-11891. OsSecureRandom should lazily fill its reservoir (asuresh) (Arun Suresh: rev f0db797be28ca221d540c6a3accd6bff9a7996fa)

        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/random/OsSecureRandom.java
        • hadoop-common-project/hadoop-common/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #914 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/914/ ) HADOOP-11891 . OsSecureRandom should lazily fill its reservoir (asuresh) (Arun Suresh: rev f0db797be28ca221d540c6a3accd6bff9a7996fa) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/random/OsSecureRandom.java hadoop-common-project/hadoop-common/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk #2112 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2112/)
        HADOOP-11891. OsSecureRandom should lazily fill its reservoir (asuresh) (Arun Suresh: rev f0db797be28ca221d540c6a3accd6bff9a7996fa)

        • hadoop-common-project/hadoop-common/CHANGES.txt
        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/random/OsSecureRandom.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2112 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2112/ ) HADOOP-11891 . OsSecureRandom should lazily fill its reservoir (asuresh) (Arun Suresh: rev f0db797be28ca221d540c6a3accd6bff9a7996fa) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/random/OsSecureRandom.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #171 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/171/)
        HADOOP-11891. OsSecureRandom should lazily fill its reservoir (asuresh) (Arun Suresh: rev f0db797be28ca221d540c6a3accd6bff9a7996fa)

        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/random/OsSecureRandom.java
        • hadoop-common-project/hadoop-common/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #171 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/171/ ) HADOOP-11891 . OsSecureRandom should lazily fill its reservoir (asuresh) (Arun Suresh: rev f0db797be28ca221d540c6a3accd6bff9a7996fa) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/random/OsSecureRandom.java hadoop-common-project/hadoop-common/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #181 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/181/)
        HADOOP-11891. OsSecureRandom should lazily fill its reservoir (asuresh) (Arun Suresh: rev f0db797be28ca221d540c6a3accd6bff9a7996fa)

        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/random/OsSecureRandom.java
        • hadoop-common-project/hadoop-common/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #181 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/181/ ) HADOOP-11891 . OsSecureRandom should lazily fill its reservoir (asuresh) (Arun Suresh: rev f0db797be28ca221d540c6a3accd6bff9a7996fa) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/random/OsSecureRandom.java hadoop-common-project/hadoop-common/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2130 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2130/)
        HADOOP-11891. OsSecureRandom should lazily fill its reservoir (asuresh) (Arun Suresh: rev f0db797be28ca221d540c6a3accd6bff9a7996fa)

        • hadoop-common-project/hadoop-common/CHANGES.txt
        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/random/OsSecureRandom.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2130 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2130/ ) HADOOP-11891 . OsSecureRandom should lazily fill its reservoir (asuresh) (Arun Suresh: rev f0db797be28ca221d540c6a3accd6bff9a7996fa) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/random/OsSecureRandom.java

          People

          • Assignee:
            asuresh Arun Suresh
            Reporter:
            asuresh Arun Suresh
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development