Hadoop Common
  1. Hadoop Common
  2. HADOOP-7335

Force entropy to come from non-true random for tests

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.22.0
    • Fix Version/s: 0.22.0
    • Component/s: build, test
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Passing the system property -Djava.security.egd=file:///dev/urandom forces the JVM to seed its PRNG from non-true random (/dev/urandom) instead of the true random (/dev/random). This makes the tests run faster, since without it they often hang waiting for entropy while Jetty is initializing.

      We should turn this on for the test targets by default, so developers/hudson boxes don't have to make this change system-wide or use workarounds like rngtools.

      1. hadoop-7335.txt
        0.6 kB
        Todd Lipcon

        Issue Links

          Activity

          Hide
          Todd Lipcon added a comment -

          Here's a patch to pass that flag to tests.

          I'd appreciate if someone can test this on Windows/OSX to make sure it doesn't break their platforms.

          Show
          Todd Lipcon added a comment - Here's a patch to pass that flag to tests. I'd appreciate if someone can test this on Windows/OSX to make sure it doesn't break their platforms.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12481032/hadoop-7335.txt
          against trunk revision 1129982.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/553//testReport/
          Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/553//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/553//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12481032/hadoop-7335.txt against trunk revision 1129982. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/553//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/553//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/553//console This message is automatically generated.
          Hide
          Konstantin Boudnik added a comment -

          Looks good, except that according to JAAS reference guide it suppose to have a single '/' in the file URL. Which doesn't look right to me

          Show
          Konstantin Boudnik added a comment - Looks good, except that according to JAAS reference guide it suppose to have a single '/' in the file URL. Which doesn't look right to me
          Hide
          Todd Lipcon added a comment -

          Yea, I just tried changing my environment to file:/dev/urandom and it doesn't work correctly - I see Jetty hanging at startup. jstack shows it trying to read entropy. If I use file:///dev/urandom as I did in this patch, it starts immediately every time.

          Show
          Todd Lipcon added a comment - Yea, I just tried changing my environment to file:/dev/urandom and it doesn't work correctly - I see Jetty hanging at startup. jstack shows it trying to read entropy. If I use file:///dev/urandom as I did in this patch, it starts immediately every time.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          > I'd appreciate if someone can test this on Windows/OSX ...

          Tried TestHttpServer and TestServletFilter. Seems working fine.

          Show
          Tsz Wo Nicholas Sze added a comment - > I'd appreciate if someone can test this on Windows/OSX ... Tried TestHttpServer and TestServletFilter . Seems working fine.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          I mean tried on Windows/Cygwin.

          Show
          Tsz Wo Nicholas Sze added a comment - I mean tried on Windows/Cygwin.
          Hide
          Eli Collins added a comment -

          +1 lgtm

          OSX has supported /dev/urandom since at least 10.2. From http://developer.apple.com/library/mac/#documentation/darwin/reference/manpages/man4/random.4.html

          /dev/urandom is a compatibility nod to Linux. On Linux, /dev/urandom will produce lower quality output if the entropy pool drains, while /dev/random will prefer to block and wait for additional entropy to be collected. With Yarrow, this choice and distinction is not necessary, and the two devices behave identically. You may use either.

          Show
          Eli Collins added a comment - +1 lgtm OSX has supported /dev/urandom since at least 10.2. From http://developer.apple.com/library/mac/#documentation/darwin/reference/manpages/man4/random.4.html /dev/urandom is a compatibility nod to Linux. On Linux, /dev/urandom will produce lower quality output if the entropy pool drains, while /dev/random will prefer to block and wait for additional entropy to be collected. With Yarrow, this choice and distinction is not necessary, and the two devices behave identically. You may use either.
          Hide
          Eli Collins added a comment -

          Forgot to mention, /dev/urandom has been in Solaris since 2.6.

          Show
          Eli Collins added a comment - Forgot to mention, /dev/urandom has been in Solaris since 2.6.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #642 (See https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/642/)
          HADOOP-7335. Force entropy to come from non-true random for tests. Contributed by Todd Lipcon

          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1132511
          Files :

          • /hadoop/common/trunk/CHANGES.txt
          • /hadoop/common/trunk/build.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #642 (See https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/642/ ) HADOOP-7335 . Force entropy to come from non-true random for tests. Contributed by Todd Lipcon eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1132511 Files : /hadoop/common/trunk/CHANGES.txt /hadoop/common/trunk/build.xml
          Hide
          Eli Collins added a comment -

          I've committed this to branch 22 and trunk. Thanks Todd!

          (the hudson results are green btw: https://builds.apache.org/job/Hadoop-Common-trunk-Commit/642. Not sure why they haven't been posted to jira)

          Show
          Eli Collins added a comment - I've committed this to branch 22 and trunk. Thanks Todd! (the hudson results are green btw: https://builds.apache.org/job/Hadoop-Common-trunk-Commit/642 . Not sure why they haven't been posted to jira)
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-22-branch #62 (See https://builds.apache.org/hudson/job/Hadoop-Common-22-branch/62/)
          HADOOP-7335. svn merge -c 1132511 from trunk

          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1132512
          Files :

          • /hadoop/common/branches/branch-0.22/src/docs
          • /hadoop/common/branches/branch-0.22
          • /hadoop/common/branches/branch-0.22/src/test/core
          • /hadoop/common/branches/branch-0.22/src/test/core/org/apache/hadoop/io/TestSequenceFile.java
          • /hadoop/common/branches/branch-0.22/CHANGES.txt
          • /hadoop/common/branches/branch-0.22/src/java
          • /hadoop/common/branches/branch-0.22/build.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Common-22-branch #62 (See https://builds.apache.org/hudson/job/Hadoop-Common-22-branch/62/ ) HADOOP-7335 . svn merge -c 1132511 from trunk eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1132512 Files : /hadoop/common/branches/branch-0.22/src/docs /hadoop/common/branches/branch-0.22 /hadoop/common/branches/branch-0.22/src/test/core /hadoop/common/branches/branch-0.22/src/test/core/org/apache/hadoop/io/TestSequenceFile.java /hadoop/common/branches/branch-0.22/CHANGES.txt /hadoop/common/branches/branch-0.22/src/java /hadoop/common/branches/branch-0.22/build.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk #711 (See https://builds.apache.org/hudson/job/Hadoop-Common-trunk/711/)
          HADOOP-7335. Force entropy to come from non-true random for tests. Contributed by Todd Lipcon

          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1132511
          Files :

          • /hadoop/common/trunk/CHANGES.txt
          • /hadoop/common/trunk/build.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk #711 (See https://builds.apache.org/hudson/job/Hadoop-Common-trunk/711/ ) HADOOP-7335 . Force entropy to come from non-true random for tests. Contributed by Todd Lipcon eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1132511 Files : /hadoop/common/trunk/CHANGES.txt /hadoop/common/trunk/build.xml

            People

            • Assignee:
              Todd Lipcon
              Reporter:
              Todd Lipcon
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development