ZooKeeper
  1. ZooKeeper
  2. ZOOKEEPER-516

add support for 10 minute test ie "pre-commit" test

    Details

    • Type: Task Task
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.3.0
    • Component/s: tests
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Our unit tests are now taking > 10 minutes (20 actually). We need to set things up such that we have a < 10 minute "pre commit" build, and a longer running test of tests (unbounded for all intents/purposes). Mainly this requires us to categorize our tests.

      1. ZOOKEEPER-516.patch
        86 kB
        Patrick Hunt
      2. ZOOKEEPER-516.patch
        98 kB
        Patrick Hunt

        Activity

        Patrick Hunt made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Patrick Hunt made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags [Reviewed]
        Resolution Fixed [ 1 ]
        Hide
        Patrick Hunt added a comment -

        committed to trunk.

        Show
        Patrick Hunt added a comment - committed to trunk.
        Hide
        Mahadev konar added a comment -

        +1 looks good....

        Show
        Mahadev konar added a comment - +1 looks good....
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12419662/ZOOKEEPER-516.patch
        against trunk revision 808557.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 126 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h8.grid.sp2.yahoo.net/7/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h8.grid.sp2.yahoo.net/7/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h8.grid.sp2.yahoo.net/7/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12419662/ZOOKEEPER-516.patch against trunk revision 808557. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 126 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h8.grid.sp2.yahoo.net/7/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h8.grid.sp2.yahoo.net/7/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h8.grid.sp2.yahoo.net/7/console This message is automatically generated.
        Patrick Hunt made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Patrick Hunt made changes -
        Attachment ZOOKEEPER-516.patch [ 12419662 ]
        Hide
        Patrick Hunt added a comment -

        updated patch addresses some questions from mahadev (added comments).

        setup change was a mistake - that should have been local only change. I removed that from the patch.

        I made one additional change in this patch - only a single zkServer is started for all of the c tests, rather than
        starting a new one for each test (setup/teardown). This resulting in another ~minute saved on my
        laptop for overall time to run the tests. this basically means we need to be careful in the tests to
        prefix paths for each test uniquely. (I updated a few tests for this).

        Show
        Patrick Hunt added a comment - updated patch addresses some questions from mahadev (added comments). setup change was a mistake - that should have been local only change. I removed that from the patch. I made one additional change in this patch - only a single zkServer is started for all of the c tests, rather than starting a new one for each test (setup/teardown). This resulting in another ~minute saved on my laptop for overall time to run the tests. this basically means we need to be careful in the tests to prefix paths for each test uniquely. (I updated a few tests for this).
        Patrick Hunt made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Hide
        Mahadev konar added a comment -

        the patch looks good... I have a few comments/questions -

        • why did we add this?
        +          <sysproperty key="zookeeper.DigestAuthenticationProvider.superDigest"
        +                       value="super:D/InIHSb7yEEbrWz8b9l71RjZJU=" />
              
        
        • and this change, is this a fix for python tests?
          +                            library_dirs=["/usr/local/lib","../../../build/test/test-cppunit/.libs"])
           
        • and I am not sure if I understand what the following code is doing
          +        
          +        OperatingSystemMXBean osMbean =
          +            ManagementFactory.getOperatingSystemMXBean();
          +        if (osMbean != null && osMbean instanceof UnixOperatingSystemMXBean) {
          +            UnixOperatingSystemMXBean unixos =
          +                (UnixOperatingSystemMXBean)osMbean;
          +            LOG.info("Initial fdcount is: "
          +                    + unixos.getOpenFileDescriptorCount());
          +        }
           
          

          can you please explain?

        Show
        Mahadev konar added a comment - the patch looks good... I have a few comments/questions - why did we add this? + <sysproperty key= "zookeeper.DigestAuthenticationProvider.superDigest" + value= " super :D/InIHSb7yEEbrWz8b9l71RjZJU=" /> and this change, is this a fix for python tests? + library_dirs=[ "/usr/local/lib" , "../../../build/test/test-cppunit/.libs" ]) and I am not sure if I understand what the following code is doing + + OperatingSystemMXBean osMbean = + ManagementFactory.getOperatingSystemMXBean(); + if (osMbean != null && osMbean instanceof UnixOperatingSystemMXBean) { + UnixOperatingSystemMXBean unixos = + (UnixOperatingSystemMXBean)osMbean; + LOG.info( "Initial fdcount is: " + + unixos.getOpenFileDescriptorCount()); + } can you please explain?
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12419349/ZOOKEEPER-516.patch
        against trunk revision 808557.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 98 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h8.grid.sp2.yahoo.net/6/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h8.grid.sp2.yahoo.net/6/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h8.grid.sp2.yahoo.net/6/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12419349/ZOOKEEPER-516.patch against trunk revision 808557. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 98 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h8.grid.sp2.yahoo.net/6/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h8.grid.sp2.yahoo.net/6/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h8.grid.sp2.yahoo.net/6/console This message is automatically generated.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12419349/ZOOKEEPER-516.patch
        against trunk revision 808557.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 98 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h8.grid.sp2.yahoo.net/5/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h8.grid.sp2.yahoo.net/5/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h8.grid.sp2.yahoo.net/5/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12419349/ZOOKEEPER-516.patch against trunk revision 808557. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 98 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h8.grid.sp2.yahoo.net/5/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h8.grid.sp2.yahoo.net/5/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h8.grid.sp2.yahoo.net/5/console This message is automatically generated.
        Patrick Hunt made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Patrick Hunt made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Patrick Hunt made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Assignee Patrick Hunt [ phunt ]
        Patrick Hunt made changes -
        Field Original Value New Value
        Attachment ZOOKEEPER-516.patch [ 12419349 ]
        Hide
        Patrick Hunt added a comment -

        this patch gets the time under 10min for

        ant -Dtest.quick=yes -Dtest.junit.fork.mode=once clean test

        on my old/slow/singlecore laptop. notice that this is from "clean" so no only are the tests being
        run but the code (java and c) is included in the time as well.

        Notes - Java:
        My laptop I was able to reduce the runtime by over 30% just by cleaning up waits
        fixed a problem in fork once mode for superuser auth
        test.quick (off by default) when turned on will skip the hammer tests
        cleanuped the client cleanup test - instead of forcing us to run out of fds I use the jvm instrumentation
        to monitor the number of active fds. (although this really just works on unix)

        Notes - C/C++:
        I added elapsed time for each of the tests
        I didn't change much here wrt timing. there's some time that could be saved (esp around client est)
        I did fix the logging of zk client output - it was turned off for some reason (now output similar to java clients)
        I also fixed some of the logging formats used (esp around hex formatting of session/xid)

        Show
        Patrick Hunt added a comment - this patch gets the time under 10min for ant -Dtest.quick=yes -Dtest.junit.fork.mode=once clean test on my old/slow/singlecore laptop. notice that this is from "clean" so no only are the tests being run but the code (java and c) is included in the time as well. Notes - Java: My laptop I was able to reduce the runtime by over 30% just by cleaning up waits fixed a problem in fork once mode for superuser auth test.quick (off by default) when turned on will skip the hammer tests cleanuped the client cleanup test - instead of forcing us to run out of fds I use the jvm instrumentation to monitor the number of active fds. (although this really just works on unix) Notes - C/C++: I added elapsed time for each of the tests I didn't change much here wrt timing. there's some time that could be saved (esp around client est) I did fix the logging of zk client output - it was turned off for some reason (now output similar to java clients) I also fixed some of the logging formats used (esp around hex formatting of session/xid)
        Patrick Hunt created issue -

          People

          • Assignee:
            Patrick Hunt
            Reporter:
            Patrick Hunt
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development