Harmony
  1. Harmony
  2. HARMONY-2918

Contribution of reliability test suite

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Contributions
    • Labels:
      None

      Description

      Reliability test suite consisting of 72 reliability tests. Covers mainly kernel, nio, io, net, text, zip classes functionality.

      1. 9newreltests.zip
        40 kB
        Aleksey Ignatenko
      2. DekkerTest.java
        5 kB
        George Timoshenko
      3. DekkerTest.java
        5 kB
        tatyana doubtsova
      4. DekkerTest.java
        5 kB
        tatyana doubtsova
      5. fix.test.reports.patch
        3 kB
        tatyana doubtsova
      6. fix.test.reports.patch
        4 kB
        tatyana doubtsova
      7. fix.test.reports.patch
        2 kB
        Ivaschenko, Aleksei V
      8. fix.test.reports.patch
        2 kB
        Ivaschenko, Aleksei V
      9. jlmanagementtest.zip
        10 kB
        Aleksey Ignatenko
      10. reliability_test_suite.zip
        217 kB
        Oleg Oleinik
      11. reliability_v1.zip
        277 kB
        Aleksey Ignatenko
      12. src_4_thrd_tests_19_03_07.zip
        15 kB
        Oleg Oleinik
      13. src4_zip1_clsload3_tests.zip
        21 kB
        Aleksey Ignatenko
      14. TestDevGuide.txt
        6 kB
        Svetlana Konovalova
      15. TestDevGuide.txt
        5 kB
        Oleg Oleinik
      16. xlist_implementation.zip
        4 kB
        tatyana doubtsova

        Issue Links

          Activity

          Hide
          Oleg Oleinik added a comment -

          reliability_test_suite.zip is reliability test suite.

          Show
          Oleg Oleinik added a comment - reliability_test_suite.zip is reliability test suite.
          Hide
          tatyana doubtsova added a comment -

          I've attached modified DekkerTest.java
          The previous implementation of the test contained volatile array and relyed on array elements as on volatile variables. Due to specification there is no way in Java to have the elements of an array to be volatile. So I've replaced array with variables.
          Also loop condition
          while (current != worker && k < 100000)

          {k++;}

          was replaced with
          while (current != worker) { }
          as Dekker algorithm describes.

          Show
          tatyana doubtsova added a comment - I've attached modified DekkerTest.java The previous implementation of the test contained volatile array and relyed on array elements as on volatile variables. Due to specification there is no way in Java to have the elements of an array to be volatile. So I've replaced array with variables. Also loop condition while (current != worker && k < 100000) {k++;} was replaced with while (current != worker) { } as Dekker algorithm describes.
          Hide
          Geir Magnusson Jr added a comment -

          started vote

          Show
          Geir Magnusson Jr added a comment - started vote
          Hide
          Geir Magnusson Jr added a comment -

          vote passed

          Show
          Geir Magnusson Jr added a comment - vote passed
          Hide
          Oleg Oleinik added a comment -

          4 new tests for Threads, ThreadGroup, ThreadLocal

          Show
          Oleg Oleinik added a comment - 4 new tests for Threads, ThreadGroup, ThreadLocal
          Hide
          Oleg Oleinik added a comment -

          I created four new tests covering methods of Thread, ThreadLocal and ThreadGroup:

          • api.kernel.thread.ExcptHandlerTest.ExcptHandlerTest covers
            Thread's UncaughtExceptionHandler and DefaultUncaughtExceptionHandler.
          • api.kernel.thread.StackTraceTest.StackTraceTest covers Thread's
            getStackTrace() and getAllStackTraces().
          • api.kernel.thread.ThreadLocalTest.ThreadLocalTest covers ThreadLocal.
          • api.kernel.threadgroup.EnumerateTest covers Thread's and
            ThreadGroup's enumerate().

          Test set-up.

          1. Unpack src_4_thrd_tests_19_03_07.zip into test suite root
          directory in which there is src/ directory with existing tests).
          This should create new directories and test files.

          2. Add these lines to run.api.kernel.xml under <target name="run.api.kernel.thread">:

          <run-pjava-test test="api.kernel.thread.ExcptHandlerTest.ExcptHandlerTest" />
          <run-pjava-test test="api.kernel.thread.ThreadLocalTest.ThreadLocalTest" />
          <run-pjava-test test="api.kernel.thread.StackTraceTest.StackTraceTest" />
          <run-pjava-test test="api.kernel.threadgroup.EnumerateTest" />

          One can specify test parameters described in the comments in the beginning of
          each test, but, I think, default parameters specified in the tests are sufficient.

          3. Compile and run tests as described in README:

          cd <TEST SUITE ROOT DIR>
          ant
          cd run
          ant -f run.api.kernel.xml run.api.kernel.thread

          Test run results.

          i tried the tests on WinXP, HDK: svn = r519348, (Mar 17 2007),
          Windows/ia32/msvc 1310, release build (test execution time - 5 minutes):

          • api.kernel.thread.ExcptHandlerTest.ExcptHandlerTest exits
            abnormally on StackOverflowError. If remove the case causing
            StackOverflowError, the test abnormally completes on ~110
            iteration showing that it took all available in system memory.
            Passes on RI.
          • api.kernel.thread.StackTraceTest.StackTraceTest failed on 161'st
            iteration with:

          Exception during test execution: java.util.NoSuchElementException
          java.util.NoSuchElementException
          at java.util.WeakHashMap$HashIterator.next(WeakHashMap.java:162)

          at java.util.AbstractCollection.toArray(AbstractCollection.java)

          at java.lang.ThreadGroup.copyThreads(ThreadGroup.java)
          at java.lang.ThreadGroup.enumerate(ThreadGroup.java:456)
          at java.lang.ThreadGroup.enumerate(ThreadGroup.java:472)

          at java.lang.ThreadGroup.enumerate(ThreadGroup.java)

          at java.lang.Thread.getAllStackTraces(Thread.java)

          Passes on RI.

          • api.kernel.thread.ThreadLocalTest.ThreadLocalTest failed on
            the 1-st iteration with test failure log message:

          Thread ID=3775 checked value of thread ID=943's instance
          ThreadLocal variable: get() returned wrong object
          null instead of org.apache.harmony.test.reliability.
          api.kernel.thread.ThreadLocalTest.SomeObject@20585688

          that signals that ThreadLocal's get sometimes return null
          instead of previously set object.

          Fails on RI with similar message. I failed to find a mistake
          in the test...

          • api.kernel.threadgroup.EnumerateTest hangs on ~150 iteration,
            but, runs OK if the test operates only with non-daemon Threads
            and ThreadGroups.

          Passes on RI.

          Show
          Oleg Oleinik added a comment - I created four new tests covering methods of Thread, ThreadLocal and ThreadGroup: api.kernel.thread.ExcptHandlerTest.ExcptHandlerTest covers Thread's UncaughtExceptionHandler and DefaultUncaughtExceptionHandler. api.kernel.thread.StackTraceTest.StackTraceTest covers Thread's getStackTrace() and getAllStackTraces(). api.kernel.thread.ThreadLocalTest.ThreadLocalTest covers ThreadLocal. api.kernel.threadgroup.EnumerateTest covers Thread's and ThreadGroup's enumerate(). Test set-up. 1. Unpack src_4_thrd_tests_19_03_07.zip into test suite root directory in which there is src/ directory with existing tests). This should create new directories and test files. 2. Add these lines to run.api.kernel.xml under <target name="run.api.kernel.thread">: <run-pjava-test test="api.kernel.thread.ExcptHandlerTest.ExcptHandlerTest" /> <run-pjava-test test="api.kernel.thread.ThreadLocalTest.ThreadLocalTest" /> <run-pjava-test test="api.kernel.thread.StackTraceTest.StackTraceTest" /> <run-pjava-test test="api.kernel.threadgroup.EnumerateTest" /> One can specify test parameters described in the comments in the beginning of each test, but, I think, default parameters specified in the tests are sufficient. 3. Compile and run tests as described in README: cd <TEST SUITE ROOT DIR> ant cd run ant -f run.api.kernel.xml run.api.kernel.thread Test run results. i tried the tests on WinXP, HDK: svn = r519348, (Mar 17 2007), Windows/ia32/msvc 1310, release build (test execution time - 5 minutes): api.kernel.thread.ExcptHandlerTest.ExcptHandlerTest exits abnormally on StackOverflowError. If remove the case causing StackOverflowError, the test abnormally completes on ~110 iteration showing that it took all available in system memory. Passes on RI. api.kernel.thread.StackTraceTest.StackTraceTest failed on 161'st iteration with: Exception during test execution: java.util.NoSuchElementException java.util.NoSuchElementException at java.util.WeakHashMap$HashIterator.next(WeakHashMap.java:162) at java.util.AbstractCollection.toArray(AbstractCollection.java) at java.lang.ThreadGroup.copyThreads(ThreadGroup.java) at java.lang.ThreadGroup.enumerate(ThreadGroup.java:456) at java.lang.ThreadGroup.enumerate(ThreadGroup.java:472) at java.lang.ThreadGroup.enumerate(ThreadGroup.java) at java.lang.Thread.getAllStackTraces(Thread.java) Passes on RI. api.kernel.thread.ThreadLocalTest.ThreadLocalTest failed on the 1-st iteration with test failure log message: Thread ID=3775 checked value of thread ID=943's instance ThreadLocal variable: get() returned wrong object null instead of org.apache.harmony.test.reliability. api.kernel.thread.ThreadLocalTest.SomeObject@20585688 that signals that ThreadLocal's get sometimes return null instead of previously set object. Fails on RI with similar message. I failed to find a mistake in the test... api.kernel.threadgroup.EnumerateTest hangs on ~150 iteration, but, runs OK if the test operates only with non-daemon Threads and ThreadGroups. Passes on RI.
          Hide
          Oleg Oleinik added a comment -

          Decided to share vision of reliability testing according to which the suite was/is (and, probably, should be) created.

          Attached TestDevGuide.txt describes what is tested by reliability tests, general test requirements, approach to test development.

          Show
          Oleg Oleinik added a comment - Decided to share vision of reliability testing according to which the suite was/is (and, probably, should be) created. Attached TestDevGuide.txt describes what is tested by reliability tests, general test requirements, approach to test development.
          Hide
          Svetlana Konovalova added a comment -

          I'm attaching the updated and reviewed version of the Reliability Test Development Guide.

          Oleg, thanks a lot for your help!

          Cheers,
          Sveta

          Show
          Svetlana Konovalova added a comment - I'm attaching the updated and reviewed version of the Reliability Test Development Guide. Oleg, thanks a lot for your help! Cheers, Sveta
          Hide
          tatyana doubtsova added a comment -

          Please, find attached the exclude list implementation for relaibility test suite.
          Current conetnt of x-list - due to failures investigation.
          Also there is transformation of tests results to Junit format - for easier test reports parsing and unification.
          ant-contrib required.

          Show
          tatyana doubtsova added a comment - Please, find attached the exclude list implementation for relaibility test suite. Current conetnt of x-list - due to failures investigation. Also there is transformation of tests results to Junit format - for easier test reports parsing and unification. ant-contrib required.
          Hide
          tatyana doubtsova added a comment -

          Just unzip xlist_implementation.zip into

          {testsuite}

          /run directory. the archive contains 2 files:
          exclude - the list of the tests that currently fail
          run.xml - updated

          Show
          tatyana doubtsova added a comment - Just unzip xlist_implementation.zip into {testsuite} /run directory. the archive contains 2 files: exclude - the list of the tests that currently fail run.xml - updated
          Hide
          Aleksey Ignatenko added a comment -

          Added 4 tests in src4_zip1_clsload3_tests.zip

          • on api.zip/jars
          • 3 on classloading/unloading
          Show
          Aleksey Ignatenko added a comment - Added 4 tests in src4_zip1_clsload3_tests.zip on api.zip/jars 3 on classloading/unloading
          Hide
          Aleksey Ignatenko added a comment -

          Contributed 9newreltests.zip:

          + 9 new reliablity tests on different areas:

          • classloading/unloading
          • thread
          • util
          • stack
          • finalization
          • gc
          Show
          Aleksey Ignatenko added a comment - Contributed 9newreltests.zip: + 9 new reliablity tests on different areas: classloading/unloading thread util stack finalization gc
          Hide
          Aleksey Ignatenko added a comment -

          The order of patching initial contribution with additional tests is the following:
          1. unpack reliability_test_suite.zip
          2. unpack and add src_4_thrd_tests_19_03_07.zip
          3. unpack and add src4_zip1_clsload3_tests.zip
          4. unpack and add 9newreltests.zip

          All xml files are to be written over old ones in the order above.

          Show
          Aleksey Ignatenko added a comment - The order of patching initial contribution with additional tests is the following: 1. unpack reliability_test_suite.zip 2. unpack and add src_4_thrd_tests_19_03_07.zip 3. unpack and add src4_zip1_clsload3_tests.zip 4. unpack and add 9newreltests.zip All xml files are to be written over old ones in the order above.
          Hide
          Aleksey Ignatenko added a comment -

          Added +1 reliablity test on j.l.management package.
          To run one need [to copy above previously applied patches.

          Show
          Aleksey Ignatenko added a comment - Added +1 reliablity test on j.l.management package. To run one need [to copy above previously applied patches.
          Hide
          Aleksey Ignatenko added a comment -

          unpack and copy jlmanagementtest.zip.
          to run this test one need to execute:
          "ant -f run.xml run.api.kernel.management"

          Show
          Aleksey Ignatenko added a comment - unpack and copy jlmanagementtest.zip. to run this test one need to execute: "ant -f run.xml run.api.kernel.management"
          Hide
          Pavel Pervov added a comment -

          Aleksey,

          Please, repack all these archives into single one. Not it only required to track order of unpacking, root folders differ in all these archives and it is required to move files around to construct working copy.

          Show
          Pavel Pervov added a comment - Aleksey, Please, repack all these archives into single one. Not it only required to track order of unpacking, root folders differ in all these archives and it is required to move files around to construct working copy.
          Hide
          Pavel Pervov added a comment -

          Is it possible to run single test?

          Show
          Pavel Pervov added a comment - Is it possible to run single test?
          Hide
          Pavel Pervov added a comment -

          Reliability test runner should first detect presence of java to test before blindly executing it and report if executable is not found.

          Show
          Pavel Pervov added a comment - Reliability test runner should first detect presence of java to test before blindly executing it and report if executable is not found.
          Hide
          Aleksey Ignatenko added a comment -

          Updated suit, attached reliability_v1.zip
          What is done:
          1. Removed Win line ends ('^M') in tests
          2. Updated style (revomed tabs, alinged front indents)
          3. Fixed 8 issues in tests:

          • api.kernel.thread.ThreadLocalTest (forced at least 1 iteration as test could run 0 iterations)
          • api.zip.ZipEntryIterateThreadingTest (made number of threads lower to satisfy 1 iteration 5 minutes run requirement for 1 test)
          • api.io.IOTest (updated right version of jar file being tested)
          • ArrayElemFinalizationTest (updated invalid test pass criterion)
          • api.nio.buffers.ByteBufferallocateTest (fixed incorrect test behaviour)
          • api.kernel.hooks.AddRmPropertiesHooksTest (added missed IAE catch)
          • ClassAttributesTest.java (updated test to pass on linux - eliminated XServer requirement)
          • ClassReflectionTest.java (updated test to pass on linux - eliminated XServer requirement)

          See http://wiki.apache.org/harmony/Reliability_Tests_on_DRLVM for details of incorrect tests.

          Show
          Aleksey Ignatenko added a comment - Updated suit, attached reliability_v1.zip What is done: 1. Removed Win line ends ('^M') in tests 2. Updated style (revomed tabs, alinged front indents) 3. Fixed 8 issues in tests: api.kernel.thread.ThreadLocalTest (forced at least 1 iteration as test could run 0 iterations) api.zip.ZipEntryIterateThreadingTest (made number of threads lower to satisfy 1 iteration 5 minutes run requirement for 1 test) api.io.IOTest (updated right version of jar file being tested) ArrayElemFinalizationTest (updated invalid test pass criterion) api.nio.buffers.ByteBufferallocateTest (fixed incorrect test behaviour) api.kernel.hooks.AddRmPropertiesHooksTest (added missed IAE catch) ClassAttributesTest.java (updated test to pass on linux - eliminated XServer requirement) ClassReflectionTest.java (updated test to pass on linux - eliminated XServer requirement) See http://wiki.apache.org/harmony/Reliability_Tests_on_DRLVM for details of incorrect tests.
          Hide
          George Timoshenko added a comment -

          Here is the fixed DekkerTest.

          I fixed 2 problems:

          1) method 'run' in case of failure returns without releasing the critical section
          So the test hangs insted of correct failure

          2) as the second thread can work beween

          base.commonVar = vars[threadNum];

          and

          if (base.commonVar != vars[threanNum])

          this comparison must be changed in the following way:

          if (base.commonVar != vars[0] &&
          base.commonVar != vars[1])
          {....

          Show
          George Timoshenko added a comment - Here is the fixed DekkerTest. I fixed 2 problems: 1) method 'run' in case of failure returns without releasing the critical section So the test hangs insted of correct failure 2) as the second thread can work beween base.commonVar = vars [threadNum] ; and if (base.commonVar != vars [threanNum] ) this comparison must be changed in the following way: if (base.commonVar != vars [0] && base.commonVar != vars [1] ) {....
          Hide
          tatyana doubtsova added a comment -

          Goerge, I agree with your DekkerTest correction. Thank you for pointing it out.

          Show
          tatyana doubtsova added a comment - Goerge, I agree with your DekkerTest correction. Thank you for pointing it out.
          Hide
          Stepan Mishura added a comment -

          The contribution was committed to BTI branch 2.0 at r545554.
          Please check that it looks as you expected.

          Show
          Stepan Mishura added a comment - The contribution was committed to BTI branch 2.0 at r545554. Please check that it looks as you expected.
          Hide
          Ivaschenko, Aleksei V added a comment -

          Hello,
          I've created a little patch (fix.test.reports.patch) that fixes bug in test reports generation algorithm in reliability test suite. This bug lead XSLT transformation (xml report -> plain text for notification) to fail.

          Show
          Ivaschenko, Aleksei V added a comment - Hello, I've created a little patch (fix.test.reports.patch) that fixes bug in test reports generation algorithm in reliability test suite. This bug lead XSLT transformation (xml report -> plain text for notification) to fail.
          Show
          Aleksey Ignatenko added a comment - to get tests: svn co -r HEAD http://svn.apache.org/repos/asf/harmony/enhanced/buildtest/branches/2.0/tests/reliability
          Hide
          Ivaschenko, Aleksei V added a comment -

          fixed typo in patch fix.test.reports.patch

          Show
          Ivaschenko, Aleksei V added a comment - fixed typo in patch fix.test.reports.patch
          Hide
          tatyana doubtsova added a comment -

          This is updated fix.test.report.patch which also comtains old report directory removing

          Show
          tatyana doubtsova added a comment - This is updated fix.test.report.patch which also comtains old report directory removing

            People

            • Assignee:
              Stepan Mishura
              Reporter:
              Oleg Oleinik
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development