RELIABILITY TEST DEVELOPMENT GUIDE 1. ABOUT THIS DOCUMENT 1.1 Purpose This document is targeted at the reliability test suite developers. The document describes general tests requirements, test development process, points out sources of possible bugs and sets areas for further covering. The information can be helpful for future development of the reliability test suite. For more information, see README stored in the test suite root directory. 1.2 Intended Audience The target audience for the document includes engineers interested in using and further developing of the reliability test suit. 1.3 Conventions This document uses the unified conventions for the DRL documentation kit. 2. OVERVIEW The reliability test suite is responsible for checking the stable work of the tested Java* run time for long period of time. 1.1. DEFINITION In the current document, by reliability we mean Java* run-time ability to execute workloads successfully for a given, conceivably long, period of time. [1] 1.2 GOAL The implementation goal is to create a set of tests that meet the following requirements: 1. To be short in contrast to long applications, since small test failure causes are easier to analyze. 2. To be executed for the given period of time in order to: - find accumulated mistake bugs, for example file and socket handler leakage, memory leakage - find intermittent bugs in the run time 3. To cover all API areas, assumed as most probable sources of mistakes mentioned above, and key VM/JIT mechanisms. 1.3 SOURCES OF POSSIBLE BUGS - C Heap memory leakage - C Heap memory fragmentation - Handle leakage of unresolved resources, such as file handlers, sockets, graphics resources - Dynamic tables with temporary added, but not removed during run-time work items - Java Heap memory leakage - Java Heap memory fragmentation, for example Large Object Space - Finalization failures - Wrong thread synchronization in thread safe API - Internal race conditions - Wild pointers - Root-set enumeration failures - Write barrier related bugs 2. GENERAL TEST REQUIREMENTS - Each test is repeated in a cycle in the same JVM during the specified period of time. - Each iteration contains dedicated sequence of steps that stimulate possible memory and resource leakage or intermittent failure via intensive usage of target API or stimulating target run-time mechanisms. - A test is PASSED if: - no crash or hang happens - the test successfully passes all the checks inserted in the test Otherwise, a test is FAILED. - A test should be as small as possible, but at the same time complete enough to test the target API or the run-time mechanism, depending on the developer’s requirements. - A test should check only one source it is targeted to. - The reliability test framework calls System.gc() explicitly after each test iteration. - Test execution time is determined by the parameter specified by the developer in the “settings” file. Total test suite execution time should be reasonable and depend on developer’s requirements. - Tests may be aggressive consuming a lot of recourses, but not stressful for the run time. - Tests must be configurable: the developer should set key parameters affecting the run-time behavior, such as the number of concurrently running threads, opened files, in the “setting file”. - Tests should be re-enterable, thread-safe, reliable, that is manage resources correctly, for example close I/O resources correctly, wait for threads to join(), etc. 3. RELIABILTY TEST FRAMEWORK Here is the example of the framework simplified model: while (!timeout){ int result = Class.forName().newInstance().test(args); if (result == FAILED){ System.exit(result); } System.gc(); } 4. TEST DEVELOPMENT PROCESS Basic steps: If the VM/JIT mechanism is covered: 1. Analyze the mechanism. 2. Create scenarios for maximum complete mechanism testing. 3. Implement the scenarios. 4. Experiment with the implemented scenarios – tests - varying parameters and trying on Reference Implementation. 5. Add the tests to the suite. If the hy* library is covered: 1. In the source code, find Java-to-native entry-point methods. 2. In the source code, find J2SE public API to access the entry-point methods. 3. Cover all entry-point methods with tests via public J2SE API. 4. Experiment with the tests varying parameters and trying on RI. 6. Add the tests to the suite. Additional Steps: 1. Consider creation/implementing scenarios that emulate the work of known public algorithms and certain tools. 2. Investigate API or the mechanism implementation code to create a test that finds bugs with the highest probability. 3. If during experiments a trivial test implementing weakest possible scenario crashes the run time, it means that the test finds a bug and you can include it into the suite. Here is the example of a trivial test: for (int i = 0; i < 1000; ++i) { new Defalter(); if (i % 100 == 0) System.gc(); } 5. AREAS FOR FURTHER COVERING Summary: - Key third-party native libraries (ICU, ZLIB) - Harmony native libraries (hy*) - Key VM mechanisms - Key JIT mechanisms - Resource leakage-risky API (IO, Networking, etc.) - Native interfaces – JNI, JVMTI, JDWP - GUI API Details: 1. ICU: hytext, nio.charset 2. ZLIB: hyzlib, hyarchive 3. Hy* libs: hynio, et 4. Kernel classes (from java.lang). 5. java.lang.Thread* API 6. Class Loading / Unloading 7. Stack Iteration/ Exception handling 8. JIT compilation and optimization 9. GC (including java.lang.ref) 10. Finalization 11. VM Verifier 12. I/O API (inclusing nio channels) 13. Networking API 14. The rest of hyluni 15. Reflection 16. Util concurrent 17. Java text BIDI 18. JAR files 19. Managing graphics resources 20. Math (hymath, fdlibm) 21. Serialization 22. Resource management (via Class, CLassLoader, etc.) 23. JNI, JVMTI, JDWP 6. REFERENCES [1] IEEE http://ieeexplore.ieee.org