Reliability Test Development Guide. 1. VISION OF RELIABILITY TESTING. 1.1. DEFINITION. By reliability we mean ability of runtime correctly execute workloads for given (assuming large) period of time (interpretation of definition by IEEE). 1.2 GOAL. Create a set of tests which: 1. Are small and easy to analyze failure cause (as opposed to large application). 2. Being executed for specified time reveal: * accumulative (resource leakage) and * intermittent bugs in runtime. 3. Cover all “risky” API areas and key VM/JIT mechanisms. 1.3 SOURCES OF POSSIBLE BUGS. * C Heap memory leakage (‘new‘ and no ‘delete’) * C Heap memory fragmentation * Handle leakage (not freed handlers of files, sockets, graphics resources) * Dynamic tables whose content is changed during runtime work – items are added, but not removed as expected. * Java Heap memory leakage (Java objects leakage) * Java Heap memory fragmentation (for example Large Object Space) * Finalization failures * Wrong thread synchronization in thread safe API * Internal race conditions * Wild pointers * Root set enumeration failures * Write barriers related bugs * Other 2. GENERAL TEST REQUIREMENTS. - Each test is repeated in a cycle in the same JVM for the specified time. - Each iteration contains dedicated sequence of steps which stimulate possible leakage or intermittent failure via intensive usage of target API or stimulating target runtime mechanisms. - The test should be considered PASSED if: * there is no crash or hang * the test successfully passes all the checks inserted in the test FAILED otherwise. - The test should be as small as possible and appropriate. - The test should avoid stimulation of other possible sources of unreliability. - System.gc() should be explicitly called after each test iteration. - Test execution time is determined by a parameter specified in “settings” file. Total test suite execution time should be reasonable. – Tests may be aggressive, but not stressing runtime. - Tests should be configurable – key parameters which significantly affect runtime behavior (number of concurrently running threads, opened files, etc.) should be set in “setting” file (not hardcoded in test). – Tests should be re-enterable, thread-safe, reliable (i.e. manage resources correctly – close IO, wait for threads to join(), etc.). 3. RELIABILTY TEST FRAMEWORK. Simplified model of the framework is: while (!timeout){ int result = Class.forName().newInstance().test(args); if (result == FAILED){ System.exit(result); } System.gc(); } 4. APPROACH TO TEST DEVELOPMENT. If VM/JIT mechanism is covered: 1. Learn the mechanism. 2. Create scenarios for maximum complete mechanism testing. 3. Implement, experiment (vary parameters, try on RI), add to the suite. If hy* library covered: 1. Find Java-to-native entry point methods. 2. Find J2SE public API through which the entry points methods are accessible. 3. Cover all entry point methods by tests via public J2SE API. 4. Implement scenario, experiment (vary parameters, try on RI), add to the suite. Additionally, consider creation/implementing scenarios which are most likely used in real-life (for example – known public algorithms, emulation of work of some tool). Investigate how API or mechanism implementation code works to create a test which finds bugs with higher probability. If experiments show that trivial test (implements weakest possible scenario), like: for (int i = 0; i < 1000; ++i) { new Defalter(); if (i % 100 == 0) System.gc(); } crashes HY runtime, then this this is a candidate for inclusion into the suite – it finds bug. 5. AREAS TO BE CONSIDERED FOR COVERING. Summary: * Key third-party native libraries (ICU, ZLIB) * Harmony native libraries (hy*) * Key VM mechanisms * Key JIT mechanisms * Resource leakage-risky API (IO, Networking, etc.). * Native interfaces – JNI, JVMTI, JDWP * GUI API Details: 1) ICU: hytext, nio.charset 2) ZLIB: hyzlib, hyarchive. 3) Hy* libs: hynio, etc. 4) Kernel classes (from java.lang). 5) java.lang.Thread* API. 6) Class Loading / Unloading. 7) Stack Iteration/ Exception handling. 8) JIT compilation and optimization. 9) GC (including java.lang.ref) 10) Finalization. 11) VM Verifier. 12) I/O API (inclusing nio channels) 13) Networking API 14) The rest of hyluni. 15) Reflection. 16) Util concurrent. 17) Java text BIDI 18) JAR files 19) Managing graphics resources. 20) Math (hymath, fdlibm) 21) Serialization. 22) Resource management (via Class, CLassLoader, etc.). 23) JNI, JVMTI, JDWP