Index: src/docbkx/developer.xml
===================================================================
--- src/docbkx/developer.xml (revision 1390622)
+++ src/docbkx/developer.xml (working copy)
@@ -356,31 +356,35 @@
Unit Tests
-HBase unit tests are subdivided into three categories: small, medium and large, with
-corresponding JUnit categories:
+HBase unit tests are subdivided into four categories: small, medium, large, and
+integration with corresponding JUnit categories:
SmallTests, MediumTests,
-LargeTests. JUnit categories are denoted using java annotations
-and look like this in your unit test code.
+LargeTests, IntegrationTests.
+JUnit categories are denoted using java annotations and look like this in your unit test code.
...
@Category(SmallTests.class)
public class TestHRegionInfo {
-
@Test
public void testCreateHRegionInfoName() throws Exception {
// ...
}
-...
- @org.junit.Rule
- public org.apache.hadoop.hbase.ResourceCheckerJUnitRule cu =
- new org.apache.hadoop.hbase.ResourceCheckerJUnitRule();
}
-The above example shows how to mark a test as belonging to the small category. The @org.junit.Rule
-lines on the end are also necessary. Add them to each new unit test file. They are needed by the categorization process.
-HBase uses a patched maven surefire plugin and maven profiles to implement its unit test characterizations.
+The above example shows how to mark a unit test as belonging to the small category.
+All unit tests in HBase have a categorization.
+
+The first three categories, small, medium, and large are for tests run when
+you type $ mvn test; i.e. these three categorizations are for
+HBase unit tests. The integration category is for not for unit tests but for integration
+tests. These are run when you invoke $ mvn verify. Integration tests
+are described TODO: POINTER_TO_INTEGRATION_TEST_SECTION and will not be discussed further
+in this section on HBase unit tests.
+Read the below to figure which annotation of the set small, medium, and large to
+put on your new HBase unit test.
+
-SmallTests
+SmallTestsSmallTestsSmall tests are executed in a shared JVM. We put in this category all the tests that can
be executed quickly in a shared JVM. The maximum execution time for a small test is 15 seconds,
@@ -388,7 +392,7 @@
-MediumTests
+MediumTestsMediumTestsMedium tests represent tests that must be executed
before proposing a patch. They are designed to run in less than 30 minutes altogether,
and are quite stable in their results. They are designed to last less than 50 seconds
@@ -397,13 +401,18 @@
-LargeTests
+LargeLargeTestsLarge tests are everything else. They are typically integration-like
tests, regression tests for specific bugs, timeout tests, performance tests.
They are executed before a commit on the pre-integration machines. They can be run on
the developer machine as well.
+
+HBase uses a patched maven surefire plugin and maven profiles to implement
+its unit test characterizations.
+
+Running tests
@@ -426,7 +435,7 @@
will execute small tests in a single JVM then medium and large tests in a separate JVM for each test.
Medium and large tests are NOT executed if there is an error in a small test.
Large tests are NOT executed if there is an error in a small or medium test.
-There is one report for small tests, and one report for medium and large tests if they are executed
+There is one report for small tests, and one report for medium and large tests if they are executed.
@@ -442,7 +451,8 @@
The -P localTests will remove the JUnit category effect (without this specific profile,
-the categories are taken into account). It will actually use the official release of surefire
+the categories are taken into account). It will actually use the official release of maven surefire,
+rather than our custom surefire plugin,
and the old connector (The HBase build uses a patched version of the maven surefire plugin).
Each junit tests is executed in a separate JVM (A fork per test class). There is no
parallelization when localTests profile is set. You will see a new message at the end of the
@@ -460,6 +470,25 @@
+
+Running tests faster
+
+By default, $ mvn test -P runAllTests runs 5 tests in parallel.
+It can be increased on a developer's machine. Allowing that you can have 2
+tests in parallel per core, and you need about 2Gb of memory per test,
+if you have a 8 cores and 24Gb box, you can have 16 tests in parallel.
+To run 16 in parallel, do this:
+mvn test -P runAllTests -Dsurefire.secondPartThreadCount=16.
+To increase the speed, you can as well use a ramdisk. You will need 2Gb of memory
+to run all tests. You will also need to delete the files between two test run.
+The typical way to configure a ramdisk on Linux is:
+$ sudo mkdir /ram2G
+sudo mount -t tmpfs -o size=2048M tmpfs /ram2G
+You can then use it to run all HBase tests with the command:
+mvn test -P runAllTests -Dsurefire.secondPartThreadCount=16 -Dtest.build.data.basedirectory=/ram2G
+
+
+
hbasetests.shIt's also possible to use the script hbasetests.sh. This script runs the medium and
@@ -474,7 +503,27 @@
second time, in a separate jvm and without parallelisation.
+
+Test Resource CheckerTest Resource Checker
+
+A custom Maven SureFire plugin listener checks a number of resources before
+and after each HBase unit test runs and logs its findings at the end of the test
+output files which can be found in target/surefire-reports
+per Maven module (Tests write test reports named for the test class into this directory.
+Check the *-out.txt files). The resources counted are the number
+of threads, the number of file descriptors, etc. If the number has increased, it adds
+a LEAK? comment in the logs. As you can have an HBase instance
+running in the background, some threads can be deleted/created without any specific
+action in the test. However, if the test does not work as expected, or if the test
+should not impact these resources, it's worth checking these log lines
+...hbase.ResourceChecker(157): before... and
+...hbase.ResourceChecker(157): after.... For example:
+
+2012-09-26 09:22:15,315 INFO [pool-1-thread-1] hbase.ResourceChecker(157): after: regionserver.TestColumnSeeking#testReseeking Thread=65 (was 65), OpenFileDescriptor=107 (was 107), MaxFileDescriptor=10240 (was 10240), ConnectionCount=1 (was 1)
+
+
+
Writing Tests
@@ -531,7 +580,6 @@
-