Here's a path that should test bulk loads.

It starts an MR job that creates linked chains

The format of rows is like this:

rk -> Long
d:<< Chain Id >> -> Random Data.
l:<< Chain Id >> -> Row Key of the next link in the chain
s:<< Chain Id >> -> The step in the chain that his link is.

All chains start on row 0.

So we create these chains and then walk over them in an MR job.

• Added lots of java doc comments.
• Added a delete table to the initial setup to allow running this repeatedly on the same cluster.

First read of the patch looks good. Will take a second look shortly.

+1, v1 looks good to me

Version that should work with security.

Thanks for the reviews.

IntegrationTestBulkLoad.java is missing the license header.

+          if (lc.getRk() != 0l) throw new RuntimeException("Chains should all start at 0 rk");

nit: use '0L' instead of '0l' which looks a lot like '01'.

+    conf.setStrings("io.serializations", conf.get("io.serializations"),
+        KeyValueSerialization.class.getName());

Why does this become necessary (or, rather, why wasn't it necessary before)?

+
+    void setFirstRow(byte[] userInput);
+
+    void setLastRow(byte[] userInput);

nit: please javadoc interface methods.

Looks like a nice addition, Elliott. Once you get this merged, would you mind adding launch instructions and any relevant parameters to the release notes?

• Changed 0l -> 0L
• Added header
• Moved initCredentials to HFileOutputFormat.configureIncrementalLoad

Not sure why the io serializations weren't there before. Outputing KeyValues null pointers for me without it. Maybe everyone was just adding that during testing ?

Minor issue – in

runLinkedListMRJob(int iteration)

, your delete comes at the end of the function. Should there be an issue and the test fails, it will not reach that point and there will be leftover cruft that could affect other tests. One way to get around this is to use a try-catch-finally (or just try-finally) pattern, with the deletion being in the 'finally' portion.

Granted, you only have one test, but if you write more, or if you run this on an actual cluster, cleanup becomes important.

Not sure why the io serializations weren't there before. Outputing KeyValues null pointers for me without it. Maybe everyone was just adding that during testing ?

Very strange indeed. Does TableOutputFormat suffer the same pain? Perhaps this can be set in TableMapReduceUtils instead of on a per-output formatter basis? I wonder if this is related to the refactoring stack was doing around these interfaces?

+1

Added a timestamp to the path name so left over files can be correlated to error's in tests.

Does TableOutputFormat suffer the same pain?

TableOutputFormat adds a serializer for mutations.

+1. Skimmed the patch looks great. Can you add your first comment to the class javadoc so that users do not have to read this jira or whole code.

Here's what I'm planning to commit soon. It has an expanded Javadoc like Enis was asking for.

Committed revision 1491657.

Integrated in HBase-TRUNK #4171 (See https://builds.apache.org/job/HBase-TRUNK/4171/)
HBASE-8672 Create an Integration test for Bulk Loads (Revision 1491656)

Result = SUCCESS
eclark :
Files :

• /hbase/trunk/hbase-it/src/test/java/org/apache/hadoop/hbase/mapreduce/IntegrationTestBulkLoad.java
• /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java
• /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/RegionSplitter.java
• /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java

Integrated in hbase-0.95 #235 (See https://builds.apache.org/job/hbase-0.95/235/)
HBASE-8672 Create an Integration test for Bulk Loads (Revision 1491657)

Result = FAILURE
eclark :
Files :

• /hbase/branches/0.95/hbase-it/src/test/java/org/apache/hadoop/hbase/mapreduce/IntegrationTestBulkLoad.java
• /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java
• /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/util/RegionSplitter.java
• /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java