[HBASE-2615] M/R on bulk imported tables

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.20.3, 0.20.4
• Fix Version/s: 0.20.5, 0.90.0

Environment

os.arch=amd64; os.version=2.6.9-67.ELsmp; java.version=1.6.0_15; java.vendor=Sun Microsystems Inc.

• Hadoop Flags: Reviewed
• Assignee: Michael Stack
• Reporter: Azza Abouzeid
• Created: 26/May/10 16:40
• Updated: 12/Oct/12 06:15
• Resolved: 06/Jun/10 06:51

Description

We are bulk importing using loadtable.rb and running M/R jobs using HBase as input.

We're taking the following steps:
1a. Load HBase with a M/R job using the normal API.
OR
1b. Load HBase with bulk import.

THEN

2a. Using the shell, do a "count" over the table.
OR
2b. Run a M/R job that scans the whole HBase table (and nothing else).

Of the 4 combos, 3 are fine: 1a+2a, 1a+2b, 1b+2a. We're having trouble with 1b+2b. When we run the M/R job, it doesn't seem to read in any records, but there are no explicit errors in either the Hadoop or HBase logs.

Any ideas on what might be wrong with the bulk import to cause this problem? We confirmed this problem exists in both hbase-0.20.3 and hbase-0.20.4.

We have created dummy data (see attached). This is the test case:

After loading the data into HDFS. In hbase shell:
create 'tiny', 'values'

Execute:

{HBASE-HOME}/bin/hbase org.jruby.Main {HBASE-HOME}

/bin/loadtable.rb tiny tinytable

Then run the simple row counter

{HADOOP-HOME}

/bin/hadoop jar

{HBASE-HOME}

/hbase-0.20.x.jar rowcounter tiny values

Notice that map input records read is always zero. We confirmed that other mapreduce jobs do not execute the map function at all, always returning 0 records.

We also ran a major_compaction of all Hbase tables (.META. and .ROOT. as well) but this did not fix the problem.