HBase
  1. HBase
  2. HBASE-5636

TestTableMapReduce doesn't work properly.

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.92.1, 0.94.0
    • Fix Version/s: 0.94.0, 0.95.0
    • Component/s: test
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      No map function is called because there are no test data put before test starts.

      The following three tests are in the same situation:

      • org.apache.hadoop.hbase.mapred.TestTableMapReduce
      • org.apache.hadoop.hbase.mapreduce.TestTableMapReduce
      • org.apache.hadoop.hbase.mapreduce.TestMulitthreadedTableMapper
      1. HBASE-5636.patch
        26 kB
        Takuya Ueshin
      2. HBASE-5636-v2.patch
        26 kB
        Takuya Ueshin

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          Integrated in HBase-0.94-security #7 (See https://builds.apache.org/job/HBase-0.94-security/7/)
          HBASE-5663 HBASE-5636 MultithreadedTableMapper doesn't work (Takuya Ueshin) (Revision 1308354)

          Result = SUCCESS
          tedyu :
          Files :

          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/mapreduce/MultithreadedTableMapper.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapred/TestTableMapReduce.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestMulitthreadedTableMapper.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestMultithreadedTableMapper.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduce.java
          Show
          Hudson added a comment - Integrated in HBase-0.94-security #7 (See https://builds.apache.org/job/HBase-0.94-security/7/ ) HBASE-5663 HBASE-5636 MultithreadedTableMapper doesn't work (Takuya Ueshin) (Revision 1308354) Result = SUCCESS tedyu : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/mapreduce/MultithreadedTableMapper.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapred/TestTableMapReduce.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestMulitthreadedTableMapper.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestMultithreadedTableMapper.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduce.java
          Hide
          Ted Yu added a comment -

          You're suggesting we refine the logic of finding failed tests in dev-support/test-patch.sh ?

          That sounds good.

          Show
          Ted Yu added a comment - You're suggesting we refine the logic of finding failed tests in dev-support/test-patch.sh ? That sounds good.
          Hide
          Nicolas Spiegelberg added a comment -

          @Ted: we should list the tests as expected failures without Hadoop 1.0.2 then. Hadoop QA should be able to point out when a new patch introduces test failures & avoid forcing the user to understand the context of features that other people are working on.

          Show
          Nicolas Spiegelberg added a comment - @Ted: we should list the tests as expected failures without Hadoop 1.0.2 then. Hadoop QA should be able to point out when a new patch introduces test failures & avoid forcing the user to understand the context of features that other people are working on.
          Hide
          Ted Yu added a comment -

          A careful check of the failed test:
          https://builds.apache.org/job/PreCommit-HBASE-Build/1369/testReport/junit/org.apache.hadoop.hbase.mapreduce/TestMultithreadedTableMapper/testMultithreadedTableMapper/

          would reveal the following:

          2012-04-02 14:54:57,385 DEBUG [pool-1-thread-1] mapreduce.TableInputFormatBase(194): getSplits: split -> 24 -> asf011.sp2.ygridcore.net.:yyy,
          java.lang.NumberFormatException: For input string: "18446743988145906374"
          	at java.lang.NumberFormatException.forInputString(NumberFormatException.java:48)
          	at java.lang.Long.parseLong(Long.java:422)
          	at java.lang.Long.parseLong(Long.java:468)
          	at org.apache.hadoop.util.ProcfsBasedProcessTree.constructProcessInfo(ProcfsBasedProcessTree.java:413)
          	at org.apache.hadoop.util.ProcfsBasedProcessTree.getProcessTree(ProcfsBasedProcessTree.java:148)
          	at org.apache.hadoop.util.LinuxResourceCalculatorPlugin.getProcResourceValues(LinuxResourceCalculatorPlugin.java:401)
          	at org.apache.hadoop.mapred.Task.initialize(Task.java:536)
          	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:353)
          	at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
          

          This should have been fixed by MAPREDUCE-3583

          Once hadoop 1.0.2 comes out, we won't see the above exceptions.

          Show
          Ted Yu added a comment - A careful check of the failed test: https://builds.apache.org/job/PreCommit-HBASE-Build/1369/testReport/junit/org.apache.hadoop.hbase.mapreduce/TestMultithreadedTableMapper/testMultithreadedTableMapper/ would reveal the following: 2012-04-02 14:54:57,385 DEBUG [pool-1-thread-1] mapreduce.TableInputFormatBase(194): getSplits: split -> 24 -> asf011.sp2.ygridcore.net.:yyy, java.lang.NumberFormatException: For input string: "18446743988145906374" at java.lang.NumberFormatException.forInputString(NumberFormatException.java:48) at java.lang. Long .parseLong( Long .java:422) at java.lang. Long .parseLong( Long .java:468) at org.apache.hadoop.util.ProcfsBasedProcessTree.constructProcessInfo(ProcfsBasedProcessTree.java:413) at org.apache.hadoop.util.ProcfsBasedProcessTree.getProcessTree(ProcfsBasedProcessTree.java:148) at org.apache.hadoop.util.LinuxResourceCalculatorPlugin.getProcResourceValues(LinuxResourceCalculatorPlugin.java:401) at org.apache.hadoop.mapred.Task.initialize(Task.java:536) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:353) at org.apache.hadoop.mapred.Child$4.run(Child.java:255) This should have been fixed by MAPREDUCE-3583 Once hadoop 1.0.2 comes out, we won't see the above exceptions.
          Hide
          Nicolas Spiegelberg added a comment -

          @Ted: It looks like this patch is introducing intermittent unit test failures in the trunk. Can we revert or fix? https://builds.apache.org/job/PreCommit-HBASE-Build/1369/

          Show
          Nicolas Spiegelberg added a comment - @Ted: It looks like this patch is introducing intermittent unit test failures in the trunk. Can we revert or fix? https://builds.apache.org/job/PreCommit-HBASE-Build/1369/
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK-security #156 (See https://builds.apache.org/job/HBase-TRUNK-security/156/)
          HBASE-5663 HBASE-5636 MultithreadedTableMapper doesn't work (Takuya Ueshin) (Revision 1308353)

          Result = FAILURE
          tedyu :
          Files :

          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/MultithreadedTableMapper.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapred/TestTableMapReduce.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapreduce/TestMulitthreadedTableMapper.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapreduce/TestMultithreadedTableMapper.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduce.java
          Show
          Hudson added a comment - Integrated in HBase-TRUNK-security #156 (See https://builds.apache.org/job/HBase-TRUNK-security/156/ ) HBASE-5663 HBASE-5636 MultithreadedTableMapper doesn't work (Takuya Ueshin) (Revision 1308353) Result = FAILURE tedyu : Files : /hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/MultithreadedTableMapper.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapred/TestTableMapReduce.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapreduce/TestMulitthreadedTableMapper.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapreduce/TestMultithreadedTableMapper.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduce.java
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK #2703 (See https://builds.apache.org/job/HBase-TRUNK/2703/)
          HBASE-5663 HBASE-5636 MultithreadedTableMapper doesn't work (Takuya Ueshin) (Revision 1308353)

          Result = SUCCESS
          tedyu :
          Files :

          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/MultithreadedTableMapper.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapred/TestTableMapReduce.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapreduce/TestMulitthreadedTableMapper.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapreduce/TestMultithreadedTableMapper.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduce.java
          Show
          Hudson added a comment - Integrated in HBase-TRUNK #2703 (See https://builds.apache.org/job/HBase-TRUNK/2703/ ) HBASE-5663 HBASE-5636 MultithreadedTableMapper doesn't work (Takuya Ueshin) (Revision 1308353) Result = SUCCESS tedyu : Files : /hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/MultithreadedTableMapper.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapred/TestTableMapReduce.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapreduce/TestMulitthreadedTableMapper.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapreduce/TestMultithreadedTableMapper.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduce.java
          Hide
          Hudson added a comment -

          Integrated in HBase-0.94 #75 (See https://builds.apache.org/job/HBase-0.94/75/)
          HBASE-5663 HBASE-5636 MultithreadedTableMapper doesn't work (Takuya Ueshin) (Revision 1308354)

          Result = FAILURE
          tedyu :
          Files :

          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/mapreduce/MultithreadedTableMapper.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapred/TestTableMapReduce.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestMulitthreadedTableMapper.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestMultithreadedTableMapper.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduce.java
          Show
          Hudson added a comment - Integrated in HBase-0.94 #75 (See https://builds.apache.org/job/HBase-0.94/75/ ) HBASE-5663 HBASE-5636 MultithreadedTableMapper doesn't work (Takuya Ueshin) (Revision 1308354) Result = FAILURE tedyu : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/mapreduce/MultithreadedTableMapper.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapred/TestTableMapReduce.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestMulitthreadedTableMapper.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestMultithreadedTableMapper.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduce.java
          Hide
          Ted Yu added a comment -

          Integrated to trunk and 0.94.

          Thanks for the patch, Takuya.

          Show
          Ted Yu added a comment - Integrated to trunk and 0.94. Thanks for the patch, Takuya.
          Hide
          Ted Yu added a comment -

          Without patch from HBASE-5663, I saw the following in test output:

          2012-03-31 07:55:46,743 DEBUG [main] mapreduce.TableInputFormatBase(194): getSplits: split -> 24 -> 192.168.0.17:yyy,
          java.io.IOException: java.lang.NoSuchMethodException: org.apache.hadoop.mapreduce.Mapper$Context.<init>(org.apache.hadoop.conf.Configuration, org.apache.hadoop.mapred.TaskAttemptID, org.apache.hadoop.hbase.mapreduce.MultithreadedTableMapper$SubMapRecordReader, org.apache.hadoop.hbase.mapreduce.MultithreadedTableMapper$SubMapRecordWriter, org.apache.hadoop.hbase.mapreduce.TableOutputCommitter, org.apache.hadoop.hbase.mapreduce.MultithreadedTableMapper$SubMapStatusReporter, org.apache.hadoop.hbase.mapreduce.TableSplit)
            at org.apache.hadoop.hbase.mapreduce.MultithreadedTableMapper$MapRunner.<init>(MultithreadedTableMapper.java:260)
            at org.apache.hadoop.hbase.mapreduce.MultithreadedTableMapper.run(MultithreadedTableMapper.java:133)
            at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
            at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
            at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
            at java.security.AccessController.doPrivileged(Native Method)
            at javax.security.auth.Subject.doAs(Subject.java:396)
          
          Show
          Ted Yu added a comment - Without patch from HBASE-5663 , I saw the following in test output: 2012-03-31 07:55:46,743 DEBUG [main] mapreduce.TableInputFormatBase(194): getSplits: split -> 24 -> 192.168.0.17:yyy, java.io.IOException: java.lang.NoSuchMethodException: org.apache.hadoop.mapreduce.Mapper$Context.<init>(org.apache.hadoop.conf.Configuration, org.apache.hadoop.mapred.TaskAttemptID, org.apache.hadoop.hbase.mapreduce.MultithreadedTableMapper$SubMapRecordReader, org.apache.hadoop.hbase.mapreduce.MultithreadedTableMapper$SubMapRecordWriter, org.apache.hadoop.hbase.mapreduce.TableOutputCommitter, org.apache.hadoop.hbase.mapreduce.MultithreadedTableMapper$SubMapStatusReporter, org.apache.hadoop.hbase.mapreduce.TableSplit) at org.apache.hadoop.hbase.mapreduce.MultithreadedTableMapper$MapRunner.<init>(MultithreadedTableMapper.java:260) at org.apache.hadoop.hbase.mapreduce.MultithreadedTableMapper.run(MultithreadedTableMapper.java:133) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370) at org.apache.hadoop.mapred.Child$4.run(Child.java:255) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396)
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12520285/HBASE-5636-v2.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 10 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hbase.regionserver.TestColumnSeeking

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1331//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1331//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1331//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12520285/HBASE-5636-v2.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 10 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.regionserver.TestColumnSeeking Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1331//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1331//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1331//console This message is automatically generated.
          Hide
          Takuya Ueshin added a comment -

          I attached a patch v2.
          Maybe I forgot --no-prefix option of git-diff command.

          Show
          Takuya Ueshin added a comment - I attached a patch v2. Maybe I forgot --no-prefix option of git-diff command.
          Hide
          Ted Yu added a comment -

          Changes for mapred/TestTableMapReduce.java and mapreduce/TestTableMapReduce.java couldn't apply cleanly.
          Can you upload a new patch for trunk ?

          Thanks

          Show
          Ted Yu added a comment - Changes for mapred/TestTableMapReduce.java and mapreduce/TestTableMapReduce.java couldn't apply cleanly. Can you upload a new patch for trunk ? Thanks
          Hide
          Takuya Ueshin added a comment -

          Oh, I'm very sorry.

          Here is the diff of TestMulitthreadedTableMapper.java:

          diff --git a/src/test/java/org/apache/hadoop/hbase/mapreduce/TestMulitthreadedTableMapper.java b/src/test/java/org/apache/hadoop/hbase/mapreduce/TestMulitthreadedTableMapper.java
          index cc5b1df..ad34dd2 100644
          --- a/src/test/java/org/apache/hadoop/hbase/mapreduce/TestMulitthreadedTableMapper.java
          +++ b/src/test/java/org/apache/hadoop/hbase/mapreduce/TestMulitthreadedTableMapper.java
          @@ -19,6 +19,7 @@ package org.apache.hadoop.hbase.mapreduce;
           
           import java.io.File;
           import java.io.IOException;
          +import java.util.Iterator;
           import java.util.Map;
           import java.util.NavigableMap;
           
          @@ -28,7 +29,6 @@ import org.apache.hadoop.conf.Configuration;
           import org.apache.hadoop.fs.FileUtil;
           import org.apache.hadoop.fs.Path;
           import org.apache.hadoop.hbase.*;
          -import org.apache.hadoop.hbase.client.HBaseAdmin;
           import org.apache.hadoop.hbase.client.HTable;
           import org.apache.hadoop.hbase.client.Put;
           import org.apache.hadoop.hbase.client.Result;
          @@ -56,19 +56,17 @@ public class TestMulitthreadedTableMapper {
             private static final Log LOG = LogFactory.getLog(TestMulitthreadedTableMapper.class);
             private static final HBaseTestingUtility UTIL =
                 new HBaseTestingUtility();
          -  static final String MULTI_REGION_TABLE_NAME = "mrtest";
          +  static final byte[] MULTI_REGION_TABLE_NAME = Bytes.toBytes("mrtest");
             static final byte[] INPUT_FAMILY = Bytes.toBytes("contents");
             static final byte[] OUTPUT_FAMILY = Bytes.toBytes("text");
             static final int    NUMBER_OF_THREADS = 10;
           
             @BeforeClass
             public static void beforeClass() throws Exception {
          -    HTableDescriptor desc = new HTableDescriptor(MULTI_REGION_TABLE_NAME);
          -    desc.addFamily(new HColumnDescriptor(INPUT_FAMILY));
          -    desc.addFamily(new HColumnDescriptor(OUTPUT_FAMILY));
               UTIL.startMiniCluster();
          -    HBaseAdmin admin = new HBaseAdmin(UTIL.getConfiguration());
          -    admin.createTable(desc, HBaseTestingUtility.KEYS);
          +    HTable table = UTIL.createTable(MULTI_REGION_TABLE_NAME, new byte[][] {INPUT_FAMILY, OUTPUT_FAMILY});
          +    UTIL.createMultiRegions(table, INPUT_FAMILY);
          +    UTIL.loadTable(table, INPUT_FAMILY);
               UTIL.startMiniMapReduceCluster();
             }
           
          @@ -149,7 +147,7 @@ public class TestMulitthreadedTableMapper {
                     IdentityTableReducer.class, job);
                 FileOutputFormat.setOutputPath(job, new Path("test"));
                 LOG.info("Started " + Bytes.toString(table.getTableName()));
          -      job.waitForCompletion(true);
          +      assertTrue(job.waitForCompletion(true));
                 LOG.info("After map/reduce completion");
                 // verify map-reduce results
                 verify(Bytes.toString(table.getTableName()));
          @@ -203,7 +201,10 @@ public class TestMulitthreadedTableMapper {
               scan.addFamily(OUTPUT_FAMILY);
               ResultScanner scanner = table.getScanner(scan);
               try {
          -      for (Result r : scanner) {
          +      Iterator<Result> itr = scanner.iterator();
          +      assertTrue(itr.hasNext());
          +      while(itr.hasNext()) {
          +        Result r = itr.next();
                   if (LOG.isDebugEnabled()) {
                     if (r.size() > 2 ) {
                       throw new IOException("Too many results, expected 2 got " +
          
          Show
          Takuya Ueshin added a comment - Oh, I'm very sorry. Here is the diff of TestMulitthreadedTableMapper.java: diff --git a/src/test/java/org/apache/hadoop/hbase/mapreduce/TestMulitthreadedTableMapper.java b/src/test/java/org/apache/hadoop/hbase/mapreduce/TestMulitthreadedTableMapper.java index cc5b1df..ad34dd2 100644 --- a/src/test/java/org/apache/hadoop/hbase/mapreduce/TestMulitthreadedTableMapper.java +++ b/src/test/java/org/apache/hadoop/hbase/mapreduce/TestMulitthreadedTableMapper.java @@ -19,6 +19,7 @@ package org.apache.hadoop.hbase.mapreduce; import java.io.File; import java.io.IOException; +import java.util.Iterator; import java.util.Map; import java.util.NavigableMap; @@ -28,7 +29,6 @@ import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.FileUtil; import org.apache.hadoop.fs.Path; import org.apache.hadoop.hbase.*; -import org.apache.hadoop.hbase.client.HBaseAdmin; import org.apache.hadoop.hbase.client.HTable; import org.apache.hadoop.hbase.client.Put; import org.apache.hadoop.hbase.client.Result; @@ -56,19 +56,17 @@ public class TestMulitthreadedTableMapper { private static final Log LOG = LogFactory.getLog(TestMulitthreadedTableMapper.class); private static final HBaseTestingUtility UTIL = new HBaseTestingUtility(); - static final String MULTI_REGION_TABLE_NAME = "mrtest"; + static final byte[] MULTI_REGION_TABLE_NAME = Bytes.toBytes("mrtest"); static final byte[] INPUT_FAMILY = Bytes.toBytes("contents"); static final byte[] OUTPUT_FAMILY = Bytes.toBytes("text"); static final int NUMBER_OF_THREADS = 10; @BeforeClass public static void beforeClass() throws Exception { - HTableDescriptor desc = new HTableDescriptor(MULTI_REGION_TABLE_NAME); - desc.addFamily(new HColumnDescriptor(INPUT_FAMILY)); - desc.addFamily(new HColumnDescriptor(OUTPUT_FAMILY)); UTIL.startMiniCluster(); - HBaseAdmin admin = new HBaseAdmin(UTIL.getConfiguration()); - admin.createTable(desc, HBaseTestingUtility.KEYS); + HTable table = UTIL.createTable(MULTI_REGION_TABLE_NAME, new byte[][] {INPUT_FAMILY, OUTPUT_FAMILY}); + UTIL.createMultiRegions(table, INPUT_FAMILY); + UTIL.loadTable(table, INPUT_FAMILY); UTIL.startMiniMapReduceCluster(); } @@ -149,7 +147,7 @@ public class TestMulitthreadedTableMapper { IdentityTableReducer.class, job); FileOutputFormat.setOutputPath(job, new Path("test")); LOG.info("Started " + Bytes.toString(table.getTableName())); - job.waitForCompletion(true); + assertTrue(job.waitForCompletion(true)); LOG.info("After map/reduce completion"); // verify map-reduce results verify(Bytes.toString(table.getTableName())); @@ -203,7 +201,10 @@ public class TestMulitthreadedTableMapper { scan.addFamily(OUTPUT_FAMILY); ResultScanner scanner = table.getScanner(scan); try { - for (Result r : scanner) { + Iterator<Result> itr = scanner.iterator(); + assertTrue(itr.hasNext()); + while(itr.hasNext()) { + Result r = itr.next(); if (LOG.isDebugEnabled()) { if (r.size() > 2 ) { throw new IOException("Too many results, expected 2 got " +
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12520270/HBASE-5636.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 10 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1329//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12520270/HBASE-5636.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 10 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1329//console This message is automatically generated.
          Hide
          Ted Yu added a comment -

          Since TestMultithreadedTableMapper.java is a new file, can you outline the changes you made on top of TestMulitthreadedTableMapper ?
          I noticed this:

          +    UTIL.loadTable(table, INPUT_FAMILY);
          
          Show
          Ted Yu added a comment - Since TestMultithreadedTableMapper.java is a new file, can you outline the changes you made on top of TestMulitthreadedTableMapper ? I noticed this: + UTIL.loadTable(table, INPUT_FAMILY);
          Hide
          Takuya Ueshin added a comment -

          I attached a patch file for trunk.
          This includes patches for TestMulitthreadedTableMapper.

          Show
          Takuya Ueshin added a comment - I attached a patch file for trunk. This includes patches for TestMulitthreadedTableMapper.
          Hide
          Takuya Ueshin added a comment -

          I'm sorry, TestMulitthreadedTableMapper was not included in 0.92.x.

          Show
          Takuya Ueshin added a comment - I'm sorry, TestMulitthreadedTableMapper was not included in 0.92.x.
          Hide
          Takuya Ueshin added a comment -

          I filed the new issue of the bug.

          By the way, the TestCase name 'TestMulitthreadedTableMapper' is typo, isn't it?
          I will rename it to 'TestMultithreadedTableMapper'.

          Show
          Takuya Ueshin added a comment - I filed the new issue of the bug. By the way, the TestCase name 'TestMulitthreadedTableMapper' is typo, isn't it? I will rename it to 'TestMultithreadedTableMapper'.
          Hide
          Takuya Ueshin added a comment -

          I found another bug when I was working on HBASE-5636.

          Show
          Takuya Ueshin added a comment - I found another bug when I was working on HBASE-5636 .
          Hide
          Takuya Ueshin added a comment -

          Yes, I'm going to work this week or weekend.

          Show
          Takuya Ueshin added a comment - Yes, I'm going to work this week or weekend.
          Hide
          Enis Soztutar added a comment -

          Very good find. It seems that HBASE-4503 might have caused it. Do you plan to work on this?

          Show
          Enis Soztutar added a comment - Very good find. It seems that HBASE-4503 might have caused it. Do you plan to work on this?

            People

            • Assignee:
              Takuya Ueshin
              Reporter:
              Takuya Ueshin
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development