HBase
  1. HBase
  2. HBASE-5663

MultithreadedTableMapper doesn't work.

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.94.0
    • Fix Version/s: 0.94.0, 0.95.0
    • Component/s: mapreduce
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      MapReduce job using MultithreadedTableMapper goes down throwing the following Exception:

      java.io.IOException: java.lang.NoSuchMethodException: org.apache.hadoop.mapreduce.Mapper$Context.<init>(org.apache.hadoop.conf.Configuration, org.apache.hadoop.mapred.TaskAttemptID, org.apache.hadoop.hbase.mapreduce.MultithreadedTableMapper$SubMapRecordReader, org.apache.hadoop.hbase.mapreduce.MultithreadedTableMapper$SubMapRecordWriter, org.apache.hadoop.hbase.mapreduce.TableOutputCommitter, org.apache.hadoop.hbase.mapreduce.MultithreadedTableMapper$SubMapStatusReporter, org.apache.hadoop.hbase.mapreduce.TableSplit)
      	at org.apache.hadoop.hbase.mapreduce.MultithreadedTableMapper$MapRunner.<init>(MultithreadedTableMapper.java:260)
      	at org.apache.hadoop.hbase.mapreduce.MultithreadedTableMapper.run(MultithreadedTableMapper.java:133)
      	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
      	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
      	at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:396)
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1083)
      	at org.apache.hadoop.mapred.Child.main(Child.java:249)
      Caused by: java.lang.NoSuchMethodException: org.apache.hadoop.mapreduce.Mapper$Context.<init>(org.apache.hadoop.conf.Configuration, org.apache.hadoop.mapred.TaskAttemptID, org.apache.hadoop.hbase.mapreduce.MultithreadedTableMapper$SubMapRecordReader, org.apache.hadoop.hbase.mapreduce.MultithreadedTableMapper$SubMapRecordWriter, org.apache.hadoop.hbase.mapreduce.TableOutputCommitter, org.apache.hadoop.hbase.mapreduce.MultithreadedTableMapper$SubMapStatusReporter, org.apache.hadoop.hbase.mapreduce.TableSplit)
      	at java.lang.Class.getConstructor0(Class.java:2706)
      	at java.lang.Class.getConstructor(Class.java:1657)
      	at org.apache.hadoop.hbase.mapreduce.MultithreadedTableMapper$MapRunner.<init>(MultithreadedTableMapper.java:241)
      	... 8 more
      

      This occured when the tasks are creating MapRunner threads.

      1. 5663+5636.txt
        29 kB
        Ted Yu
      2. HBASE-5663.patch
        4 kB
        Takuya Ueshin

        Issue Links

          Activity

          Hide
          Takuya Ueshin added a comment -

          I attached a patch file.

          Could someone please review and refactor it?
          I think this is a little dirty code but I couldn't find a better way.

          Show
          Takuya Ueshin added a comment - I attached a patch file. Could someone please review and refactor it? I think this is a little dirty code but I couldn't find a better way.
          Hide
          Ted Yu added a comment -

          We could check the Exception was caused by NoSuchMethodException.
          Since the retry is in MapRunner ctor, the above may not be needed.

          Show
          Ted Yu added a comment - We could check the Exception was caused by NoSuchMethodException. Since the retry is in MapRunner ctor, the above may not be needed.
          Hide
          Takuya Ueshin added a comment -

          No, any MapReduce Job using current MultithreadedTableMapper would not work.

          When the Mapper starts, it creates some threads and 'original' Mapper object which should be called by the thread and MapperContext object used by the original mapper.
          But because the constructor call of MapperContext through reflection is wrong, it throws the NoSuchMethodException.

          As I reported in an issue HBASE-5636, trunk version's test case TestMulitthreadedTableMapper has a bug, so please test by new version test attached in it.

          Show
          Takuya Ueshin added a comment - No, any MapReduce Job using current MultithreadedTableMapper would not work. When the Mapper starts, it creates some threads and 'original' Mapper object which should be called by the thread and MapperContext object used by the original mapper. But because the constructor call of MapperContext through reflection is wrong, it throws the NoSuchMethodException. As I reported in an issue HBASE-5636 , trunk version's test case TestMulitthreadedTableMapper has a bug, so please test by new version test attached in it.
          Hide
          Ted Yu added a comment -
          Show
          Ted Yu added a comment - So the comment in patch about hadoop versions can be removed. See the error I got when using hadoop 1.0: https://issues.apache.org/jira/browse/HBASE-5636?focusedCommentId=13243180&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13243180
          Hide
          Ted Yu added a comment -

          Combined patch for HBASE-5663 and HBASE-5636

          Show
          Ted Yu added a comment - Combined patch for HBASE-5663 and HBASE-5636
          Hide
          Ted Yu added a comment -

          I ran the following tests based on combined patch and they passed:
          TestMultithreadedTableMapper, TestColumnSeeking, TestTableMapReduce

          Looks like Hadoop QA is not working. I am running test suite on Linux.

          Show
          Ted Yu added a comment - I ran the following tests based on combined patch and they passed: TestMultithreadedTableMapper, TestColumnSeeking, TestTableMapReduce Looks like Hadoop QA is not working. I am running test suite on Linux.
          Hide
          Takuya Ueshin added a comment -

          Oh, I misread what you wrote.
          Thank you for your support.

          Show
          Takuya Ueshin added a comment - Oh, I misread what you wrote. Thank you for your support.
          Hide
          Ted Yu added a comment -
          Results :
          
          Tests run: 541, Failures: 0, Errors: 0, Skipped: 0
          ...
          Results :
          
          Tests run: 615, Failures: 0, Errors: 0, Skipped: 2
          
          [INFO] ------------------------------------------------------------------------
          [INFO] BUILD SUCCESS
          [INFO] ------------------------------------------------------------------------
          [INFO] Total time: 35:45.595s
          

          I will integrate Monday if there is no objection.

          Show
          Ted Yu added a comment - Results : Tests run: 541, Failures: 0, Errors: 0, Skipped: 0 ... Results : Tests run: 615, Failures: 0, Errors: 0, Skipped: 2 [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 35:45.595s I will integrate Monday if there is no objection.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12520786/5663%2B5636.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 12 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1361//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1361//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1361//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12520786/5663%2B5636.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 12 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1361//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1361//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1361//console This message is automatically generated.
          Hide
          Ted Yu added a comment -

          Integrated to trunk and 0.94.

          Thanks for the patch, Takuya.

          Show
          Ted Yu added a comment - Integrated to trunk and 0.94. Thanks for the patch, Takuya.
          Hide
          Hudson added a comment -

          Integrated in HBase-0.94 #75 (See https://builds.apache.org/job/HBase-0.94/75/)
          HBASE-5663 HBASE-5636 MultithreadedTableMapper doesn't work (Takuya Ueshin) (Revision 1308354)

          Result = FAILURE
          tedyu :
          Files :

          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/mapreduce/MultithreadedTableMapper.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapred/TestTableMapReduce.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestMulitthreadedTableMapper.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestMultithreadedTableMapper.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduce.java
          Show
          Hudson added a comment - Integrated in HBase-0.94 #75 (See https://builds.apache.org/job/HBase-0.94/75/ ) HBASE-5663 HBASE-5636 MultithreadedTableMapper doesn't work (Takuya Ueshin) (Revision 1308354) Result = FAILURE tedyu : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/mapreduce/MultithreadedTableMapper.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapred/TestTableMapReduce.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestMulitthreadedTableMapper.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestMultithreadedTableMapper.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduce.java
          Hide
          Lars Hofhansl added a comment -

          Can this be closed?

          Show
          Lars Hofhansl added a comment - Can this be closed?
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK #2703 (See https://builds.apache.org/job/HBase-TRUNK/2703/)
          HBASE-5663 HBASE-5636 MultithreadedTableMapper doesn't work (Takuya Ueshin) (Revision 1308353)

          Result = SUCCESS
          tedyu :
          Files :

          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/MultithreadedTableMapper.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapred/TestTableMapReduce.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapreduce/TestMulitthreadedTableMapper.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapreduce/TestMultithreadedTableMapper.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduce.java
          Show
          Hudson added a comment - Integrated in HBase-TRUNK #2703 (See https://builds.apache.org/job/HBase-TRUNK/2703/ ) HBASE-5663 HBASE-5636 MultithreadedTableMapper doesn't work (Takuya Ueshin) (Revision 1308353) Result = SUCCESS tedyu : Files : /hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/MultithreadedTableMapper.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapred/TestTableMapReduce.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapreduce/TestMulitthreadedTableMapper.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapreduce/TestMultithreadedTableMapper.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduce.java
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK-security #156 (See https://builds.apache.org/job/HBase-TRUNK-security/156/)
          HBASE-5663 HBASE-5636 MultithreadedTableMapper doesn't work (Takuya Ueshin) (Revision 1308353)

          Result = FAILURE
          tedyu :
          Files :

          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/MultithreadedTableMapper.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapred/TestTableMapReduce.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapreduce/TestMulitthreadedTableMapper.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapreduce/TestMultithreadedTableMapper.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduce.java
          Show
          Hudson added a comment - Integrated in HBase-TRUNK-security #156 (See https://builds.apache.org/job/HBase-TRUNK-security/156/ ) HBASE-5663 HBASE-5636 MultithreadedTableMapper doesn't work (Takuya Ueshin) (Revision 1308353) Result = FAILURE tedyu : Files : /hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/MultithreadedTableMapper.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapred/TestTableMapReduce.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapreduce/TestMulitthreadedTableMapper.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapreduce/TestMultithreadedTableMapper.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduce.java
          Hide
          Hudson added a comment -

          Integrated in HBase-0.94-security #7 (See https://builds.apache.org/job/HBase-0.94-security/7/)
          HBASE-5663 HBASE-5636 MultithreadedTableMapper doesn't work (Takuya Ueshin) (Revision 1308354)

          Result = SUCCESS
          tedyu :
          Files :

          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/mapreduce/MultithreadedTableMapper.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapred/TestTableMapReduce.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestMulitthreadedTableMapper.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestMultithreadedTableMapper.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduce.java
          Show
          Hudson added a comment - Integrated in HBase-0.94-security #7 (See https://builds.apache.org/job/HBase-0.94-security/7/ ) HBASE-5663 HBASE-5636 MultithreadedTableMapper doesn't work (Takuya Ueshin) (Revision 1308354) Result = SUCCESS tedyu : Files : /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/mapreduce/MultithreadedTableMapper.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapred/TestTableMapReduce.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestMulitthreadedTableMapper.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestMultithreadedTableMapper.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduce.java

            People

            • Assignee:
              Takuya Ueshin
              Reporter:
              Takuya Ueshin
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development