Details

    • Type: Sub-task
    • Status: Patch Available
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: build, Usability
    • Labels:
    1. archetype_mr_prototype.zip
      37 kB
      Daniel Vimont
    2. HBASE-14879-v1.patch
      43 kB
      Daniel Vimont
    3. HBASE-14879-v2.patch
      43 kB
      Daniel Vimont

      Activity

      Hide
      daniel_vimont Daniel Vimont added a comment -

      Attaching a zip file, archetype_mr_prototype.zip, containing the components of a Maven archetype suitable for testing in a sandbox environment. It is provided separately here so that folks can try it out and comment on it before we try to incorporate it into the HBase project and push the archetype into the Maven central repository.

      The MapReduce job code that is generated from this archetype is called CopyTableExample.java, and it is derived from CopyTable in the HBase project. Also included is a corresponding test class TestCopyTableExample.java.

      This archetype generates a prefab project for the user which (a) automatically establishes for them all required Maven dependencies, (b) provides a complete example of HBase-oriented MapReduce job code, and (c) provides a complete example of test code showing how to use the HBaseTestUtility to easily and effectively test a MapReduce job.

      It's recommended to do the following in a separate sandbox environment, not in an active development environment. (I use VirtualBox to set up a fresh Ubuntu environment whenever I need a new sandbox.)

      INSTRUCTIONS
      (1) Download and unzip archetype_mr_prototype.zip.
      (2) Go into the archetype_mr_prototype directory, and execute: mvn install
      This will install the archetype (which is called hbase-mapreduce-job-archetype, version 1.1.2-alpha) into your local Maven repository.
      (3) To create a new project based upon the archetype, go to the directory in which you want the project folder and its contents to be created, and execute the following:
      mvn archetype:generate -DarchetypeCatalog=local
      This will interactively ask you to (a) choose the number of the archetype you want to use from a numbered list that is displayed (in this case, choose the number corresponding to “hbase-mapreduce-job-archetype”), (b) specify groupId (e.g., “org.example”), (c) specify artifactId (e.g., “test-hbase-mapreduce-project”), (d) specify version (default is “1.0-SNAPSHOT”), and (e) specify package (e.g., “org.example.testpackage”).

      Your generated project components will be found in a new directory named with the artifactId value that you specified above. Going to the new project directory and running mvn install from the command line should result in downloading of all dependencies, compiling of the provided Java classes, and running (hopefully to success) of all the tests in the test class. Note that the tests could take 3 to 5 minutes to complete.

      Show
      daniel_vimont Daniel Vimont added a comment - Attaching a zip file, archetype_mr_prototype.zip , containing the components of a Maven archetype suitable for testing in a sandbox environment. It is provided separately here so that folks can try it out and comment on it before we try to incorporate it into the HBase project and push the archetype into the Maven central repository. The MapReduce job code that is generated from this archetype is called CopyTableExample.java , and it is derived from CopyTable in the HBase project. Also included is a corresponding test class TestCopyTableExample.java . This archetype generates a prefab project for the user which (a) automatically establishes for them all required Maven dependencies, (b) provides a complete example of HBase-oriented MapReduce job code, and (c) provides a complete example of test code showing how to use the HBaseTestUtility to easily and effectively test a MapReduce job. It's recommended to do the following in a separate sandbox environment, not in an active development environment. (I use VirtualBox to set up a fresh Ubuntu environment whenever I need a new sandbox.) INSTRUCTIONS (1) Download and unzip archetype_mr_prototype.zip . (2) Go into the archetype_mr_prototype directory, and execute: mvn install This will install the archetype (which is called hbase-mapreduce-job-archetype , version 1.1.2-alpha ) into your local Maven repository. (3) To create a new project based upon the archetype, go to the directory in which you want the project folder and its contents to be created, and execute the following: mvn archetype:generate -DarchetypeCatalog=local This will interactively ask you to (a) choose the number of the archetype you want to use from a numbered list that is displayed (in this case, choose the number corresponding to “hbase-mapreduce-job-archetype”), (b) specify groupId (e.g., “org.example”), (c) specify artifactId (e.g., “test-hbase-mapreduce-project”), (d) specify version (default is “1.0-SNAPSHOT”), and (e) specify package (e.g., “org.example.testpackage”). Your generated project components will be found in a new directory named with the artifactId value that you specified above. Going to the new project directory and running mvn install from the command line should result in downloading of all dependencies, compiling of the provided Java classes, and running (hopefully to success) of all the tests in the test class. Note that the tests could take 3 to 5 minutes to complete.
      Hide
      daniel_vimont Daniel Vimont added a comment -

      Submitting patch: HBASE-14879-v1.patch

      This adds a new archetype for an HBase mapreduce job.

      This represents the final archetype of the three that were originally listed in HBASE-14876.

      Show
      daniel_vimont Daniel Vimont added a comment - Submitting patch: HBASE-14879 -v1.patch This adds a new archetype for an HBase mapreduce job. This represents the final archetype of the three that were originally listed in HBASE-14876 .
      Hide
      hadoopqa Hadoop QA added a comment -
      -1 overall



      Vote Subsystem Runtime Comment
      +1 hbaseanti 0m 0s Patch does not have any anti-patterns.
      +1 @author 0m 0s The patch does not contain any @author tags.
      +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
      0 mvndep 0m 33s Maven dependency ordering for branch
      +1 mvninstall 5m 24s master passed
      +1 compile 1m 45s master passed with JDK v1.8.0
      +1 compile 1m 18s master passed with JDK v1.7.0_79
      +1 checkstyle 0m 34s master passed
      +1 mvneclipse 0m 42s master passed
      0 findbugs 0m 0s Skipped branch modules with no Java source: hbase-archetypes hbase-archetypes/hbase-archetype-builder
      +1 findbugs 0m 0s master passed
      +1 javadoc 0m 58s master passed with JDK v1.8.0
      +1 javadoc 0m 28s master passed with JDK v1.7.0_79
      0 mvndep 0m 13s Maven dependency ordering for patch
      +1 mvninstall 2m 24s the patch passed
      +1 compile 2m 17s the patch passed with JDK v1.8.0
      +1 javac 2m 17s the patch passed
      +1 compile 1m 38s the patch passed with JDK v1.7.0_79
      +1 javac 1m 38s the patch passed
      -1 checkstyle 0m 17s hbase-archetypes: patch generated 6 new + 0 unchanged - 0 fixed = 6 total (was 0)
      -1 checkstyle 0m 17s hbase-archetypes/hbase-mapreduce-job: patch generated 6 new + 0 unchanged - 0 fixed = 6 total (was 0)
      -1 checkstyle 0m 17s hbase-archetypes/hbase-archetype-builder: patch generated 6 new + 0 unchanged - 0 fixed = 6 total (was 0)
      +1 mvneclipse 1m 0s the patch passed
      +1 shellcheck 0m 5s There were no new shellcheck issues.
      +1 shelldocs 0m 2s There were no new shelldocs issues.
      +1 whitespace 0m 0s Patch has no whitespace issues.
      +1 xml 0m 2s The patch has no ill-formed XML file.
      +1 hadoopcheck 35m 35s Patch does not cause any errors with Hadoop 2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1.
      0 findbugs 0m 0s Skipped patch modules with no Java source: hbase-archetypes hbase-archetypes/hbase-archetype-builder
      +1 findbugs 0m 49s the patch passed
      +1 javadoc 1m 19s the patch passed with JDK v1.8.0
      +1 javadoc 0m 42s the patch passed with JDK v1.7.0_79
      +1 unit 5m 57s hbase-archetypes in the patch passed.
      +1 unit 3m 35s hbase-mapreduce-job in the patch passed.
      +1 unit 0m 38s hbase-archetype-builder in the patch passed.
      +1 asflicense 0m 29s Patch does not generate ASF License warnings.
      69m 46s



      Subsystem Report/Notes
      JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12799898/HBASE-14879-v1.patch
      JIRA Issue HBASE-14879
      Optional Tests asflicense shellcheck shelldocs javac javadoc unit xml compile findbugs hadoopcheck hbaseanti checkstyle
      uname Linux asf910.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
      Build tool maven
      Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
      git revision master / b6617b4
      Default Java 1.7.0_79
      Multi-JDK versions /home/jenkins/tools/java/jdk1.8.0:1.8.0 /usr/local/jenkins/java/jdk1.7.0_79:1.7.0_79
      shellcheck v0.3.3 (This is an old version that has serious bugs. Consider upgrading.)
      checkstyle https://builds.apache.org/job/PreCommit-HBASE-Build/1529/artifact/patchprocess/diff-checkstyle-hbase-archetypes.txt
      checkstyle https://builds.apache.org/job/PreCommit-HBASE-Build/1529/artifact/patchprocess/diff-checkstyle-hbase-archetypes_hbase-mapreduce-job.txt
      checkstyle https://builds.apache.org/job/PreCommit-HBASE-Build/1529/artifact/patchprocess/diff-checkstyle-hbase-archetypes_hbase-archetype-builder.txt
      Test Results https://builds.apache.org/job/PreCommit-HBASE-Build/1529/testReport/
      modules C: hbase-archetypes hbase-archetypes/hbase-mapreduce-job hbase-archetypes/hbase-archetype-builder U: hbase-archetypes
      Console output https://builds.apache.org/job/PreCommit-HBASE-Build/1529/console
      Powered by Apache Yetus 0.2.1 http://yetus.apache.org

      This message was automatically generated.

      Show
      hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment +1 hbaseanti 0m 0s Patch does not have any anti-patterns. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 33s Maven dependency ordering for branch +1 mvninstall 5m 24s master passed +1 compile 1m 45s master passed with JDK v1.8.0 +1 compile 1m 18s master passed with JDK v1.7.0_79 +1 checkstyle 0m 34s master passed +1 mvneclipse 0m 42s master passed 0 findbugs 0m 0s Skipped branch modules with no Java source: hbase-archetypes hbase-archetypes/hbase-archetype-builder +1 findbugs 0m 0s master passed +1 javadoc 0m 58s master passed with JDK v1.8.0 +1 javadoc 0m 28s master passed with JDK v1.7.0_79 0 mvndep 0m 13s Maven dependency ordering for patch +1 mvninstall 2m 24s the patch passed +1 compile 2m 17s the patch passed with JDK v1.8.0 +1 javac 2m 17s the patch passed +1 compile 1m 38s the patch passed with JDK v1.7.0_79 +1 javac 1m 38s the patch passed -1 checkstyle 0m 17s hbase-archetypes: patch generated 6 new + 0 unchanged - 0 fixed = 6 total (was 0) -1 checkstyle 0m 17s hbase-archetypes/hbase-mapreduce-job: patch generated 6 new + 0 unchanged - 0 fixed = 6 total (was 0) -1 checkstyle 0m 17s hbase-archetypes/hbase-archetype-builder: patch generated 6 new + 0 unchanged - 0 fixed = 6 total (was 0) +1 mvneclipse 1m 0s the patch passed +1 shellcheck 0m 5s There were no new shellcheck issues. +1 shelldocs 0m 2s There were no new shelldocs issues. +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. +1 hadoopcheck 35m 35s Patch does not cause any errors with Hadoop 2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. 0 findbugs 0m 0s Skipped patch modules with no Java source: hbase-archetypes hbase-archetypes/hbase-archetype-builder +1 findbugs 0m 49s the patch passed +1 javadoc 1m 19s the patch passed with JDK v1.8.0 +1 javadoc 0m 42s the patch passed with JDK v1.7.0_79 +1 unit 5m 57s hbase-archetypes in the patch passed. +1 unit 3m 35s hbase-mapreduce-job in the patch passed. +1 unit 0m 38s hbase-archetype-builder in the patch passed. +1 asflicense 0m 29s Patch does not generate ASF License warnings. 69m 46s Subsystem Report/Notes JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12799898/HBASE-14879-v1.patch JIRA Issue HBASE-14879 Optional Tests asflicense shellcheck shelldocs javac javadoc unit xml compile findbugs hadoopcheck hbaseanti checkstyle uname Linux asf910.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh git revision master / b6617b4 Default Java 1.7.0_79 Multi-JDK versions /home/jenkins/tools/java/jdk1.8.0:1.8.0 /usr/local/jenkins/java/jdk1.7.0_79:1.7.0_79 shellcheck v0.3.3 (This is an old version that has serious bugs. Consider upgrading.) checkstyle https://builds.apache.org/job/PreCommit-HBASE-Build/1529/artifact/patchprocess/diff-checkstyle-hbase-archetypes.txt checkstyle https://builds.apache.org/job/PreCommit-HBASE-Build/1529/artifact/patchprocess/diff-checkstyle-hbase-archetypes_hbase-mapreduce-job.txt checkstyle https://builds.apache.org/job/PreCommit-HBASE-Build/1529/artifact/patchprocess/diff-checkstyle-hbase-archetypes_hbase-archetype-builder.txt Test Results https://builds.apache.org/job/PreCommit-HBASE-Build/1529/testReport/ modules C: hbase-archetypes hbase-archetypes/hbase-mapreduce-job hbase-archetypes/hbase-archetype-builder U: hbase-archetypes Console output https://builds.apache.org/job/PreCommit-HBASE-Build/1529/console Powered by Apache Yetus 0.2.1 http://yetus.apache.org This message was automatically generated.
      Hide
      daniel_vimont Daniel Vimont added a comment -

      Ugh! Neglected to run checkstyle before I submitted this!

      Will make corrections and submit a new version of the patch...

      Show
      daniel_vimont Daniel Vimont added a comment - Ugh! Neglected to run checkstyle before I submitted this! Will make corrections and submit a new version of the patch...
      Hide
      daniel_vimont Daniel Vimont added a comment -

      v2 patch being submitted, with checkstyle issues corrected.

      Show
      daniel_vimont Daniel Vimont added a comment - v2 patch being submitted, with checkstyle issues corrected.
      Hide
      hadoopqa Hadoop QA added a comment -
      +1 overall



      Vote Subsystem Runtime Comment
      +1 hbaseanti 0m 0s Patch does not have any anti-patterns.
      +1 @author 0m 0s The patch does not contain any @author tags.
      +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
      0 mvndep 0m 27s Maven dependency ordering for branch
      +1 mvninstall 3m 43s master passed
      +1 compile 0m 59s master passed with JDK v1.8.0
      +1 compile 0m 49s master passed with JDK v1.7.0_79
      +1 checkstyle 0m 23s master passed
      +1 mvneclipse 0m 29s master passed
      0 findbugs 0m 0s Skipped branch modules with no Java source: hbase-archetypes hbase-archetypes/hbase-archetype-builder
      +1 findbugs 0m 0s master passed
      +1 javadoc 0m 36s master passed with JDK v1.8.0
      +1 javadoc 0m 19s master passed with JDK v1.7.0_79
      0 mvndep 0m 9s Maven dependency ordering for patch
      +1 mvninstall 1m 39s the patch passed
      +1 compile 1m 24s the patch passed with JDK v1.8.0
      +1 javac 1m 24s the patch passed
      +1 compile 1m 21s the patch passed with JDK v1.7.0_79
      +1 javac 1m 21s the patch passed
      +1 checkstyle 0m 35s the patch passed
      +1 mvneclipse 0m 47s the patch passed
      +1 shellcheck 0m 4s There were no new shellcheck issues.
      +1 shelldocs 0m 1s There were no new shelldocs issues.
      +1 whitespace 0m 0s Patch has no whitespace issues.
      +1 xml 0m 2s The patch has no ill-formed XML file.
      +1 hadoopcheck 27m 53s Patch does not cause any errors with Hadoop 2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1.
      0 findbugs 0m 0s Skipped patch modules with no Java source: hbase-archetypes hbase-archetypes/hbase-archetype-builder
      +1 findbugs 0m 41s the patch passed
      +1 javadoc 0m 58s the patch passed with JDK v1.8.0
      +1 javadoc 0m 31s the patch passed with JDK v1.7.0_79
      +1 unit 5m 25s hbase-archetypes in the patch passed.
      +1 unit 3m 16s hbase-mapreduce-job in the patch passed.
      +1 unit 0m 26s hbase-archetype-builder in the patch passed.
      +1 asflicense 0m 23s Patch does not generate ASF License warnings.
      53m 46s



      Subsystem Report/Notes
      JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12799920/HBASE-14879-v2.patch
      JIRA Issue HBASE-14879
      Optional Tests asflicense shellcheck shelldocs javac javadoc unit xml compile findbugs hadoopcheck hbaseanti checkstyle
      uname Linux proserpina.apache.org 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
      Build tool maven
      Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
      git revision master / 03f3c39
      Default Java 1.7.0_79
      Multi-JDK versions /home/jenkins/tools/java/jdk1.8.0:1.8.0 /usr/local/jenkins/java/jdk1.7.0_79:1.7.0_79
      shellcheck v0.3.3 (This is an old version that has serious bugs. Consider upgrading.)
      Test Results https://builds.apache.org/job/PreCommit-HBASE-Build/1533/testReport/
      modules C: hbase-archetypes hbase-archetypes/hbase-mapreduce-job hbase-archetypes/hbase-archetype-builder U: hbase-archetypes
      Console output https://builds.apache.org/job/PreCommit-HBASE-Build/1533/console
      Powered by Apache Yetus 0.2.1 http://yetus.apache.org

      This message was automatically generated.

      Show
      hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment +1 hbaseanti 0m 0s Patch does not have any anti-patterns. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 27s Maven dependency ordering for branch +1 mvninstall 3m 43s master passed +1 compile 0m 59s master passed with JDK v1.8.0 +1 compile 0m 49s master passed with JDK v1.7.0_79 +1 checkstyle 0m 23s master passed +1 mvneclipse 0m 29s master passed 0 findbugs 0m 0s Skipped branch modules with no Java source: hbase-archetypes hbase-archetypes/hbase-archetype-builder +1 findbugs 0m 0s master passed +1 javadoc 0m 36s master passed with JDK v1.8.0 +1 javadoc 0m 19s master passed with JDK v1.7.0_79 0 mvndep 0m 9s Maven dependency ordering for patch +1 mvninstall 1m 39s the patch passed +1 compile 1m 24s the patch passed with JDK v1.8.0 +1 javac 1m 24s the patch passed +1 compile 1m 21s the patch passed with JDK v1.7.0_79 +1 javac 1m 21s the patch passed +1 checkstyle 0m 35s the patch passed +1 mvneclipse 0m 47s the patch passed +1 shellcheck 0m 4s There were no new shellcheck issues. +1 shelldocs 0m 1s There were no new shelldocs issues. +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. +1 hadoopcheck 27m 53s Patch does not cause any errors with Hadoop 2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. 0 findbugs 0m 0s Skipped patch modules with no Java source: hbase-archetypes hbase-archetypes/hbase-archetype-builder +1 findbugs 0m 41s the patch passed +1 javadoc 0m 58s the patch passed with JDK v1.8.0 +1 javadoc 0m 31s the patch passed with JDK v1.7.0_79 +1 unit 5m 25s hbase-archetypes in the patch passed. +1 unit 3m 16s hbase-mapreduce-job in the patch passed. +1 unit 0m 26s hbase-archetype-builder in the patch passed. +1 asflicense 0m 23s Patch does not generate ASF License warnings. 53m 46s Subsystem Report/Notes JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12799920/HBASE-14879-v2.patch JIRA Issue HBASE-14879 Optional Tests asflicense shellcheck shelldocs javac javadoc unit xml compile findbugs hadoopcheck hbaseanti checkstyle uname Linux proserpina.apache.org 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh git revision master / 03f3c39 Default Java 1.7.0_79 Multi-JDK versions /home/jenkins/tools/java/jdk1.8.0:1.8.0 /usr/local/jenkins/java/jdk1.7.0_79:1.7.0_79 shellcheck v0.3.3 (This is an old version that has serious bugs. Consider upgrading.) Test Results https://builds.apache.org/job/PreCommit-HBASE-Build/1533/testReport/ modules C: hbase-archetypes hbase-archetypes/hbase-mapreduce-job hbase-archetypes/hbase-archetype-builder U: hbase-archetypes Console output https://builds.apache.org/job/PreCommit-HBASE-Build/1533/console Powered by Apache Yetus 0.2.1 http://yetus.apache.org This message was automatically generated.
      Hide
      jmhsieh Jonathan Hsieh added a comment - - edited

      Hey Daniel Vimont, I've given this patch a spin. I assumed the instructions for testing are the same as the for the zip except for moving to the hbase-archetype module dir instead of the zip's dir

      I'm actually running into some problems building an instance of the mr project after running 'mvn archetype:generate'. I'm hitting stuff like this:

      2016-05-10 11:56:53,258 ERROR [ProcedureExecutor-0] master.TableStateManager: Unable to get table hbase:namespace state
      org.apache.hadoop.hbase.TableNotFoundException: hbase:namespace
              at org.apache.hadoop.hbase.master.TableStateManager.getTableState(TableStateManager.java:174)
              at org.apache.hadoop.hbase.master.TableStateManager.isTableState(TableStateManager.java:131)
              at org.apache.hadoop.hbase.master.AssignmentManager.isDisabledorDisablingRegionInRIT(AssignmentManager.java:1221)
       ...
      2016-05-10 11:57:14,827 ERROR [ProcedureExecutor-3] master.TableStateManager: Unable to get table testCopyTable1 state
      org.apache.hadoop.hbase.TableNotFoundException: testCopyTable1
              at org.apache.hadoop.hbase.master.TableStateManager.getTableState(TableStateManager.java:174)
              at org.apache.hadoop.hbase.master.TableStateManager.isTableState(TableStateManager.java:131)
              at org.apache.hadoop.hbase.master.AssignmentManager.isDisabledorDisablingRegionInRIT(AssignmentManager.java:1221)
              at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:739)
      ...
      2016-05-10 11:57:17,203 ERROR [ProcedureExecutor-0] master.TableStateManager: Unable to get table testCopyTable2 state
      org.apache.hadoop.hbase.TableNotFoundException: testCopyTable2
              at org.apache.hadoop.hbase.master.TableStateManager.getTableState(TableStateManager.java:174)
              at org.apache.hadoop.hbase.master.TableStateManager.isTableState(TableStateManager.java:131)
              at org.apache.hadoop.hbase.master.AssignmentManager.isDisabledorDisablingRegionInRIT(AssignmentManager.java:1221)
              at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:739)
              at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1576)
              at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1555)
      ...
      2016-05-10 11:01:47,347 ERROR [Node Status Updater] nodemanager.NodeStatusUpdaterImpl: Caught exception in status-updater
      java.lang.OutOfMemoryError: PermGen space
      

      If I ran the test with this command

      MAVEN_OPTS="-XX:MaxPermSize=2g -Xmx5g" mvn install 
      

      and still ran into the same table not found and permgen problems.

      I'm not sure if this is a recent problem introduce – does this (still) work for you?

      Show
      jmhsieh Jonathan Hsieh added a comment - - edited Hey Daniel Vimont , I've given this patch a spin. I assumed the instructions for testing are the same as the for the zip except for moving to the hbase-archetype module dir instead of the zip's dir I'm actually running into some problems building an instance of the mr project after running 'mvn archetype:generate'. I'm hitting stuff like this: 2016-05-10 11:56:53,258 ERROR [ProcedureExecutor-0] master.TableStateManager: Unable to get table hbase:namespace state org.apache.hadoop.hbase.TableNotFoundException: hbase:namespace at org.apache.hadoop.hbase.master.TableStateManager.getTableState(TableStateManager.java:174) at org.apache.hadoop.hbase.master.TableStateManager.isTableState(TableStateManager.java:131) at org.apache.hadoop.hbase.master.AssignmentManager.isDisabledorDisablingRegionInRIT(AssignmentManager.java:1221) ... 2016-05-10 11:57:14,827 ERROR [ProcedureExecutor-3] master.TableStateManager: Unable to get table testCopyTable1 state org.apache.hadoop.hbase.TableNotFoundException: testCopyTable1 at org.apache.hadoop.hbase.master.TableStateManager.getTableState(TableStateManager.java:174) at org.apache.hadoop.hbase.master.TableStateManager.isTableState(TableStateManager.java:131) at org.apache.hadoop.hbase.master.AssignmentManager.isDisabledorDisablingRegionInRIT(AssignmentManager.java:1221) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:739) ... 2016-05-10 11:57:17,203 ERROR [ProcedureExecutor-0] master.TableStateManager: Unable to get table testCopyTable2 state org.apache.hadoop.hbase.TableNotFoundException: testCopyTable2 at org.apache.hadoop.hbase.master.TableStateManager.getTableState(TableStateManager.java:174) at org.apache.hadoop.hbase.master.TableStateManager.isTableState(TableStateManager.java:131) at org.apache.hadoop.hbase.master.AssignmentManager.isDisabledorDisablingRegionInRIT(AssignmentManager.java:1221) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:739) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1576) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1555) ... 2016-05-10 11:01:47,347 ERROR [Node Status Updater] nodemanager.NodeStatusUpdaterImpl: Caught exception in status-updater java.lang.OutOfMemoryError: PermGen space If I ran the test with this command MAVEN_OPTS= "-XX:MaxPermSize=2g -Xmx5g" mvn install and still ran into the same table not found and permgen problems. I'm not sure if this is a recent problem introduce – does this (still) work for you?
      Hide
      daniel_vimont Daniel Vimont added a comment -

      I will look into this and get back to you soon. Thanks!

      Show
      daniel_vimont Daniel Vimont added a comment - I will look into this and get back to you soon. Thanks!
      Hide
      daniel_vimont Daniel Vimont added a comment -

      I did a fresh clone of the hbase.git repository, applied the patch, ran a "clean install", generated a map-reduce project using the new archetype, then successfully ran a "clean install" on the newly-generated mapreduce project. All its tests ran successfully.

      One important thing to note: I did NOT have a local instance of HBase running when I did all this.

      While I was NOT able to recreate your problem, I DID encounter a problem when I started up a local instance of HBase and reran the "clean install" on the newly-generated mapreduce project. In that case the test failed due to port contention:

           2016-05-11 12:02:07,183 ERROR [main] regionserver.HRegionServer: Failed binding http info server to port: 16010
           2016-05-11 12:02:07,184 ERROR [main] hbase.MiniHBaseCluster: Error starting cluster
           java.lang.RuntimeException: Failed construction of Master: class org.apache.hadoop.hbase.master.HMasterAddress already in use
      

      I did some googling about to try to get to the root of this, and found an old 1.0.x JIRA entry that seemed to relate: HBASE-13479

      Following the advice implied in this JIRA entry, I added an hbase-site.xml file to the the src/test/resources directory of my newly-generated mapreduce project as follows (using completely arbritrarily-chosen port numbers, to differentiate from the default 16010, 16020, and 16030 ports):

          <configuration>
              <property>
                  <name>hbase.regionserver.port</name>
                  <value>16050</value>
              </property>
              <property>
                  <name>hbase.master.info.port</name>
                  <value>16060</value>
              </property>
              <property>
                  <name>hbase.regionserver.info.port</name>
                  <value>16070</value>
              </property>
          </configuration>
      

      After I put this hbase-site.xml file in place, a "clean install" in the new project completes successfully, even when a local instance of HBase is running.

      Now, since I as yet have no idea as to the nature of the problem you are experiencing, I don't know whether the addition of an hbase-site.xml file (with the above properties) to your new project's src/test/resources directory will resolve your problem or not.

      Show
      daniel_vimont Daniel Vimont added a comment - I did a fresh clone of the hbase.git repository, applied the patch, ran a "clean install", generated a map-reduce project using the new archetype, then successfully ran a "clean install" on the newly-generated mapreduce project. All its tests ran successfully. One important thing to note: I did NOT have a local instance of HBase running when I did all this. While I was NOT able to recreate your problem, I DID encounter a problem when I started up a local instance of HBase and reran the "clean install" on the newly-generated mapreduce project. In that case the test failed due to port contention: 2016-05-11 12:02:07,183 ERROR [main] regionserver.HRegionServer: Failed binding http info server to port: 16010 2016-05-11 12:02:07,184 ERROR [main] hbase.MiniHBaseCluster: Error starting cluster java.lang.RuntimeException: Failed construction of Master: class org.apache.hadoop.hbase.master.HMasterAddress already in use I did some googling about to try to get to the root of this, and found an old 1.0.x JIRA entry that seemed to relate: HBASE-13479 Following the advice implied in this JIRA entry, I added an hbase-site.xml file to the the src/test/resources directory of my newly-generated mapreduce project as follows (using completely arbritrarily-chosen port numbers, to differentiate from the default 16010, 16020, and 16030 ports): <configuration> <property> <name> hbase.regionserver.port </name> <value> 16050 </value> </property> <property> <name> hbase.master.info.port </name> <value> 16060 </value> </property> <property> <name> hbase.regionserver.info.port </name> <value> 16070 </value> </property> </configuration> After I put this hbase-site.xml file in place, a "clean install" in the new project completes successfully, even when a local instance of HBase is running. Now, since I as yet have no idea as to the nature of the problem you are experiencing, I don't know whether the addition of an hbase-site.xml file (with the above properties) to your new project's src/test/resources directory will resolve your problem or not.
      Hide
      daniel_vimont Daniel Vimont added a comment -

      BTW, the instructions for both testing current archetypes and adding new archetypes are in hbase/hbase-archetypes/README.md, which is easier to read when it is rendered on a GitHub page: https://github.com/dvimont/test_hbasearchetypes_readme

      Show
      daniel_vimont Daniel Vimont added a comment - BTW, the instructions for both testing current archetypes and adding new archetypes are in hbase/hbase-archetypes/README.md, which is easier to read when it is rendered on a GitHub page: https://github.com/dvimont/test_hbasearchetypes_readme
      Hide
      daniel_vimont Daniel Vimont added a comment -

      UPDATE on this: I am now seeing the errors that you pointed out above, and am investigating further...

      Show
      daniel_vimont Daniel Vimont added a comment - UPDATE on this: I am now seeing the errors that you pointed out above, and am investigating further...
      Hide
      daniel_vimont Daniel Vimont added a comment -

      UPDATE on this: I am now seeing the errors that you pointed out above, and am investigating further...

      Show
      daniel_vimont Daniel Vimont added a comment - UPDATE on this: I am now seeing the errors that you pointed out above, and am investigating further...
      Hide
      daniel_vimont Daniel Vimont added a comment -

      I am looking further at the output containing the various ERROR messages, and also looking at the HBase source code (AssignmentManager, TableStateManager, MetaTableAccessor) that is behind the ERROR message generation.

      In the course of the mapreduce-job test processing, TableStateManager#isTableState is called from the AssignmentManager class to find out whether a table is in a state of DISABLED or DISABLING. Apparently, in the case of these tables, their status cannot be determined by MetaTableAccessor, so a TableNotFoundException is thrown which is caught and results in the ERROR message being outputted. However, on the subsequent lines of output, we see the tables being found and managed without a problem.

      For example, here is one of the ERROR messages:

      2016-05-11 14:51:11,849 ERROR [B.defaultRpcServer.handler=17,queue=2,port=60857] master.TableStateManager: Unable to get table testCopyTable1 state
      org.apache.hadoop.hbase.TableNotFoundException: testCopyTable1
      

      The very next line is an INFO message, showing that the table has been ENABLED:

      2016-05-11 14:51:11,855 INFO  [ProcedureExecutor-3] hbase.MetaTableAccessor: Updated table testCopyTable1 state to ENABLED in META
      

      Owing to the fact that all test procedures in the mapreduce-job project are succeeding, it seems likely to me that the ERROR messages do not signal any problem in the mapreduce-job code or the test code that is executing it; they may signal a problem in AssignmentManager/TableStateManager/MetaTableAccessor processing (in that the states of apparently-existing tables seem to not be accessible or are failing to be looked up effectively).

      Show
      daniel_vimont Daniel Vimont added a comment - I am looking further at the output containing the various ERROR messages, and also looking at the HBase source code (AssignmentManager, TableStateManager, MetaTableAccessor) that is behind the ERROR message generation. In the course of the mapreduce-job test processing, TableStateManager#isTableState is called from the AssignmentManager class to find out whether a table is in a state of DISABLED or DISABLING. Apparently, in the case of these tables, their status cannot be determined by MetaTableAccessor, so a TableNotFoundException is thrown which is caught and results in the ERROR message being outputted. However, on the subsequent lines of output, we see the tables being found and managed without a problem. For example, here is one of the ERROR messages: 2016-05-11 14:51:11,849 ERROR [B.defaultRpcServer.handler=17,queue=2,port=60857] master.TableStateManager: Unable to get table testCopyTable1 state org.apache.hadoop.hbase.TableNotFoundException: testCopyTable1 The very next line is an INFO message, showing that the table has been ENABLED: 2016-05-11 14:51:11,855 INFO [ProcedureExecutor-3] hbase.MetaTableAccessor: Updated table testCopyTable1 state to ENABLED in META Owing to the fact that all test procedures in the mapreduce-job project are succeeding, it seems likely to me that the ERROR messages do not signal any problem in the mapreduce-job code or the test code that is executing it; they may signal a problem in AssignmentManager/TableStateManager/MetaTableAccessor processing (in that the states of apparently-existing tables seem to not be accessible or are failing to be looked up effectively).
      Hide
      jmhsieh Jonathan Hsieh added a comment -

      I encountered the port conflict issue, I don't think that is a huge deal. If you ^C the 'mvn install' of the archetype instance, the process doesn't die right away and causes the port conflict on a subsequent run. I just 'jps'ed and then killed the offending surefire instance.

      I tried adding 'clean' to the hbase mvn build and the hbase+mr job mvn build and still got to the same place.

      The code and the logic in the test util to create tables looks fine to me, so I think I agree that there might be an issue where you pointed at. I also think it may be something environmental (the heap settings or something else related to jvm?) – This could explain why the hbaseproject mr tests are working under its pom, and the archetype instance ones are failing or flakey (different env settings).

      I'd rather not commit this specific patch until I can get its artifact to work or if someone else can get it to work.

      Would you mind digging a little further down the environmental path?

      Show
      jmhsieh Jonathan Hsieh added a comment - I encountered the port conflict issue, I don't think that is a huge deal. If you ^C the 'mvn install' of the archetype instance, the process doesn't die right away and causes the port conflict on a subsequent run. I just 'jps'ed and then killed the offending surefire instance. I tried adding 'clean' to the hbase mvn build and the hbase+mr job mvn build and still got to the same place. The code and the logic in the test util to create tables looks fine to me, so I think I agree that there might be an issue where you pointed at. I also think it may be something environmental (the heap settings or something else related to jvm?) – This could explain why the hbaseproject mr tests are working under its pom, and the archetype instance ones are failing or flakey (different env settings). I'd rather not commit this specific patch until I can get its artifact to work or if someone else can get it to work. Would you mind digging a little further down the environmental path?
      Hide
      daniel_vimont Daniel Vimont added a comment -

      Yes, I'll dig deeper into this...

      Show
      daniel_vimont Daniel Vimont added a comment - Yes, I'll dig deeper into this...
      Hide
      daniel_vimont Daniel Vimont added a comment -

      What we are experiencing seems to be the standard behavior of the HBaseTestingUtility.

      The problem can be replicated through execution of a single line of code:

      new HBaseTestingUtility().startMiniCluster();
      

      This results in the following output, regarding the 'hbase:namespace' metatable, which the miniCluster is apparently in the process of creating when this message is issued:

      2016-05-12 12:45:33,413 ERROR [ProcedureExecutor-0] master.TableStateManager: Unable to get table hbase:namespace state
      org.apache.hadoop.hbase.TableNotFoundException: hbase:namespace
      

      To more fully replicate what WE are experiencing (a series of those same ERROR messages when a user table is created), the following three lines of code may be executed:

      final HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility();
      TEST_UTIL.startMiniCluster();
      TEST_UTIL.createTable(TableName.valueOf("table1"), Bytes.toBytes("cf1"));
      

      This results in the following output:

      2016-05-12 12:56:54,052 ERROR [ProcedureExecutor-0] master.TableStateManager: Unable to get table hbase:namespace state
      org.apache.hadoop.hbase.TableNotFoundException: hbase:namespace
      ...
      2016-05-12 12:56:56,496 ERROR [ProcedureExecutor-3] master.TableStateManager: Unable to get table table1 state
      org.apache.hadoop.hbase.TableNotFoundException: table1
      ...
      2016-05-12 12:56:56,550 ERROR [B.defaultRpcServer.handler=11,queue=2,port=38763] master.TableStateManager: Unable to get table table1 state
      org.apache.hadoop.hbase.TableNotFoundException: table1
      

      As you can see, whenever a user table is created (as opposed to a system metatable), a similar ERROR message is issued twice in the course of table creation.

      Show
      daniel_vimont Daniel Vimont added a comment - What we are experiencing seems to be the standard behavior of the HBaseTestingUtility. The problem can be replicated through execution of a single line of code: new HBaseTestingUtility().startMiniCluster(); This results in the following output, regarding the 'hbase:namespace' metatable, which the miniCluster is apparently in the process of creating when this message is issued: 2016-05-12 12:45:33,413 ERROR [ProcedureExecutor-0] master.TableStateManager: Unable to get table hbase:namespace state org.apache.hadoop.hbase.TableNotFoundException: hbase:namespace To more fully replicate what WE are experiencing (a series of those same ERROR messages when a user table is created), the following three lines of code may be executed: final HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility(); TEST_UTIL.startMiniCluster(); TEST_UTIL.createTable(TableName.valueOf( "table1" ), Bytes.toBytes( "cf1" )); This results in the following output: 2016-05-12 12:56:54,052 ERROR [ProcedureExecutor-0] master.TableStateManager: Unable to get table hbase:namespace state org.apache.hadoop.hbase.TableNotFoundException: hbase:namespace ... 2016-05-12 12:56:56,496 ERROR [ProcedureExecutor-3] master.TableStateManager: Unable to get table table1 state org.apache.hadoop.hbase.TableNotFoundException: table1 ... 2016-05-12 12:56:56,550 ERROR [B.defaultRpcServer.handler=11,queue=2,port=38763] master.TableStateManager: Unable to get table table1 state org.apache.hadoop.hbase.TableNotFoundException: table1 As you can see, whenever a user table is created (as opposed to a system metatable), a similar ERROR message is issued twice in the course of table creation.

        People

        • Assignee:
          daniel_vimont Daniel Vimont
          Reporter:
          ndimiduk Nick Dimiduk
        • Votes:
          0 Vote for this issue
          Watchers:
          5 Start watching this issue

          Dates

          • Created:
            Updated:

            Development