Details

    • Type: New Feature
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 2.0.0
    • Fix Version/s: None
    • Component/s: build, Usability
    • Labels:
    • Release Note:
      Hide
      HBASE-14876 - new infrastructure for Maven archetypes

      Patches are being submitted via the sub-tasks of this task. Please see release notes of the sub-tasks for details.
      Show
      HBASE-14876 - new infrastructure for Maven archetypes Patches are being submitted via the sub-tasks of this task. Please see release notes of the sub-tasks for details.
    • Tags:
      archetype

      Description

      To help onboard new users, we should provide maven archetypes for hbase client applications. Off the top of my head, we should have templates for

      • hbase client application with all dependencies
      • hbase client application using client-shaded jar
      • mapreduce application with hbase as input and output (ie, copy table)
      1. HBASE-14876-v2.patch
        93 kB
        Daniel Vimont
      2. HBASE-14876.patch
        93 kB
        Daniel Vimont
      3. archetype_shaded_prototype01.zip
        23 kB
        Daniel Vimont
      4. archetype_prototype02.zip
        23 kB
        Daniel Vimont
      5. archetype_prototype.zip
        18 kB
        Daniel Vimont

        Activity

        Hide
        gliptak Gabor Liptak added a comment -

        Nick Dimiduk How do you see archetypes packaged (and published)? Thanks

        https://maven.apache.org/guides/mini/guide-creating-archetypes.html

        Show
        gliptak Gabor Liptak added a comment - Nick Dimiduk How do you see archetypes packaged (and published)? Thanks https://maven.apache.org/guides/mini/guide-creating-archetypes.html
        Hide
        ndimiduk Nick Dimiduk added a comment -

        Published via maven repository just like the rest of our jars. Unless I miss-understand how archetypes work...

        Show
        ndimiduk Nick Dimiduk added a comment - Published via maven repository just like the rest of our jars. Unless I miss-understand how archetypes work...
        Hide
        daniel_vimont Daniel Vimont added a comment -

        Besides wanting to get my hands dirty and start making some coding contributions to the HBase project, I've also been wanting to deepen my knowledge and understanding of Maven. While I'm not free to assign myself to this issue right at the moment, I could be available in a December/January timeframe to do some R&D and see what I could get going on this issue. With that said, if there are any Maven mavens around who want to deal with this more expeditiously, they should go for it now.

        Show
        daniel_vimont Daniel Vimont added a comment - Besides wanting to get my hands dirty and start making some coding contributions to the HBase project, I've also been wanting to deepen my knowledge and understanding of Maven. While I'm not free to assign myself to this issue right at the moment, I could be available in a December/January timeframe to do some R&D and see what I could get going on this issue. With that said, if there are any Maven mavens around who want to deal with this more expeditiously, they should go for it now.
        Hide
        daniel_vimont Daniel Vimont added a comment -

        Okay, I went ahead and assigned this to myself, since this overlaps perfectly with some other work that was on the top of my to-do list.

        I should have a prototype of a maven-archetype to share here shortly... (The Maven folks really have put together a slick architecture, making it all quite easy and quick to create new archetypes.)

        Show
        daniel_vimont Daniel Vimont added a comment - Okay, I went ahead and assigned this to myself, since this overlaps perfectly with some other work that was on the top of my to-do list. I should have a prototype of a maven-archetype to share here shortly... (The Maven folks really have put together a slick architecture, making it all quite easy and quick to create new archetypes.)
        Hide
        daniel_vimont Daniel Vimont added a comment -

        BTW, I see that there are Maven plugins available for installing and controlling webservers, so I will look into creating a similar plugin devoted to installing and controlling a simple, standalone instance of HBase. (This might be outside of the scope of this JIRA issue, justifying a separate JIRA entry.)

        Show
        daniel_vimont Daniel Vimont added a comment - BTW, I see that there are Maven plugins available for installing and controlling webservers, so I will look into creating a similar plugin devoted to installing and controlling a simple, standalone instance of HBase. (This might be outside of the scope of this JIRA issue, justifying a separate JIRA entry.)
        Hide
        ndimiduk Nick Dimiduk added a comment -

        Thanks Daniel Vimont. A maven plugin for minicluster is definitely out of scope here. There's already a ticket for this, have a look at HBASE-14118.

        Show
        ndimiduk Nick Dimiduk added a comment - Thanks Daniel Vimont . A maven plugin for minicluster is definitely out of scope here. There's already a ticket for this, have a look at HBASE-14118 .
        Hide
        daniel_vimont Daniel Vimont added a comment -

        Hmmm, can't quite follow the extended discussion over there on HBASE-14118 (looks like it may be lost in licensing purgatory?).

        Anyway, yes – I will focus solely on getting together a maven-archetype based architecture.

        Show
        daniel_vimont Daniel Vimont added a comment - Hmmm, can't quite follow the extended discussion over there on HBASE-14118 (looks like it may be lost in licensing purgatory?). Anyway, yes – I will focus solely on getting together a maven-archetype based architecture.
        Hide
        daniel_vimont Daniel Vimont added a comment -

        Attaching a prototype of a simple hbase-client archetype.

        Show
        daniel_vimont Daniel Vimont added a comment - Attaching a prototype of a simple hbase-client archetype.
        Hide
        daniel_vimont Daniel Vimont added a comment -

        I’ve attached a prototype of an archetype which you can download and install locally, so that you can create a test project from it and see if this is what you had in mind. (Ultimately, the real HBase archetypes will be housed in the Central Maven repository. The steps below are only used to install an archetype into your local Maven repository for testing purposes.)

        This archetype is based upon the standard Maven “quickstart” archetype (“maven-archetype-quickstart”). I started with a project generated from that, made a few changes to the project’s POM (chiefly, adding a single dependency upon “hbase-client” – version 1.1.2) and added sample client code (“HelloHBase.java”).

        INSTRUCTIONS
        (1) Download and unzip “archetype_prototype.zip”.
        (2) Go into the “archetype_prototype” directory, and execute: mvn install
        This will install the archetype (which is called “simple-hbase-client-archetype”) into your local Maven repository.
        (3) To create a new project based upon the archetype, go to the directory in which you want the project folder and its contents to be created (e.g., “NetBeansProjects”), and execute the following: mvn archetype:generate -DarchetypeCatalog=local
        This will interactively ask you to (a) choose from a numbered list which archetype you want to use (in this case, choose the number corresponding to “simple-hbase-client-archetype”), (b) specify groupId (e.g., “org.example”), (c) specify artifactId (e.g., “test-hbase-client-project”), (d) specify version (default is “1.0-SNAPSHOT”), (e) specify package (e.g., “org.example.testpackage”).

        Your new project artifacts will be found in a new directory with the same name as the artifactId value that you specified above. Drilling down in the src directory, you’ll find a single file, “HelloHBase.java”, which serves as kind of a quicky tutorial for complete newbies, or which can simply be deleted by those who are ready to write their own client code.

        A few points to consider…
        VERSIONING: Each archetype needs to be versioned. In the case of the prototype that I produced, I went with the following versioning scheme: since this archetype has only one direct dependency (hbase-client), I assigned to the archetype the same version as the version of hbase-client that is used (i.e. 1.1.2). With this versioning scheme, the presumption is that whenever a new version of hbase-client is deployed to the central Maven repository, a new version of this archetype will need to be deployed as well (with both items having the same version number).

        LIFE-CYCLE: After working with archetypes for a few days and experimenting with the various functions that are available, I could foresee the following steps being followed to come out with a new “version” of an archetype.
        (1) Use “mvn archetype:generate” to generate a new project from the current version of the archetype.
        (2) Make desired modifications (to the Java code, to the POM dependencies, whatever) to the project generated in step 1.
        (3) Generate a new version of the archetype by executing “mvn archetype:create-from-project“ against the revised project from step 2.

        These three steps just give a broad overview; details on these steps (and other required steps) will need to be fleshed out. In fact, I should be able to come up with pretty detailed documentation on all of this, since I've been taking rather detailed notes as I've been stumbling along...

        Show
        daniel_vimont Daniel Vimont added a comment - I’ve attached a prototype of an archetype which you can download and install locally, so that you can create a test project from it and see if this is what you had in mind. (Ultimately, the real HBase archetypes will be housed in the Central Maven repository. The steps below are only used to install an archetype into your local Maven repository for testing purposes.) This archetype is based upon the standard Maven “quickstart” archetype (“maven-archetype-quickstart”). I started with a project generated from that, made a few changes to the project’s POM (chiefly, adding a single dependency upon “hbase-client” – version 1.1.2) and added sample client code (“HelloHBase.java”). INSTRUCTIONS (1) Download and unzip “archetype_prototype.zip”. (2) Go into the “archetype_prototype” directory, and execute: mvn install This will install the archetype (which is called “simple-hbase-client-archetype”) into your local Maven repository. (3) To create a new project based upon the archetype, go to the directory in which you want the project folder and its contents to be created (e.g., “NetBeansProjects”), and execute the following: mvn archetype:generate -DarchetypeCatalog=local This will interactively ask you to (a) choose from a numbered list which archetype you want to use (in this case, choose the number corresponding to “simple-hbase-client-archetype”), (b) specify groupId (e.g., “org.example”), (c) specify artifactId (e.g., “test-hbase-client-project”), (d) specify version (default is “1.0-SNAPSHOT”), (e) specify package (e.g., “org.example.testpackage”). Your new project artifacts will be found in a new directory with the same name as the artifactId value that you specified above. Drilling down in the src directory, you’ll find a single file, “HelloHBase.java”, which serves as kind of a quicky tutorial for complete newbies, or which can simply be deleted by those who are ready to write their own client code. A few points to consider… VERSIONING : Each archetype needs to be versioned. In the case of the prototype that I produced, I went with the following versioning scheme: since this archetype has only one direct dependency (hbase-client), I assigned to the archetype the same version as the version of hbase-client that is used (i.e. 1.1.2). With this versioning scheme, the presumption is that whenever a new version of hbase-client is deployed to the central Maven repository, a new version of this archetype will need to be deployed as well (with both items having the same version number). LIFE-CYCLE : After working with archetypes for a few days and experimenting with the various functions that are available, I could foresee the following steps being followed to come out with a new “version” of an archetype. (1) Use “mvn archetype:generate” to generate a new project from the current version of the archetype. (2) Make desired modifications (to the Java code, to the POM dependencies, whatever) to the project generated in step 1. (3) Generate a new version of the archetype by executing “mvn archetype:create-from-project“ against the revised project from step 2. These three steps just give a broad overview; details on these steps (and other required steps) will need to be fleshed out. In fact, I should be able to come up with pretty detailed documentation on all of this, since I've been taking rather detailed notes as I've been stumbling along...
        Hide
        daniel_vimont Daniel Vimont added a comment -

        Providing corrected version of the prototype in a new zip file.

        Fixes following problem: The simple client application generated from the archetype was generating warnings that log4j was not configured properly.

        This was resolved by copying log4j.properties from the hbase/conf directory into the base* project's src/main/resources directory.

        *"Base" project refers to project from which the archetype was derived.

        Show
        daniel_vimont Daniel Vimont added a comment - Providing corrected version of the prototype in a new zip file. Fixes following problem: The simple client application generated from the archetype was generating warnings that log4j was not configured properly. This was resolved by copying log4j.properties from the hbase/conf directory into the base* project's src/main/resources directory. *"Base" project refers to project from which the archetype was derived.
        Hide
        daniel_vimont Daniel Vimont added a comment -

        A new prototype of an archetype, this one to fulfill the second requirement: to produce client application with dependency upon hbase-shaded-client.

        This one may also be installed in a sandbox environment to work with, by following the same instructions that were listed above for the first prototype.

        Show
        daniel_vimont Daniel Vimont added a comment - A new prototype of an archetype, this one to fulfill the second requirement: to produce client application with dependency upon hbase-shaded-client . This one may also be installed in a sandbox environment to work with, by following the same instructions that were listed above for the first prototype.
        Hide
        daniel_vimont Daniel Vimont added a comment -

        This comment refers to the zip file I just uploaded: archetype_shaded_prototype01.zip

        Show
        daniel_vimont Daniel Vimont added a comment - This comment refers to the zip file I just uploaded: archetype_shaded_prototype01.zip
        Hide
        daniel_vimont Daniel Vimont added a comment -

        I just uploaded a MapReduce archetype for testing and comments over on HBASE-14879. Now I'm going to make some improvements to the hbase-client and hbase-shaded-client archetypes (based on what I learned putting together the MapReduce archetype).

        Show
        daniel_vimont Daniel Vimont added a comment - I just uploaded a MapReduce archetype for testing and comments over on HBASE-14879 . Now I'm going to make some improvements to the hbase-client and hbase-shaded-client archetypes (based on what I learned putting together the MapReduce archetype).
        Hide
        ndimiduk Nick Dimiduk added a comment -

        Thanks for picking this up, Daniel Vimont. I was expecting we'd include the archetypes in our repository as maven submodules of the root-level HBase maven project. That way they stay in lock-step with the API's they're designed to consume. Is that possible? Would you mind attaching patches that introducing your work in that form? They'll be easier for folks to review.

        Show
        ndimiduk Nick Dimiduk added a comment - Thanks for picking this up, Daniel Vimont . I was expecting we'd include the archetypes in our repository as maven submodules of the root-level HBase maven project. That way they stay in lock-step with the API's they're designed to consume. Is that possible? Would you mind attaching patches that introducing your work in that form? They'll be easier for folks to review.
        Hide
        daniel_vimont Daniel Vimont added a comment -

        Yes, I'll put the new artifacts into the project as you suggest. Thanks for clarifying where they should go.

        Show
        daniel_vimont Daniel Vimont added a comment - Yes, I'll put the new artifacts into the project as you suggest. Thanks for clarifying where they should go.
        Hide
        daniel_vimont Daniel Vimont added a comment -

        Will be submitting a patch with the new archetypes soon. It's all pretty much "in the can" – just need to add a readme file now.

        Show
        daniel_vimont Daniel Vimont added a comment - Will be submitting a patch with the new archetypes soon. It's all pretty much "in the can" – just need to add a readme file now.
        Hide
        ndimiduk Nick Dimiduk added a comment -

        Excellent!

        Show
        ndimiduk Nick Dimiduk added a comment - Excellent!
        Hide
        daniel_vimont Daniel Vimont added a comment -

        Attaching patch to add infrastructure for Maven archetypes.
        After applying the patch, see README.txt in the directory 'hbase-archetypes' (new subdirectory of 'hbase') for details.

        Here is the opening paragraph of the README.txt file:
        =================
        The hbase-archetypes subproject of hbase provides an infrastructure for creation and maintenance of Maven archetypes pertinent to HBase. Upon deployment to the archetype catalog of the central Maven repository, these archetypes may be used by end-user developers to autogenerate completely configured Maven projects (including fully-functioning sample code) through invocation of the archetype:generate goal of the maven-archetype-plugin.
        ========

        The README.txt file also contains several paragraphs under the heading, "Notes for contributors to the HBase project", which explains the layout of 'hbase-archetypes', and how archetypes are created and installed into the local Maven repository, ready for deployment to the central Maven repository. It also outlines how new archetypes may be developed and added to the collection in the future.

        Show
        daniel_vimont Daniel Vimont added a comment - Attaching patch to add infrastructure for Maven archetypes. After applying the patch, see README.txt in the directory 'hbase-archetypes' (new subdirectory of 'hbase') for details. Here is the opening paragraph of the README.txt file: ================= The hbase-archetypes subproject of hbase provides an infrastructure for creation and maintenance of Maven archetypes pertinent to HBase. Upon deployment to the archetype catalog of the central Maven repository, these archetypes may be used by end-user developers to autogenerate completely configured Maven projects (including fully-functioning sample code) through invocation of the archetype:generate goal of the maven-archetype-plugin. ======== The README.txt file also contains several paragraphs under the heading, "Notes for contributors to the HBase project", which explains the layout of 'hbase-archetypes', and how archetypes are created and installed into the local Maven repository, ready for deployment to the central Maven repository. It also outlines how new archetypes may be developed and added to the collection in the future.
        Hide
        daniel_vimont Daniel Vimont added a comment -

        Note that the README text is also conveniently positioned at the top of the patch!

        Show
        daniel_vimont Daniel Vimont added a comment - Note that the README text is also conveniently positioned at the top of the patch!
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12780685/HBASE-14876.patch
        against master branch at commit 073e00c0053cb351dbba265112a6e756c932f875.
        ATTACHMENT ID: 12780685

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 16 new or modified tests.

        -1 javac. The patch appears to cause mvn compile goal to fail with Hadoop version 2.4.0.

        Compilation errors resume:
        [ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.4.0:exec (run-createArchetypes-script) on project hbase-archetypes: Command execution failed. Cannot run program "./createArchetypes.sh" (in directory "/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/hbase/hbase-archetypes"): error=13, Permission denied -> [Help 1]
        [ERROR]
        [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
        [ERROR] Re-run Maven using the -X switch to enable full debug logging.
        [ERROR]
        [ERROR] For more information about the errors and possible solutions, please read the following articles:
        [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
        [ERROR]
        [ERROR] After correcting the problems, you can resume the build with the command
        [ERROR] mvn <goals> -rf :hbase-archetypes

        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/17140//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12780685/HBASE-14876.patch against master branch at commit 073e00c0053cb351dbba265112a6e756c932f875. ATTACHMENT ID: 12780685 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 16 new or modified tests. -1 javac . The patch appears to cause mvn compile goal to fail with Hadoop version 2.4.0. Compilation errors resume: [ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.4.0:exec (run-createArchetypes-script) on project hbase-archetypes: Command execution failed. Cannot run program "./createArchetypes.sh" (in directory "/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/hbase/hbase-archetypes"): error=13, Permission denied -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn <goals> -rf :hbase-archetypes Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/17140//console This message is automatically generated.
        Hide
        daniel_vimont Daniel Vimont added a comment -

        I checked to confirm that the two scripts that are included in the patch both have file mode setting of "100755" (executable), and indeed they do. Is this a Jenkins configuration issue?

        Show
        daniel_vimont Daniel Vimont added a comment - I checked to confirm that the two scripts that are included in the patch both have file mode setting of "100755" (executable), and indeed they do. Is this a Jenkins configuration issue?
        Hide
        daniel_vimont Daniel Vimont added a comment -

        Will add execution of 'chmod +x' (also via the exec-maven-plugin) prior to execution of each script; hopefully this will clean things up.

        Show
        daniel_vimont Daniel Vimont added a comment - Will add execution of 'chmod +x' (also via the exec-maven-plugin) prior to execution of each script; hopefully this will clean things up.
        Hide
        busbey Sean Busbey added a comment -

        could we get an archetype for the hbase-spark stuff in master as well?

        Show
        busbey Sean Busbey added a comment - could we get an archetype for the hbase-spark stuff in master as well?
        Hide
        daniel_vimont Daniel Vimont added a comment -

        Tell you what – let's get this new infrastructure locked down first (along with the original three archetypes). After that, I can start knocking these things out by the dozen (or anybody else can, for that matter).

        Show
        daniel_vimont Daniel Vimont added a comment - Tell you what – let's get this new infrastructure locked down first (along with the original three archetypes). After that, I can start knocking these things out by the dozen (or anybody else can, for that matter).
        Hide
        daniel_vimont Daniel Vimont added a comment -

        Canceling patch prior to upload and submission of v2 patch.

        Show
        daniel_vimont Daniel Vimont added a comment - Canceling patch prior to upload and submission of v2 patch.
        Hide
        daniel_vimont Daniel Vimont added a comment -

        Uploading v2 patch. Revised patch corrects permissions problem by adding execution of 'chmod +x' prior to execution of scripts (all done with 'exec-maven-plugin').

        Show
        daniel_vimont Daniel Vimont added a comment - Uploading v2 patch. Revised patch corrects permissions problem by adding execution of 'chmod +x' prior to execution of scripts (all done with 'exec-maven-plugin').
        Hide
        daniel_vimont Daniel Vimont added a comment -

        Note that the 'v2' patch contains a revision meant to solve a permissions problem encountered when testing the original patch. The revised patch adds execution of 'chmod +x' prior to execution of scripts.

        Show
        daniel_vimont Daniel Vimont added a comment - Note that the 'v2' patch contains a revision meant to solve a permissions problem encountered when testing the original patch. The revised patch adds execution of 'chmod +x' prior to execution of scripts.
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12780905/HBASE-14876-v2.patch
        against master branch at commit d65978fceb85dd59e7fd66e2a93832a452e4c648.
        ATTACHMENT ID: 12780905

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 16 new or modified tests.

        +1 hadoop versions. The patch compiles with all supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0 2.7.1)

        -1 javac. The applied patch generated 41 javac compiler warnings (more than the master's current 35 warnings).

        +1 protoc. The applied patch does not increase the total number of protoc compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        -1 checkstyle. The applied patch generated new checkstyle errors. Check build console for list of new errors.

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 lineLengths. The patch introduces the following lines longer than 100:
        + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
        + + Bytes.toString(MY_COLUMN_FAMILY_NAME) + ":" + Bytes.toString(MY_FIRST_COLUMN_QUALIFIER)
        +<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
        + System.err.println(" To copy 'TestTable' to a cluster that uses replication for a 1 hour window:");
        + + "--peer.adr=server1,server2,server3:2181:/hbase --families=myOldCf:myNewCf,cf2,cf3 TestTable ");
        + + " It is recommended that you set the following to >=100. A higher value uses more memory but\n"
        + + " The following should always be set to false, to prevent writing data twice, which may produce \n"
        +<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">

        +1 site. The mvn post-site goal succeeds with this patch.

        -1 core tests. The patch failed these unit tests:

        +1 zombies. No zombie tests found running at the end of the build.

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/17156//testReport/
        Release Findbugs (version 2.0.3) warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/17156//artifact/patchprocess/newFindbugsWarnings.html
        Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/17156//artifact/patchprocess/checkstyle-aggregate.html

        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/17156//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12780905/HBASE-14876-v2.patch against master branch at commit d65978fceb85dd59e7fd66e2a93832a452e4c648. ATTACHMENT ID: 12780905 +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 16 new or modified tests. +1 hadoop versions . The patch compiles with all supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0 2.7.1) -1 javac . The applied patch generated 41 javac compiler warnings (more than the master's current 35 warnings). +1 protoc . The applied patch does not increase the total number of protoc compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. -1 checkstyle . The applied patch generated new checkstyle errors. Check build console for list of new errors. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 lineLengths . The patch introduces the following lines longer than 100: + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd "> + + Bytes.toString(MY_COLUMN_FAMILY_NAME) + ":" + Bytes.toString(MY_FIRST_COLUMN_QUALIFIER) +<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd "> + System.err.println(" To copy 'TestTable' to a cluster that uses replication for a 1 hour window:"); + + "--peer.adr=server1,server2,server3:2181:/hbase --families=myOldCf:myNewCf,cf2,cf3 TestTable "); + + " It is recommended that you set the following to >=100. A higher value uses more memory but\n" + + " The following should always be set to false, to prevent writing data twice, which may produce \n" +<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd "> +1 site . The mvn post-site goal succeeds with this patch. -1 core tests . The patch failed these unit tests: +1 zombies . No zombie tests found running at the end of the build. Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/17156//testReport/ Release Findbugs (version 2.0.3) warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/17156//artifact/patchprocess/newFindbugsWarnings.html Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/17156//artifact/patchprocess/checkstyle-aggregate.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/17156//console This message is automatically generated.
        Hide
        daniel_vimont Daniel Vimont added a comment -

        Just to be sure I'm understanding:

        I see that there are six Java examples in hbase-spark/.../example/hbasecontext. Is the idea to bundle these up (along with corresponding test objects) in a Maven archetype?

        Show
        daniel_vimont Daniel Vimont added a comment - Just to be sure I'm understanding: I see that there are six Java examples in hbase-spark/.../example/hbasecontext. Is the idea to bundle these up (along with corresponding test objects) in a Maven archetype?
        Hide
        daniel_vimont Daniel Vimont added a comment -

        Okay, the latest rejection letter from Mr. Jenkins has convinced me: I'll separate the submission of the three archetypes into three different patches (one via each of the JIRA sub-tasks to this task).

        That will leave the apparently biggest challenge for last: getting the mapreduce tests to run in the Jenkins environment.

        I will also add a fourth sub-task: an archetype to bundle up the hbase-spark examples!

        Show
        daniel_vimont Daniel Vimont added a comment - Okay, the latest rejection letter from Mr. Jenkins has convinced me: I'll separate the submission of the three archetypes into three different patches (one via each of the JIRA sub-tasks to this task). That will leave the apparently biggest challenge for last: getting the mapreduce tests to run in the Jenkins environment. I will also add a fourth sub-task: an archetype to bundle up the hbase-spark examples!
        Hide
        busbey Sean Busbey added a comment -

        FYI, I resubmitted the latest patch to get built with the just revamped precommit tests. I don't think we have any other "add a new module" changes in patch available status and I'd like to verify that it works correctly.

        Show
        busbey Sean Busbey added a comment - FYI, I resubmitted the latest patch to get built with the just revamped precommit tests. I don't think we have any other "add a new module" changes in patch available status and I'd like to verify that it works correctly.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 hbaseanti 0m 0s Patch does not have any anti-patterns.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
        +1 mvninstall 4m 8s master passed
        +1 compile 1m 45s master passed with JDK v1.8.0_66
        +1 compile 2m 2s master passed with JDK v1.7.0_91
        +1 checkstyle 11m 5s master passed
        +1 mvneclipse 1m 5s master passed
        -1 findbugs 7m 39s branch/. no findbugs output file (./target/findbugsXml.xml)
        +1 javadoc 1m 31s master passed with JDK v1.8.0_66
        +1 javadoc 2m 13s master passed with JDK v1.7.0_91
        +1 mvninstall 4m 34s the patch passed
        +1 compile 3m 37s the patch passed with JDK v1.8.0_66
        +1 javac 3m 37s the patch passed
        +1 compile 3m 33s the patch passed with JDK v1.7.0_91
        +1 javac 3m 33s the patch passed
        -1 checkstyle 11m 26s Patch generated 152 new checkstyle issues in root (total was 0, now 152).
        -1 checkstyle 0m 10s Patch generated 152 new checkstyle issues in hbase-archetypes (total was 0, now 152).
        -1 checkstyle 0m 7s Patch generated 40 new checkstyle issues in hbase-archetypes/hbase-client-project (total was 0, now 40).
        -1 checkstyle 0m 7s Patch generated 72 new checkstyle issues in hbase-archetypes/hbase-mapreduce-job (total was 0, now 72).
        -1 checkstyle 0m 7s Patch generated 40 new checkstyle issues in hbase-archetypes/hbase-shaded-client-project (total was 0, now 40).
        +1 mvneclipse 1m 44s the patch passed
        -1 shellcheck 0m 4s The applied patch generated 8 new shellcheck issues (total was 456, now 464).
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 xml 0m 2s The patch has no ill-formed XML file.
        -1 hadoopcheck 1m 35s Patch causes 11 errors with Hadoop v2.4.0.
        -1 hadoopcheck 3m 2s Patch causes 11 errors with Hadoop v2.4.1.
        -1 hadoopcheck 4m 32s Patch causes 11 errors with Hadoop v2.5.0.
        -1 hadoopcheck 6m 2s Patch causes 11 errors with Hadoop v2.5.1.
        -1 hadoopcheck 7m 32s Patch causes 11 errors with Hadoop v2.5.2.
        -1 hadoopcheck 9m 6s Patch causes 11 errors with Hadoop v2.6.1.
        -1 hadoopcheck 10m 37s Patch causes 11 errors with Hadoop v2.6.2.
        -1 hadoopcheck 12m 13s Patch causes 11 errors with Hadoop v2.6.3.
        -1 hadoopcheck 13m 43s Patch causes 11 errors with Hadoop v2.7.1.
        -1 findbugs 8m 27s patch/. no findbugs output file (./target/findbugsXml.xml)
        -1 findbugs 0m 57s patch/hbase-archetypes no findbugs output file (hbase-archetypes/target/findbugsXml.xml)
        +1 javadoc 2m 7s the patch passed with JDK v1.8.0_66
        +1 javadoc 3m 7s the patch passed with JDK v1.7.0_91
        -1 unit 84m 31s root in the patch failed with JDK v1.8.0_66.
        +1 unit 4m 55s hbase-archetypes in the patch passed with JDK v1.8.0_66.
        +1 unit 0m 52s hbase-client-project in the patch passed with JDK v1.8.0_66.
        +1 unit 3m 5s hbase-mapreduce-job in the patch passed with JDK v1.8.0_66.
        +1 unit 0m 52s hbase-shaded-client-project in the patch passed with JDK v1.8.0_66.
        +1 unit 102m 56s root in the patch passed with JDK v1.7.0_91.
        +1 unit 4m 39s hbase-archetypes in the patch passed with JDK v1.7.0_91.
        +1 unit 0m 53s hbase-client-project in the patch passed with JDK v1.7.0_91.
        +1 unit 2m 55s hbase-mapreduce-job in the patch passed with JDK v1.7.0_91.
        +1 unit 0m 53s hbase-shaded-client-project in the patch passed with JDK v1.7.0_91.
        +1 asflicense 1m 11s Patch does not generate ASF License warnings.
        306m 19s



        Reason Tests
        JDK v1.8.0_66 Timed out junit tests org.apache.hadoop.hbase.client.TestMetaWithReplicas
          org.apache.hadoop.hbase.client.TestMobRestoreSnapshotFromClient
          org.apache.hadoop.hbase.client.TestMobCloneSnapshotFromClient



        Subsystem Report/Notes
        Docker Client=1.9.1 Server=1.9.1 Image:yetus/hbase:date2016-01-08
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12780905/HBASE-14876-v2.patch
        JIRA Issue HBASE-14876
        Optional Tests asflicense shellcheck javac javadoc unit xml compile findbugs hadoopcheck hbaseanti checkstyle
        uname Linux f87095e9609d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
        git revision master / 164aeb5
        checkstyle https://builds.apache.org/job/PreCommit-HBASE-Build/29/artifact/patchprocess/diff-checkstyle-root.txt
        checkstyle https://builds.apache.org/job/PreCommit-HBASE-Build/29/artifact/patchprocess/diff-checkstyle-hbase-archetypes.txt
        checkstyle https://builds.apache.org/job/PreCommit-HBASE-Build/29/artifact/patchprocess/diff-checkstyle-hbase-archetypes_hbase-client-project.txt
        checkstyle https://builds.apache.org/job/PreCommit-HBASE-Build/29/artifact/patchprocess/diff-checkstyle-hbase-archetypes_hbase-mapreduce-job.txt
        checkstyle https://builds.apache.org/job/PreCommit-HBASE-Build/29/artifact/patchprocess/diff-checkstyle-hbase-archetypes_hbase-shaded-client-project.txt
        shellcheck v0.4.1
        shellcheck https://builds.apache.org/job/PreCommit-HBASE-Build/29/artifact/patchprocess/diff-patch-shellcheck.txt
        unit https://builds.apache.org/job/PreCommit-HBASE-Build/29/artifact/patchprocess/patch-unit-root-jdk1.8.0_66.txt
        unit test logs https://builds.apache.org/job/PreCommit-HBASE-Build/29/artifact/patchprocess/patch-unit-root-jdk1.8.0_66.txt
        JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HBASE-Build/29/testReport/
        modules C: . hbase-archetypes hbase-archetypes/hbase-client-project hbase-archetypes/hbase-mapreduce-job hbase-archetypes/hbase-shaded-client-project U: .
        Max memory used 363MB
        Powered by Apache Yetus 0.1.0 http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-HBASE-Build/29/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 hbaseanti 0m 0s Patch does not have any anti-patterns. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. +1 mvninstall 4m 8s master passed +1 compile 1m 45s master passed with JDK v1.8.0_66 +1 compile 2m 2s master passed with JDK v1.7.0_91 +1 checkstyle 11m 5s master passed +1 mvneclipse 1m 5s master passed -1 findbugs 7m 39s branch/. no findbugs output file (./target/findbugsXml.xml) +1 javadoc 1m 31s master passed with JDK v1.8.0_66 +1 javadoc 2m 13s master passed with JDK v1.7.0_91 +1 mvninstall 4m 34s the patch passed +1 compile 3m 37s the patch passed with JDK v1.8.0_66 +1 javac 3m 37s the patch passed +1 compile 3m 33s the patch passed with JDK v1.7.0_91 +1 javac 3m 33s the patch passed -1 checkstyle 11m 26s Patch generated 152 new checkstyle issues in root (total was 0, now 152). -1 checkstyle 0m 10s Patch generated 152 new checkstyle issues in hbase-archetypes (total was 0, now 152). -1 checkstyle 0m 7s Patch generated 40 new checkstyle issues in hbase-archetypes/hbase-client-project (total was 0, now 40). -1 checkstyle 0m 7s Patch generated 72 new checkstyle issues in hbase-archetypes/hbase-mapreduce-job (total was 0, now 72). -1 checkstyle 0m 7s Patch generated 40 new checkstyle issues in hbase-archetypes/hbase-shaded-client-project (total was 0, now 40). +1 mvneclipse 1m 44s the patch passed -1 shellcheck 0m 4s The applied patch generated 8 new shellcheck issues (total was 456, now 464). +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. -1 hadoopcheck 1m 35s Patch causes 11 errors with Hadoop v2.4.0. -1 hadoopcheck 3m 2s Patch causes 11 errors with Hadoop v2.4.1. -1 hadoopcheck 4m 32s Patch causes 11 errors with Hadoop v2.5.0. -1 hadoopcheck 6m 2s Patch causes 11 errors with Hadoop v2.5.1. -1 hadoopcheck 7m 32s Patch causes 11 errors with Hadoop v2.5.2. -1 hadoopcheck 9m 6s Patch causes 11 errors with Hadoop v2.6.1. -1 hadoopcheck 10m 37s Patch causes 11 errors with Hadoop v2.6.2. -1 hadoopcheck 12m 13s Patch causes 11 errors with Hadoop v2.6.3. -1 hadoopcheck 13m 43s Patch causes 11 errors with Hadoop v2.7.1. -1 findbugs 8m 27s patch/. no findbugs output file (./target/findbugsXml.xml) -1 findbugs 0m 57s patch/hbase-archetypes no findbugs output file (hbase-archetypes/target/findbugsXml.xml) +1 javadoc 2m 7s the patch passed with JDK v1.8.0_66 +1 javadoc 3m 7s the patch passed with JDK v1.7.0_91 -1 unit 84m 31s root in the patch failed with JDK v1.8.0_66. +1 unit 4m 55s hbase-archetypes in the patch passed with JDK v1.8.0_66. +1 unit 0m 52s hbase-client-project in the patch passed with JDK v1.8.0_66. +1 unit 3m 5s hbase-mapreduce-job in the patch passed with JDK v1.8.0_66. +1 unit 0m 52s hbase-shaded-client-project in the patch passed with JDK v1.8.0_66. +1 unit 102m 56s root in the patch passed with JDK v1.7.0_91. +1 unit 4m 39s hbase-archetypes in the patch passed with JDK v1.7.0_91. +1 unit 0m 53s hbase-client-project in the patch passed with JDK v1.7.0_91. +1 unit 2m 55s hbase-mapreduce-job in the patch passed with JDK v1.7.0_91. +1 unit 0m 53s hbase-shaded-client-project in the patch passed with JDK v1.7.0_91. +1 asflicense 1m 11s Patch does not generate ASF License warnings. 306m 19s Reason Tests JDK v1.8.0_66 Timed out junit tests org.apache.hadoop.hbase.client.TestMetaWithReplicas   org.apache.hadoop.hbase.client.TestMobRestoreSnapshotFromClient   org.apache.hadoop.hbase.client.TestMobCloneSnapshotFromClient Subsystem Report/Notes Docker Client=1.9.1 Server=1.9.1 Image:yetus/hbase:date2016-01-08 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12780905/HBASE-14876-v2.patch JIRA Issue HBASE-14876 Optional Tests asflicense shellcheck javac javadoc unit xml compile findbugs hadoopcheck hbaseanti checkstyle uname Linux f87095e9609d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh git revision master / 164aeb5 checkstyle https://builds.apache.org/job/PreCommit-HBASE-Build/29/artifact/patchprocess/diff-checkstyle-root.txt checkstyle https://builds.apache.org/job/PreCommit-HBASE-Build/29/artifact/patchprocess/diff-checkstyle-hbase-archetypes.txt checkstyle https://builds.apache.org/job/PreCommit-HBASE-Build/29/artifact/patchprocess/diff-checkstyle-hbase-archetypes_hbase-client-project.txt checkstyle https://builds.apache.org/job/PreCommit-HBASE-Build/29/artifact/patchprocess/diff-checkstyle-hbase-archetypes_hbase-mapreduce-job.txt checkstyle https://builds.apache.org/job/PreCommit-HBASE-Build/29/artifact/patchprocess/diff-checkstyle-hbase-archetypes_hbase-shaded-client-project.txt shellcheck v0.4.1 shellcheck https://builds.apache.org/job/PreCommit-HBASE-Build/29/artifact/patchprocess/diff-patch-shellcheck.txt unit https://builds.apache.org/job/PreCommit-HBASE-Build/29/artifact/patchprocess/patch-unit-root-jdk1.8.0_66.txt unit test logs https://builds.apache.org/job/PreCommit-HBASE-Build/29/artifact/patchprocess/patch-unit-root-jdk1.8.0_66.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HBASE-Build/29/testReport/ modules C: . hbase-archetypes hbase-archetypes/hbase-client-project hbase-archetypes/hbase-mapreduce-job hbase-archetypes/hbase-shaded-client-project U: . Max memory used 363MB Powered by Apache Yetus 0.1.0 http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HBASE-Build/29/console This message was automatically generated.
        Hide
        daniel_vimont Daniel Vimont added a comment -

        I see from the output that several new checkstyle rules seem to be in place now – most notably, a maximum line length of 80, instead of 100. Is this really the new standard max-line-length for Java modules?

        Show
        daniel_vimont Daniel Vimont added a comment - I see from the output that several new checkstyle rules seem to be in place now – most notably, a maximum line length of 80, instead of 100. Is this really the new standard max-line-length for Java modules?
        Hide
        daniel_vimont Daniel Vimont added a comment -

        Just submitted a patch for the first sub-task – HBASE-14877
        This patch introduces the overall infrastructure for creation/maintenance of Maven archetypes, and also the first archetype (which end-users will use to generate a simple Maven project with hbase-client dependency).

        After this first sub-task patch is committed, I'll submit patches for the other two originally-planned archetypes (hbase-shaded-client and hbase-mapreduce), and then (time permitting) I'll also work on the fourth archetype (for hbase-spark examples).

        Show
        daniel_vimont Daniel Vimont added a comment - Just submitted a patch for the first sub-task – HBASE-14877 This patch introduces the overall infrastructure for creation/maintenance of Maven archetypes, and also the first archetype (which end-users will use to generate a simple Maven project with hbase-client dependency). After this first sub-task patch is committed, I'll submit patches for the other two originally-planned archetypes (hbase-shaded-client and hbase-mapreduce), and then (time permitting) I'll also work on the fourth archetype (for hbase-spark examples).
        Hide
        daniel_vimont Daniel Vimont added a comment -

        Besides straightforwardly dealing with the checkstyle and ASF-license errors in my first patch submission for the subtask HBASE-14877, I'm also trying to figure out how to deal with the patch's sole findbugs error:

        Could not find resource '/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/hbase-archetypes/hbase-client-project/../dev-support/findbugs-exclude.xml'

        If anyone is familiar with this findbugs issue, please let me know. Otherwise, I'll continue to dig.

        Show
        daniel_vimont Daniel Vimont added a comment - Besides straightforwardly dealing with the checkstyle and ASF-license errors in my first patch submission for the subtask HBASE-14877 , I'm also trying to figure out how to deal with the patch's sole findbugs error: Could not find resource '/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/hbase-archetypes/hbase-client-project/../dev-support/findbugs-exclude.xml' If anyone is familiar with this findbugs issue, please let me know. Otherwise, I'll continue to dig.
        Hide
        ndimiduk Nick Dimiduk added a comment -

        Sean Busbey any thoughts on the absentee fingbugs config?

        Show
        ndimiduk Nick Dimiduk added a comment - Sean Busbey any thoughts on the absentee fingbugs config?
        Hide
        daniel_vimont Daniel Vimont added a comment -

        Okay, I see the origin of the prob in the 'findbugs' plugin configuration in hbase/pom.xml:

        <configuration>
        <excludeFilterFile>$

        {project.basedir}

        /../dev-support/findbugs-exclude.xml</excludeFilterFile>
        ...
        </configuration>

        Obviously, this was set up under the assumption that all hbase subprojects would only be one level down in the directory hierarchy, whereas the infrastructure that I've set up for archetypes has the projects two levels down (e.g., hbase-archetypes/hbase-client-project).

        I will now ponder what to do about this...

        Show
        daniel_vimont Daniel Vimont added a comment - Okay, I see the origin of the prob in the 'findbugs' plugin configuration in hbase/pom.xml: <configuration> <excludeFilterFile>$ {project.basedir} /../dev-support/findbugs-exclude.xml</excludeFilterFile> ... </configuration> Obviously, this was set up under the assumption that all hbase subprojects would only be one level down in the directory hierarchy, whereas the infrastructure that I've set up for archetypes has the projects two levels down (e.g., hbase-archetypes/hbase-client-project). I will now ponder what to do about this...
        Hide
        daniel_vimont Daniel Vimont added a comment -

        Please see my reply above.

        Show
        daniel_vimont Daniel Vimont added a comment - Please see my reply above.
        Hide
        daniel_vimont Daniel Vimont added a comment -

        Okay, here's a workaround solution to the "can't find findbugs-exclude.xml" problem -->> This workaround provides instant gratification, but grates upon my sense of propriety:

        I just added the subdirectory "hbase-archetypes/dev-support" populated with a findbugs-exclude.xml file, which has a single empty <FindBugsFilter/> element in it.

        With that added, findbugs runs just fine.

        Show
        daniel_vimont Daniel Vimont added a comment - Okay, here's a workaround solution to the "can't find findbugs-exclude.xml" problem -->> This workaround provides instant gratification, but grates upon my sense of propriety: I just added the subdirectory "hbase-archetypes/dev-support" populated with a findbugs-exclude.xml file, which has a single empty <FindBugsFilter/> element in it. With that added, findbugs runs just fine.
        Hide
        daniel_vimont Daniel Vimont added a comment -

        Alright, I've put in a much cleaner solution to the "can't find the 'findbugs-excluded.xml' file" problem (a solution that rescues my sense of propriety):

        I put an overriding plugin configuration into the hbase-archetypes/pom.xml file for the 'findbugs' plugin, so now all child projects of hbase-archetypes (e.g., hbase-archetypes/hbase-client-project) will look TWO levels up for the dev-support/findbugs-excluded.xml file. (I also included a comment explaining why the override is there.)

        Show
        daniel_vimont Daniel Vimont added a comment - Alright, I've put in a much cleaner solution to the "can't find the 'findbugs-excluded.xml' file" problem (a solution that rescues my sense of propriety): I put an overriding plugin configuration into the hbase-archetypes/pom.xml file for the 'findbugs' plugin, so now all child projects of hbase-archetypes (e.g., hbase-archetypes/hbase-client-project) will look TWO levels up for the dev-support/findbugs-excluded.xml file. (I also included a comment explaining why the override is there.)
        Hide
        busbey Sean Busbey added a comment -

        that sounds perfect. is the patch for that going into this issue or one of the child issues?

        Show
        busbey Sean Busbey added a comment - that sounds perfect. is the patch for that going into this issue or one of the child issues?
        Hide
        daniel_vimont Daniel Vimont added a comment -

        It's in the "v3" patch just submitted to HBASE-14877 (the first child of this issue, which adds the hbase-archetypes infrastructure and the first archetype*).

        *The first archetype is for a simple hbase-client dependent application.

        Show
        daniel_vimont Daniel Vimont added a comment - It's in the "v3" patch just submitted to HBASE-14877 (the first child of this issue, which adds the hbase-archetypes infrastructure and the first archetype*). *The first archetype is for a simple hbase-client dependent application.
        Hide
        daniel_vimont Daniel Vimont added a comment -

        FYI, just submitted patch for subtask HBASE-14878 (hbase-shaded-client archetype).

        Show
        daniel_vimont Daniel Vimont added a comment - FYI, just submitted patch for subtask HBASE-14878 (hbase-shaded-client archetype).
        Hide
        daniel_vimont Daniel Vimont added a comment -

        The patch for HBASE-14879 (archetype for mapreduce-job) appears to have passed muster with Hadoop QA, so it may be ready to be pushed through to the master branch (and, if time permits, cherry-picking into the 1.3 build).

        Show
        daniel_vimont Daniel Vimont added a comment - The patch for HBASE-14879 (archetype for mapreduce-job) appears to have passed muster with Hadoop QA, so it may be ready to be pushed through to the master branch (and, if time permits, cherry-picking into the 1.3 build).
        Hide
        daniel_vimont Daniel Vimont added a comment -

        I’ve always presumed that the standard usage of any of these archetypes would be along the lines of the following:

        1. User runs archetype:generate against the archetype of their choice, which generates a complete Maven project for them.
        2. They start up a local instance of HBase.
        3. They run mvn install (in their IDE or from command line) to build/test/install the project.
        4. They execute the sample code, and begin using it as an active learning tool (altering the code, then rebuilding and executing it as they please).

        I’ve just come to the full realization that step 3 will NOT work if it is preceded by step 2. The testing phase of the install process will fail with a RuntimeException saying ‘HMasterAddress already in use’. Apparently a minicluster started up by the HBaseTestingUtility cannot run in an environment where a local instance of HBase is running, because both of them will go after the same default “master info port” (16010).

        Since I don’t want there to be even the slightest of problems or frustrations for a newbie HBase user to encounter when using artifacts generated from these archetypes, I’d like to modify the Test* modules that utilize the HBaseTestingUtility (i.e. all of them) to set an alternate master info port as follows...

        Replace the current instantiation of the utility:

        private static final HBaseTestingUtility TEST_UTIL  = new HBaseTestingUtility();
        

        with the following:

          private static final HBaseTestingUtility TEST_UTIL;
          static {
            Configuration conf = HBaseConfiguration.create();
            // following setting prevents 'HMasterAddress already in use' exception
            conf.setInt(HConstants.MASTER_INFO_PORT, 16060);
            TEST_UTIL = new HBaseTestingUtility(conf);
          }
        

        Unless someone knows of a reason why this shouldn’t be done (or has an alternate suggestion for dealing with this), I guess I’ll create a new JIRA entry and provide a patch soon for both of the archetypes that are already in 2.0.0.

        Show
        daniel_vimont Daniel Vimont added a comment - I’ve always presumed that the standard usage of any of these archetypes would be along the lines of the following: User runs archetype:generate against the archetype of their choice, which generates a complete Maven project for them. They start up a local instance of HBase. They run mvn install (in their IDE or from command line) to build/test/install the project. They execute the sample code, and begin using it as an active learning tool (altering the code, then rebuilding and executing it as they please). I’ve just come to the full realization that step 3 will NOT work if it is preceded by step 2. The testing phase of the install process will fail with a RuntimeException saying ‘HMasterAddress already in use’ . Apparently a minicluster started up by the HBaseTestingUtility cannot run in an environment where a local instance of HBase is running, because both of them will go after the same default “master info port” (16010). Since I don’t want there to be even the slightest of problems or frustrations for a newbie HBase user to encounter when using artifacts generated from these archetypes, I’d like to modify the Test* modules that utilize the HBaseTestingUtility (i.e. all of them) to set an alternate master info port as follows... Replace the current instantiation of the utility: private static final HBaseTestingUtility TEST_UTIL = new HBaseTestingUtility(); with the following: private static final HBaseTestingUtility TEST_UTIL; static { Configuration conf = HBaseConfiguration.create(); // following setting prevents 'HMasterAddress already in use' exception conf.setInt(HConstants.MASTER_INFO_PORT, 16060); TEST_UTIL = new HBaseTestingUtility(conf); } Unless someone knows of a reason why this shouldn’t be done (or has an alternate suggestion for dealing with this), I guess I’ll create a new JIRA entry and provide a patch soon for both of the archetypes that are already in 2.0.0.
        Hide
        ndimiduk Nick Dimiduk added a comment -

        Perhaps it's helpful: the beforeClass method of my minicluster tests tends to look like this:

                // HBaseTestingUtility takes care of ZK, HDFS, and HBase
                UTIL = HBaseTestingUtility.createLocalHTU();
                // disable UI's on test cluster.
                UTIL.getConfiguration().setInt("hbase.master.info.port", -1);
                UTIL.getConfiguration().setInt("hbase.regionserver.info.port", -1);
                UTIL.startMiniCluster();
        

        Minicluster already grabs random ports for RPC, so this last bit allows for minicluster to launch while a local hbase is concurrently running. Perhaps, instead, these should be applied by default in HBaseTestingUtility ?

        Show
        ndimiduk Nick Dimiduk added a comment - Perhaps it's helpful: the beforeClass method of my minicluster tests tends to look like this: // HBaseTestingUtility takes care of ZK, HDFS, and HBase UTIL = HBaseTestingUtility.createLocalHTU(); // disable UI's on test cluster. UTIL.getConfiguration().setInt("hbase.master.info.port", -1); UTIL.getConfiguration().setInt("hbase.regionserver.info.port", -1); UTIL.startMiniCluster(); Minicluster already grabs random ports for RPC, so this last bit allows for minicluster to launch while a local hbase is concurrently running. Perhaps, instead, these should be applied by default in HBaseTestingUtility ?
        Hide
        daniel_vimont Daniel Vimont added a comment -

        The JIRA "spam lockdown" appears to be over, so I can finally post this...

        Yes, definitely the more comprehensive solution would appear to be – setting those port values to -1 ("grab a random port") in the constructor of HBaseTestingUtility.

        Unless someone can think of a reason not to, I'll create a new JIRA entry and add something like the following lines of code to the end of the main HBaseTestingUtility constructor method:

            this.getConfiguration().setInt(HConstants.MASTER_INFO_PORT, -1);
            this.getConfiguration().setInt(HConstants.REGIONSERVER_PORT, -1);
        

        The only drawback I see to these two lines of code is that they do not allow the user to explicitly define a specific port of their choosing for the MASTER_INFO_PORT and REGIONSERVER_PORT values. To allow for such user-defined overrides, something like this could be added instead:

            Configuration conf = this.getConfiguration();
            if (conf.getInt(HConstants.MASTER_INFO_PORT, HConstants.DEFAULT_MASTER_INFOPORT)
                    == HConstants.DEFAULT_MASTER_INFOPORT) {
              conf.setInt(HConstants.MASTER_INFO_PORT, -1);
            }
            if (conf.getInt(HConstants.REGIONSERVER_PORT, HConstants.DEFAULT_REGIONSERVER_PORT)
                    == HConstants.DEFAULT_REGIONSERVER_PORT) {
              conf.setInt(HConstants.REGIONSERVER_PORT, -1);
            }
        

        I tend to think this second snippet represents the way to go.

        Show
        daniel_vimont Daniel Vimont added a comment - The JIRA "spam lockdown" appears to be over, so I can finally post this... Yes, definitely the more comprehensive solution would appear to be – setting those port values to -1 ("grab a random port") in the constructor of HBaseTestingUtility. Unless someone can think of a reason not to, I'll create a new JIRA entry and add something like the following lines of code to the end of the main HBaseTestingUtility constructor method: this .getConfiguration().setInt(HConstants.MASTER_INFO_PORT, -1); this .getConfiguration().setInt(HConstants.REGIONSERVER_PORT, -1); The only drawback I see to these two lines of code is that they do not allow the user to explicitly define a specific port of their choosing for the MASTER_INFO_PORT and REGIONSERVER_PORT values. To allow for such user-defined overrides, something like this could be added instead: Configuration conf = this .getConfiguration(); if (conf.getInt(HConstants.MASTER_INFO_PORT, HConstants.DEFAULT_MASTER_INFOPORT) == HConstants.DEFAULT_MASTER_INFOPORT) { conf.setInt(HConstants.MASTER_INFO_PORT, -1); } if (conf.getInt(HConstants.REGIONSERVER_PORT, HConstants.DEFAULT_REGIONSERVER_PORT) == HConstants.DEFAULT_REGIONSERVER_PORT) { conf.setInt(HConstants.REGIONSERVER_PORT, -1); } I tend to think this second snippet represents the way to go.
        Hide
        jmhsieh Jonathan Hsieh added a comment -

        Sounds wroth trying (in a separate jira) to see if everything blows up or if most things pass.

        Show
        jmhsieh Jonathan Hsieh added a comment - Sounds wroth trying (in a separate jira) to see if everything blows up or if most things pass.
        Hide
        daniel_vimont Daniel Vimont added a comment -

        Yep! Did it in HBASE-15835. As you can see over there, I ran into one wrinkle in my first patch submission, but everything is fine with that patch now, apparently ready for pushing through into the master branch...

        Show
        daniel_vimont Daniel Vimont added a comment - Yep! Did it in HBASE-15835 . As you can see over there, I ran into one wrinkle in my first patch submission, but everything is fine with that patch now, apparently ready for pushing through into the master branch...

          People

          • Assignee:
            daniel_vimont Daniel Vimont
            Reporter:
            ndimiduk Nick Dimiduk
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:

              Development