Patch: HBASE-14877 - add Maven archetype infrastructure and hbase-client archetype

Inserting Release Notes for this sub-task, and submitting HBASE-14877.patch

-1 overall

This message was automatically generated.

Fixed checkstyle, findbugs, and asflicense problems that were encountered with first patch submission.

-1 overall

This message was automatically generated.

I will continue to play whack-a-mole with the findbugs issues (last time it couldn't find its input; now it can't find its output).

Was relieved to read the e-mail about "Machine H2 in the precommit pool fails on hadoopchecks", and to find that the run above (with all of the unexpected, melodramatically red hadoopcheck failures) was executed on that very machine.

I'm in the process of making further adjustments to the findbugs plugin overrides that I'm putting into hbase/hbase-archetypes/pom.xml, but while I'm doing that, I think it's worth noting a peculiarity in the precommit-build-output above:

The "findbugs" step is failing in the master branch before my patch is applied. The message says that "no findbugs output file" is found in the root's ./target subdirectory. As far as I can see, the lack of findbugs output is to be expected, since there is NO java code in the root's ./src subdirectory for findbugs to inspect.

After my "v2" patch is applied, similar findbugs "failures" are reported for new subprojects that also contain no java code in their ./src subdirectory: hbase-archetypes and hbase-archetypes/hbase-archetype-builder.

Sorry to bother you again Sean Busbey but I haven't been able to keep up with your precommit improvements. You have any ideas on the findbugs output for poor Daniel here?

Submitting v3 patch with better-structured <plugin> entries in hbase/hbase-archetypes/pom.xml with regard to findbugs, emulating the entries in the findbugs configuration in the root pom.xml file.

Also added missing JUnit @Category annotations to the test class in hbase-archetypes/hbase-client-project (so now the unit tests there will run properly!).

-1 overall

This message was automatically generated.

no worries! always happy to help new folks to HBase and downstream users of Yetus, so this is a two-for-one.

The "findbugs" step is failing in the master branch before my patch is applied. The message says that "no findbugs output file" is found in the root's ./target subdirectory. As far as I can see, the lack of findbugs output is to be expected, since there is NO java code in the root's ./src subdirectory for findbugs to inspect.

Yes, yetus by default reports on the health of the repository both before and after a given patch. this is both so that folks can generally be aware of how the code base is doing (for example, there are now hbase jiras for fixing the extent findbugs errors in some modules) and so that patches that are fixing findbugs errors are obvious.

The particular bug of "no java code gives a findbugs -1" is YETUS-271, which will be fixed in the next release. If all of the modules that are failing findbugs have no java code, then I'd say they can safely be ignored.

Thanks for confirming my suspicions re: the findbugs "failures".

In the latest precommit-build-output (build #142, just above this comment), it was run through the now-notorious machine H2, and so received the predictable (and hopefully bogus) failure notifications on all hadoopcheck steps.

Thus, as far as I can see, this patch is ready for review by the powers-that-be for a potential commit through to the master branch*. (Although before that it might be nice to see a precommit run with all the hadoopcheck steps displayed in a pleasant shade of green.)

*I would appreciate it if all who review this patch could do a thoughtful reading of the README.txt file that the patch inserts into the new hbase/hbase-archetypes subdirectory, since this patch introduces (and "commits" us to) a new infrastructure for creation and maintenance of our Maven archetypes into the future. I intentionally tried to keep the README text short and to-the-point, and I do not explain my rationale for setting things up the way I did. So feel free to ask for clarification of any aspect of the new infrastructure. (If this needs to be opened up for a broader discussion on the developers mailing-list, then that would be great.)

Just wanted to check: Is there anything else I need to do to shepherd this patch forward from this point? (If it simply needs to be reviewed by others – I realize you're all very busy, so please just get to it when you can.) I wanted to be sure nobody's waiting on me to do something.

Hi Daniel Vimont these patches are on the short end of my todo list. Sorry but I might not get to it until the weekend.

No problem! I can tell by the message traffic flying by on the developers list that you're very busy!!

Submitting patch "v4" with clarified comments in code and README.txt.

As usual, README.txt is easily viewable at top of patch.

-1 overall

This message was automatically generated.

Sean Busbey: Hopefully, this latest patch submission will help you in fine-tuning the Yetus/Jenkins settings.

Notably, a new checkstyle problem was possibly identified (if the notification is to be trusted) in hbase-archetype-builder (which underwent no modifications since the last patch submission). However, when I click on the link above to go to the checkstyle report, all I get is a "requested resource is not available" notification.

Also, in the unit testing above there was a failure (or at least the report of a failure) in hbase-server, which my patch has nothing to do with.

Interesting. The checkstyle plugin generated an incorrect URL to the results. (you can find the files by browsing to the build artifacts from the test build). I'll manually trigger a new run with the unreleased version of Yetus to see if we need to file a bug or if that's been fixed already.

The file mentioned, maven-patch-checkstyle-hbase-archetypes_hbase-archetype-builder.txt, is empty. If you look at the maven output for the other module's checkstyle run and compare it to the one that is listed as failing, I think that message says "I couldn't run checkstyle" similar to the errors given for findbugs, rather than stating that there are checkstyle identified problems.

Also, in the unit testing above there was a failure (or at least the report of a failure) in hbase-server, which my patch has nothing to do with.

Yetus thought it needed to do a top level build when checking the patch. This is undesirable both because hbase-server still has some flakey tests and because checking them adds >4 hours to the build time. Deciding to do a test at the top level is probably because your patch touches the top level pom. I'll see if we can figure out a way to flag "add new modules" as an exception that doesn't queue unneeded tests.

actual review feedback below

The README is a very nice touch! You should update the ref guide's section on building and developing HBase to add a note that folks should seek out that file when working on the archetypes. I went back and forth on if this text needs to all be in the ref guide, since folks are likely to miss a README in a subdirectory. But I think the current ref guide is too confused in its audience for that to work. Since we're committing to how some maintenance will work in the future, it's worth you starting a thread on dev@hbase that directs folks attention to this ticket and specifically the README. Make sure hte thread has a subject line that starts with "[DISCUSS]", e.g. "[DISCUSS] new build tooling for creation of hbase application maven archetypes."

Similar to the above, I don't see any updates to the ref guide section on developing applications to instruct folks on using the archetypes we provide. Maybe a follow on ticket added as another subtask to the parent jira?

The README needs to have a file header so that we're in compliance with ASF policy. You can see examples of the needed text in other files in the repo. It would probably be easiest (and most readable) if you changed from a plain text file to a markdown file so that you can make the header metadata.

• example rendered README: https://github.com/apache/hbase/blob/master/dev-support/hbase_docker/README.md
• raw version showing the header: https://raw.githubusercontent.com/apache/hbase/master/dev-support/hbase_docker/README.md

This is great! Some suggestions:

• Consider converting the README to Asciidoc or Markdown, so that it is rendered nicely on Github. It would also help with navigation of the footnotes. I can help with this if you need it.
• Please add a section to http://hbase.apache.org/book.html#hbase_apis, before the Examples section, with a brief (one-paragraph?) explanation of Maven archetypes and how the HBase archetypes can help developers get started quickly, and a pointer to your README. The source for that chapter is in src/main/asciidoc/_chapters/hbase_apis.adoc.

Please let me know if I can be of any assistance at all.

+1 overall

This message was automatically generated.

+1 asflicense 0m 58s Patch does not generate ASF License warnings.

Not your problem here, natch, but this is concerning since we know the README is missing a header.

README.md
Yes, I much prefer using markdown for a README file, so it'll be a pleasure to convert it. (I saw only README.txt files in the root folder and a few others, and had incorrectly assumed that the project standard was txt files.) I'll also include the standard ASF license header (which, btw, is also missing from the README.txt files in the root directory and the hbase-examples directory – and perhaps also missing from others).

END-USER DOCUMENTATION
Note importantly that the README text I've prepared is directed specifically at the audience of HBase contributors and committers. It gets into the "how-to" of producing new archetypes. The end-user can remain blissfully unaware of all of this; all the end-user will need to do is execute a single Maven command:
mvn archetype:generate -Dfilter=org.apache:hbase
to interactively bring up a list of publicly-available HBase archetypes from which they may choose.

Regarding end-user documentation, I hope it's okay to leave that to a separate subtask. I would like to successfully get at least one publicly-available archetype loaded into the central Maven archetype directory before finalizing end-user documentation (which will explain to the end-user how to retrieve and use such an archetype). It would potentially sow confusion if we publish the instructions before we finish tinkering around with deployment of our first archetype to make it available.

DEPLOYMENT
In order to achieve deployment of a new hbase archetype to the Maven central repository, it looks to me like we have to backport the archetype infrastructure (i.e. this patch) to a publicly-available release – at least 1.1.2 (and any other version for which archetypes should be made available) – http://mvnrepository.com/artifact/org.apache.hbase/hbase – Is it correct that we need to do such a backport, or am I missing something?

I originally developed this infrastructure against v1.1.2, so backporting to that version, from my point of view, should be very easy.

DISCUSSION ABOUT ALL THIS ON dev@hbase
Yep, I'll launch a discussion as recommended very soon!

README.md
Yes, I much prefer using markdown for a README file, so it'll be a pleasure to convert it. (I saw only README.txt files in the root folder and a few others, and had incorrectly assumed that the project standard was txt files.) I'll also include the standard ASF license header (which, btw, is also missing from the README.txt files in the root directory and the hbase-examples directory – and perhaps also missing from others).

no standard. just earlier mistakes that we have yet to correct.

END-USER DOCUMENTATION
Note importantly that the README text I've prepared is directed specifically at the audience of HBase contributors and committers. It gets into the "how-to" of producing new archetypes. The end-user can remain blissfully unaware of all of this; all the end-user will need to do is execute a single Maven command:
mvn archetype:generate -Dfilter=org.apache:hbase
to interactively bring up a list of publicly-available HBase archetypes from which they may choose.

This is part of the confused audience for the reference guide I mentioned earlier. Currently the reference guide has many audiences: downstream application developers, operators, and HBase contributors. It's where our canonical docs live for e.g. making new contributions, building and running tests, etc. Hence both my concern that this will be missed if it is not at all present as well as my concern that the document isn't the best place for all of this.

Regarding end-user documentation, I hope it's okay to leave that to a separate subtask. I would like to successfully get at least one publicly-available archetype loaded into the central Maven archetype directory before finalizing end-user documentation (which will explain to the end-user how to retrieve and use such an archetype). It would potentially sow confusion if we publish the instructions before we finish tinkering around with deployment of our first archetype to make it available.

Yep, definitely fine to make it a follow on subtask. I'd just ask that the task exist prior to this jira getting closed out.

DEPLOYMENT
In order to achieve deployment of a new hbase archetype to the Maven central repository, it looks to me like we have to backport the archetype infrastructure (i.e. this patch) to a publicly-available release – at least 1.1.2 (and any other version for which archetypes should be made available) – http://mvnrepository.com/artifact/org.apache.hbase/hbase – Is it correct that we need to do such a backport, or am I missing something?

I originally developed this infrastructure against v1.1.2, so backporting to that version, from my point of view, should be very easy.

The archetypes are a new feature, so they can only be added in a minor version release. we'll probably want to make a separate backport jira to bring this to branch-1. the discussion on dev@hbase should probably include what level of "done" we're looking for before we do that. (as a point of reference, a similar discussion resulted in HBASE-14160 for backporting the hbase-spark module.)

Just added a new subtask (HBASE-15176) for "end-user-oriented documentation".

I think I've got the README.md formatting looking the way I want it to now. I posted it into a test project over on github to view it via their rendering: https://github.com/dvimont/test_hbasearchetypes_readme

Submitting v5 of the patch, which differs from previous versions of patch as follows:

(1) README.md (markdown format) replaces README.txt
(2) A new brief section has been added at the end of developers.adoc, pointing contributors to the README.md file (for info on the structure of hbase-archetypes and how to add new archetypes). The new section is currently numbered 148.9 in book.html when the site is generated.

-1 overall

This message was automatically generated.

Thanks for being patient and sorry for the long gap in feedback.

I'm fairly certain the unit test failures on the last test run are not your concern.

I did a review pass – I gave it a spin as as pretty happy. Great docs, esp good pointers to more resources. I agree with sean on the follow up items. This would be a great addition to the hbase 1.3 (the next minor version release sean alluded to).

I basically ran 'mvn install -DskipTests' on hbase with the patch applied, and then went to a new dir and ran 'mvn archetype:generate -Dfilter=hbase'. a few questions answered and voila!

I have a few questions – I'm not am expert on xsl / archetypes, and have some nits/questions from the poms/java:

hbase-client-project/pom.xml:

• change the java 1.7 compile source/target in the pom? should this be to the inherited from hbase ${compileSource} ? something that the user specifies? is that a mvn archetype convention?
• nit: should ${project.version} be something like ${hbase.project.version} so as not to confuse the 'hbase' being the project or the app on hbase as the project? or is this a "baked in" convention with archetype definitions?
  • ah after running 'mvn archetype:generate -Dfilter=hbase' i see this gets replaced with the hbase project version. So we'll just inherit whatever version hbase has.

(edit: minor formatting fixes)

Both your points relate to converting the exemplar (which SHOULD have all proper dependencies/inheritances within the hbase project hierarchy) into a standalone project (which needs to have all hbase project dependencies removed/replaced before it is used to produce an archetype). The conversion of the exemplar is elaborated upon a bit in footnote 5 of the README.

To address your points:

change the java 1.7 compile source/target in the pom? should this be inherited from hbase $compileSource?

YES, thanks for pointing this out! You are absolutely right that the value "1.7" should NOT be hardcoded into hbase-client-project/pom.xml in the maven.compiler.source and maven.compiler.target elements, but should be inherited via $compileSource. Then, just as $project.version is converted into a literal value by the maven-resources-plugin (see footnote 5 again), so too will $compileSource be converted into a literal value.

ah after running 'mvn archetype:generate -Dfilter=hbase' i see $project.version gets replaced with the hbase project version. So we'll just inherit whatever version hbase has.

EXACTLY! The variable $project.version gets replaced by an appropriate literal value by the maven-resources-plugin.

I will make $compileSource related changes according to your first bullet point above (also modifying the text of footnote 5 accordingly) and submit a revised patch.

BTW, HBASE-15176 has already been added for "end-user-oriented documentation". As soon as the first v1.3 archetype becomes publicly available via the Maven central repository, I'll add an appropriate entry to the end-user Reference Guide text.

THANKS!

Submitting v6 of the patch, which differs from previous versions of patch as follows:

(1) In hbase-archetypes/hbase-client-project/pom.xml, the elements maven.compiler.source and maven.compiler.target were incorrectly hard-coded with the value "1.7"; this has been replaced by the variable $compileSource.
(2) Footnote #5 in the hbase-archetypes/README.md file has been modified to explain that, prior to archetype creation, Maven resource filtering will replace the $compileSource variable with the literal value of the Java version being used for compilation.

-1 overall

This message was automatically generated.

It would be great if we could move ahead with this now. Please let me know if there are any additional steps that I need to take to get this pushed through into 2.0.

Thanks!

Hey Daniel Vimont, I've committed to master. Thanks for the reviews Nick Dimiduk, Sean Busbey.

I've kicked off a run with all the latest yetus features – I believe that may have resulted in the one clean precommit report. The findbugs warnings shouldn't affect subsequent patch precommit builds, but if it does I may revert.

FAILURE: Integrated in HBase-Trunk_matrix #722 (See https://builds.apache.org/job/HBase-Trunk_matrix/722/)
HBASE-14877 maven archetype: client application (Daniel Vimont) (jmhsieh: rev afa63a91d6500076178312e86288ac2393800145)

• hbase-archetypes/hbase-client-project/src/main/resources/log4j.properties
• hbase-archetypes/hbase-archetype-builder/installArchetypes.sh
• hbase-archetypes/README.md
• pom.xml
• hbase-archetypes/pom.xml
• hbase-archetypes/hbase-client-project/src/test/java/org/apache/hbase/archetypes/exemplars/client/TestHelloHBase.java
• src/main/asciidoc/_chapters/developer.adoc
• hbase-archetypes/hbase-client-project/src/main/java/org/apache/hbase/archetypes/exemplars/client/HelloHBase.java
• hbase-archetypes/hbase-archetype-builder/pom.xml
• hbase-archetypes/hbase-client-project/src/main/java/org/apache/hbase/archetypes/exemplars/client/package-info.java
• hbase-archetypes/hbase-client-project/pom.xml
• hbase-archetypes/hbase-archetype-builder/modify_exemplar_pom.xsl
• hbase-archetypes/hbase-archetype-builder/modify_archetype_pom.xsl
• hbase-archetypes/hbase-archetype-builder/createArchetypes.sh

-1 overall

This message was automatically generated.

Yup, looks clean (unit tests is not you) with the newer yetus version. Thanks again!

If "archetypes" needs to be added to the components list, feel free to put me down as the "owner"; or if it fits under one of the existing component categories, then an update of the description of that component would probably be in order.

Is there any timeframe on the 1.3 release? I will look into back-porting this item to 1.3 first (making sure there are no unanticipated "gotchas" there) before moving on to the other archetype subtasks (shaded-client, mapreduce, etc.).

Also, has a Grand Poobah of version 1.3 been assigned yet? Looking at the list of release managers only shows managers through 1.2.

BTW, I just put out a new entry on my blog about all this.

Is there any timeframe on the 1.3 release? I will look into back-porting this item to 1.3 first (making sure there are no unanticipated "gotchas" there) before moving on to the other archetype subtasks (shaded-client, mapreduce, etc.).

Not yet. we should have a formal [DISCUSS] thread on dev@ about setting a target. IIRC, the flagship features folks wanted into 1.3 were spark support and MOB.

Also, has a Grand Poobah of version 1.3 been assigned yet? Looking at the list of release managers only shows managers through 1.2.

Not yet, also something that should be determined in the [DISCUSS] thread.

Okay, since time-frames and other 1.3-oriented details are yet-to-be-determined, I'll go ahead with the other subtasks (HBASE-14878, HBASE-14879, and HBASE-15081) to get those into the master branch, while waiting for 1.3 plans to firm up.

FYI, just submitted patch for subtask HBASE-14878 (hbase-shaded-client archetype).

Mikhail Antonov – this would be the first archetype-related patch to backport to branch-1. (HBASE-14878 is dependent upon this one.)

I'm not sure whether it's appropriate to add a new "fixVersion" value to this JIRA issue, or to create a new JIRA that refers back to this one (as Sean Busbey advises in the comment above that he posted 27-Jan-13:07).

Let me know of any way I might be of assistance with the backport process.

I'd say - only open a new jira if backporting is some actual work; if cherry-picking from master applies with no errors, there's probably no need to open a separate jira for that.

Okay, I just added "1.3.0" to the fix-versions for this JIRA entry.

BTW, I have doubts that the patch/commit that worked against the master branch will work (in a cherry-pick) against branch-1. While most of the patch/commit simply adds new directories and files (which I presume would work in a cherry-pick scenario), the patch also (1) includes the insertion of a single new line into hbase/pom.xml, and (2) inserts new lines into developer.adoc.

However, I may not fully understand how the cherry-pick function works in this environment. (Maybe it's smarter than I imagine it to be!)

if you're going to do the backport here, please mark the issue as re-opened.

it's okay if cherry-picking requires a bit of massaging. You can post the branch-specific version here if you'd like precommit checks or if it ends up being substantively different from the master version.

Reopening as per Sean Busbey's advice.

Mikhail Antonov – will you be taking this from here, to incorporate this into branch-1, and on through to 1.3? If you need anything further from me, please let me know.

THANKS!

Yeah, I'll look at it tomorrow or Fri. Will ping you if I need something

Was my back-burner for a while, cherry-picked and committed to branch-1 and branch-1.3; didn't have to do almost any changes except version changes in poms

SUCCESS: Integrated in HBase-1.3-IT #717 (See https://builds.apache.org/job/HBase-1.3-IT/717/)
HBASE-14877 maven archetype: client application (Daniel Vimont) (antonov: rev f1ec4227fb57411490ab0484ba6685a0268e4a24)

• hbase-archetypes/hbase-client-project/src/main/java/org/apache/hbase/archetypes/exemplars/client/package-info.java
• hbase-archetypes/hbase-archetype-builder/modify_archetype_pom.xsl
• hbase-archetypes/hbase-client-project/src/main/resources/log4j.properties
• hbase-archetypes/hbase-client-project/src/test/java/org/apache/hbase/archetypes/exemplars/client/TestHelloHBase.java
• src/main/asciidoc/_chapters/developer.adoc
• hbase-archetypes/hbase-archetype-builder/modify_exemplar_pom.xsl
• hbase-archetypes/hbase-archetype-builder/pom.xml
• hbase-archetypes/pom.xml
• hbase-archetypes/hbase-client-project/pom.xml
• hbase-archetypes/hbase-archetype-builder/installArchetypes.sh
• hbase-archetypes/hbase-client-project/src/main/java/org/apache/hbase/archetypes/exemplars/client/HelloHBase.java
• hbase-archetypes/README.md
• pom.xml
• hbase-archetypes/hbase-archetype-builder/createArchetypes.sh

FAILURE: Integrated in HBase-1.4 #225 (See https://builds.apache.org/job/HBase-1.4/225/)
HBASE-14877 maven archetype: client application (Daniel Vimont) (antonov: rev 8b9840b4a8c4313b2432806358fc7a176f1e0855)

• hbase-archetypes/hbase-archetype-builder/pom.xml
• hbase-archetypes/hbase-client-project/src/test/java/org/apache/hbase/archetypes/exemplars/client/TestHelloHBase.java
• hbase-archetypes/hbase-client-project/src/main/resources/log4j.properties
• hbase-archetypes/hbase-archetype-builder/installArchetypes.sh
• hbase-archetypes/hbase-client-project/src/main/java/org/apache/hbase/archetypes/exemplars/client/package-info.java
• pom.xml
• hbase-archetypes/hbase-archetype-builder/modify_archetype_pom.xsl
• hbase-archetypes/hbase-archetype-builder/modify_exemplar_pom.xsl
• hbase-archetypes/hbase-client-project/pom.xml
• hbase-archetypes/hbase-archetype-builder/createArchetypes.sh
• hbase-archetypes/pom.xml
• hbase-archetypes/hbase-client-project/src/main/java/org/apache/hbase/archetypes/exemplars/client/HelloHBase.java
• hbase-archetypes/README.md
• src/main/asciidoc/_chapters/developer.adoc

FAILURE: Integrated in HBase-1.3 #747 (See https://builds.apache.org/job/HBase-1.3/747/)
HBASE-14877 maven archetype: client application (Daniel Vimont) (antonov: rev f1ec4227fb57411490ab0484ba6685a0268e4a24)

• hbase-archetypes/hbase-archetype-builder/installArchetypes.sh
• hbase-archetypes/hbase-archetype-builder/pom.xml
• hbase-archetypes/pom.xml
• hbase-archetypes/hbase-archetype-builder/modify_exemplar_pom.xsl
• hbase-archetypes/hbase-client-project/src/main/java/org/apache/hbase/archetypes/exemplars/client/HelloHBase.java
• src/main/asciidoc/_chapters/developer.adoc
• pom.xml
• hbase-archetypes/hbase-client-project/src/main/java/org/apache/hbase/archetypes/exemplars/client/package-info.java
• hbase-archetypes/README.md
• hbase-archetypes/hbase-archetype-builder/modify_archetype_pom.xsl
• hbase-archetypes/hbase-client-project/src/main/resources/log4j.properties
• hbase-archetypes/hbase-client-project/src/test/java/org/apache/hbase/archetypes/exemplars/client/TestHelloHBase.java
• hbase-archetypes/hbase-archetype-builder/createArchetypes.sh
• hbase-archetypes/hbase-client-project/pom.xml