HBase
  1. HBase
  2. HBASE-5363

Automatically run rat check on mvn release builds

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.90.5, 0.92.0
    • Fix Version/s: 0.90.6, 0.92.1, 0.94.0
    • Component/s: build
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Some of the recent hbase release failed rat checks (mvn rat:check). We should add checks likely in the mvn package phase so that this becomes a non-issue in the future.

      Here's an example from Whirr:
      https://github.com/apache/whirr/blob/trunk/pom.xml line 388 for an example.

      1. hbase-5363-0.90.patch
        2 kB
        Jonathan Hsieh
      2. hbase-5363.patch
        4 kB
        Jonathan Hsieh
      3. hbase-5363.2.patch
        4 kB
        Jonathan Hsieh

        Activity

        Hide
        stack added a comment -

        We already have this. See our pom:

                <plugin>
                  <groupId>org.apache.rat</groupId>
                  <artifactId>apache-rat-plugin</artifactId>
                  <version>0.7</version>
                    <configuration>
                      <excludes>
                        <exclude>**/.*</exclude>
                        <exclude>**/target/**</exclude>
                        <exclude>**/CHANGES.txt</exclude>
                        <exclude>**/CHANGES.txt</exclude>
                        <exclude>**/generated/**</exclude>
                        <exclude>**/conf/*</exclude>
                        <exclude>**/*.avpr</exclude>
                        <exclude>**/control</exclude>
                        <exclude>**/conffile</exclude>
                        <exclude>**/8e8ab58dcf39412da19833fcd8f687ac</exclude>
                        <!--It don't like freebsd license-->
                        <exclude>src/site/resources/css/freebsd_docbook.css</exclude>
                      </excludes>
                    </configuration>
                </plugin>
        

        And see the generated report: http://hbase.apache.org/rat-report.html

        Here is the issue where I messed w/ this stuff on trunk: HBASE-4647

        Show
        stack added a comment - We already have this. See our pom: <plugin> <groupId>org.apache.rat</groupId> <artifactId>apache-rat-plugin</artifactId> <version>0.7</version> <configuration> <excludes> <exclude>**/.*</exclude> <exclude>**/target/**</exclude> <exclude>**/CHANGES.txt</exclude> <exclude>**/CHANGES.txt</exclude> <exclude>**/generated/**</exclude> <exclude>**/conf/*</exclude> <exclude>**/*.avpr</exclude> <exclude>**/control</exclude> <exclude>**/conffile</exclude> <exclude>**/8e8ab58dcf39412da19833fcd8f687ac</exclude> <!--It don't like freebsd license--> <exclude>src/site/resources/css/freebsd_docbook.css</exclude> </excludes> </configuration> </plugin> And see the generated report: http://hbase.apache.org/rat-report.html Here is the issue where I messed w/ this stuff on trunk: HBASE-4647
        Hide
        Jonathan Hsieh added a comment -

        Stack. I see but I believe the part that I want to add is the part that executes it (executions section) and fails the tarballs are built ('mvn package').

           <plugin>
                <groupId>org.apache.rat</groupId>
                <artifactId>apache-rat-plugin</artifactId>
                <version>0.8</version>
                <executions>
                  <execution>
                    <phase>package</phase>
                    <goals>
                      <goal>check</goal>
                    </goals>
                  </execution>
                </executions>
        
        Show
        Jonathan Hsieh added a comment - Stack. I see but I believe the part that I want to add is the part that executes it (executions section) and fails the tarballs are built ('mvn package'). <plugin> <groupId>org.apache.rat</groupId> <artifactId>apache-rat-plugin</artifactId> <version>0.8</version> <executions> <execution> <phase> package </phase> <goals> <goal>check</goal> </goals> </execution> </executions>
        Hide
        Jonathan Hsieh added a comment -

        .. fails the tarballs when being built ('mvn package').

        Show
        Jonathan Hsieh added a comment - .. fails the tarballs when being built ('mvn package').
        Hide
        Todd Lipcon added a comment -

        Can we enable this only with a -Prelease build? Or part of the assembly phase? The package phase runs for every "mvn install", which is already super slow.. I think we could move this and the source jar generation to a different phase, or selectively enable with a -Prelease profile?

        Show
        Todd Lipcon added a comment - Can we enable this only with a -Prelease build? Or part of the assembly phase? The package phase runs for every "mvn install", which is already super slow.. I think we could move this and the source jar generation to a different phase, or selectively enable with a -Prelease profile?
        Hide
        Jonathan Hsieh added a comment -

        The rat site suggests running it in the mvn verify phase.

        http://incubator.apache.org/rat/apache-rat-plugin/examples/verify.html

        I added the chunk above and rat didn't seem to be run or fail the build. Investigating further to figure out which phase seems reasonable..

        As an aside, we should probably update the http://wiki.apache.org/hadoop/Hbase/HowToRelease wiki page now that the build has diverged from hadoop's now (we don't have any ant).

        Show
        Jonathan Hsieh added a comment - The rat site suggests running it in the mvn verify phase. http://incubator.apache.org/rat/apache-rat-plugin/examples/verify.html I added the chunk above and rat didn't seem to be run or fail the build. Investigating further to figure out which phase seems reasonable.. As an aside, we should probably update the http://wiki.apache.org/hadoop/Hbase/HowToRelease wiki page now that the build has diverged from hadoop's now (we don't have any ant).
        Hide
        stack added a comment -

        The rat plugin is pretty crappy in my experience too. Its hard to get it behave.

        Show
        stack added a comment - The rat plugin is pretty crappy in my experience too. Its hard to get it behave.
        Hide
        Elliott Clark added a comment -

        Though it is getting several files that need licenses.

        Show
        Elliott Clark added a comment - Though it is getting several files that need licenses.
        Hide
        Elliott Clark added a comment -

        Attached a patch to add headers to some files that were missing it according to rat.
        I couldn't get the rat plugin to run automatically on the first try so the patch doesn't have any pom changes in it.

        Show
        Elliott Clark added a comment - Attached a patch to add headers to some files that were missing it according to rat. I couldn't get the rat plugin to run automatically on the first try so the patch doesn't have any pom changes in it.
        Hide
        Jonathan Hsieh added a comment -

        Hey Elliott, thanks for the patch! Let me ask a favor – could you post the license fixes patch to this jira HBASE-5364?

        Thanks!

        Show
        Jonathan Hsieh added a comment - Hey Elliott, thanks for the patch! Let me ask a favor – could you post the license fixes patch to this jira HBASE-5364 ? Thanks!
        Hide
        Jonathan Hsieh added a comment -

        This patch adds rat checks to the package phase of the mvn build when the release profile is activated (-Prelease).

        It cleans up some of the rat excludes (ignore auto gen stuff), and at least on trunk id's 5 legit license violations.

        Show
        Jonathan Hsieh added a comment - This patch adds rat checks to the package phase of the mvn build when the release profile is activated (-Prelease). It cleans up some of the rat excludes (ignore auto gen stuff), and at least on trunk id's 5 legit license violations.
        Hide
        Jonathan Hsieh added a comment -

        To run, us 'mvn package -Prelease -DskipTests'. Note that the excludes are not used if you use 'mvn rat:check'. I need help from a mavenista to figure out how to make that work without copy paste.

        Show
        Jonathan Hsieh added a comment - To run, us 'mvn package -Prelease -DskipTests'. Note that the excludes are not used if you use 'mvn rat:check'. I need help from a mavenista to figure out how to make that work without copy paste.
        Hide
        Jonathan Hsieh added a comment -

        @Elliott, since you posted on the other jira, I deleted the attachment that you placed here.

        Show
        Jonathan Hsieh added a comment - @Elliott, since you posted on the other jira, I deleted the attachment that you placed here.
        Hide
        Jonathan Hsieh added a comment -

        patch applies on 0.92, reports 1 error (src/docbkx/build.xml)

        Show
        Jonathan Hsieh added a comment - patch applies on 0.92, reports 1 error (src/docbkx/build.xml)
        Hide
        Jonathan Hsieh added a comment -

        made a 0.90 version, reports about 40 errors (probably similar to the 0.92 branch errors)

        Show
        Jonathan Hsieh added a comment - made a 0.90 version, reports about 40 errors (probably similar to the 0.92 branch errors)
        Hide
        Jonathan Hsieh added a comment -

        v2 of the trunk/0.92 patch. v1 likely breaks website report generation.

        Show
        Jonathan Hsieh added a comment - v2 of the trunk/0.92 patch. v1 likely breaks website report generation.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12514046/hbase-5363.2.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        -1 javadoc. The javadoc tool appears to have generated -136 warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 156 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.hbase.io.hfile.TestHFileBlock

        Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/933//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/933//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/933//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12514046/hbase-5363.2.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javadoc. The javadoc tool appears to have generated -136 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 156 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.io.hfile.TestHFileBlock Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/933//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/933//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/933//console This message is automatically generated.
        Hide
        Jonathan Hsieh added a comment -

        Figured it out. Kind of evil.

        'mvn rat:check' is wrong.

        you are supposed to use

        'mvn apache-rat:check'.

        I'm tried to update this wiki but don't have perms:
        http://wiki.apache.org/hadoop/Hbase/HowToRelease

        Show
        Jonathan Hsieh added a comment - Figured it out. Kind of evil. 'mvn rat:check' is wrong. you are supposed to use 'mvn apache-rat:check'. I'm tried to update this wiki but don't have perms: http://wiki.apache.org/hadoop/Hbase/HowToRelease
        Hide
        stack added a comment -

        Whats your hadoop wikiid Jon?

        Show
        stack added a comment - Whats your hadoop wikiid Jon?
        Hide
        Jonathan Hsieh added a comment -

        JonathanHsieh

        Sent from my iPhone

        Show
        Jonathan Hsieh added a comment - JonathanHsieh Sent from my iPhone
        Hide
        stack added a comment -

        Try it now boss....

        Show
        stack added a comment - Try it now boss....
        Hide
        Jonathan Hsieh added a comment -

        Thanks! Works. Updated wiki.

        Show
        Jonathan Hsieh added a comment - Thanks! Works. Updated wiki.
        Hide
        stack added a comment -

        Committed 0.90 and 0.92 branches as well as to trunk

        Show
        stack added a comment - Committed 0.90 and 0.92 branches as well as to trunk
        Hide
        Hudson added a comment -

        Integrated in HBase-0.92 #277 (See https://builds.apache.org/job/HBase-0.92/277/)
        HBASE-5363 Automatically run rat check on mvn release builds (Revision 1243163)

        Result = SUCCESS
        stack :
        Files :

        • /hbase/branches/0.92/CHANGES.txt
        • /hbase/branches/0.92/pom.xml
        Show
        Hudson added a comment - Integrated in HBase-0.92 #277 (See https://builds.apache.org/job/HBase-0.92/277/ ) HBASE-5363 Automatically run rat check on mvn release builds (Revision 1243163) Result = SUCCESS stack : Files : /hbase/branches/0.92/CHANGES.txt /hbase/branches/0.92/pom.xml
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK-security #109 (See https://builds.apache.org/job/HBase-TRUNK-security/109/)
        HBASE-5363 Automatically run rat check on mvn release builds (Revision 1243164)

        Result = FAILURE
        stack :
        Files :

        • /hbase/trunk/pom.xml
        Show
        Hudson added a comment - Integrated in HBase-TRUNK-security #109 (See https://builds.apache.org/job/HBase-TRUNK-security/109/ ) HBASE-5363 Automatically run rat check on mvn release builds (Revision 1243164) Result = FAILURE stack : Files : /hbase/trunk/pom.xml
        Hide
        Hudson added a comment -

        Integrated in HBase-TRUNK #2660 (See https://builds.apache.org/job/HBase-TRUNK/2660/)
        HBASE-5363 Automatically run rat check on mvn release builds (Revision 1243164)

        Result = FAILURE
        stack :
        Files :

        • /hbase/trunk/pom.xml
        Show
        Hudson added a comment - Integrated in HBase-TRUNK #2660 (See https://builds.apache.org/job/HBase-TRUNK/2660/ ) HBASE-5363 Automatically run rat check on mvn release builds (Revision 1243164) Result = FAILURE stack : Files : /hbase/trunk/pom.xml
        Hide
        Hudson added a comment -

        Integrated in HBase-0.92-security #91 (See https://builds.apache.org/job/HBase-0.92-security/91/)
        HBASE-5363 Automatically run rat check on mvn release builds (Revision 1243163)

        Result = FAILURE
        stack :
        Files :

        • /hbase/branches/0.92/CHANGES.txt
        • /hbase/branches/0.92/pom.xml
        Show
        Hudson added a comment - Integrated in HBase-0.92-security #91 (See https://builds.apache.org/job/HBase-0.92-security/91/ ) HBASE-5363 Automatically run rat check on mvn release builds (Revision 1243163) Result = FAILURE stack : Files : /hbase/branches/0.92/CHANGES.txt /hbase/branches/0.92/pom.xml

          People

          • Assignee:
            Jonathan Hsieh
            Reporter:
            Jonathan Hsieh
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development