Lucene - Core
  1. Lucene - Core
  2. LUCENE-4570

Release ForbiddenAPI checker on Google Code

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.2, Trunk
    • Component/s: general/build
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      Currently there is source code in lucene/tools/src (e.g. Forbidden APIs checker ant task).

      It would be convenient if you could download this thing in your ant build from ivy (especially if maybe it included our definitions .txt files as resources).

      In general checking for locale/charset violations in this way is a pretty general useful thing for a server-side app.

      Can we either release lucene-tools.jar as an artifact, or maybe alternatively move this somewhere else as a standalone project and suck it in ourselves?

      1. LUCENE-4570.patch
        60 kB
        Uwe Schindler
      2. LUCENE-4570.patch
        61 kB
        Uwe Schindler
      3. LUCENE-4570.patch
        61 kB
        Uwe Schindler
      4. LUCENE-4570.patch
        60 kB
        Uwe Schindler
      5. LUCENE-4570.patch
        55 kB
        Uwe Schindler
      6. LUCENE-4570-maven.patch
        85 kB
        Steve Rowe
      7. LUCENE-4570-maven.patch
        99 kB
        Steve Rowe
      8. LUCENE-4570-maven-inherited.patch
        72 kB
        Steve Rowe
      9. LUCENE-4570-maven-inherited.patch
        73 kB
        Steve Rowe

        Issue Links

          Activity

          Hide
          Uwe Schindler added a comment -

          I think we should release the whole stuff, including parts of my blog post (http://blog.thetaphi.de/2012/07/default-locales-default-charsets-and.html) as documentation. The question is: Are there any restrictions from the Apache side by re-releasing source code (+ the TXT files) under a different name?

          I will go ahead already and add a ASF license header on top of our forbiddenApis/*.txt files!

          Show
          Uwe Schindler added a comment - I think we should release the whole stuff, including parts of my blog post ( http://blog.thetaphi.de/2012/07/default-locales-default-charsets-and.html ) as documentation. The question is: Are there any restrictions from the Apache side by re-releasing source code (+ the TXT files) under a different name? I will go ahead already and add a ASF license header on top of our forbiddenApis/*.txt files!
          Hide
          Robert Muir added a comment -

          +1 for the blog post as documentation. Especially the portion in red text!

          Show
          Robert Muir added a comment - +1 for the blog post as documentation. Especially the portion in red text!
          Hide
          Dawid Weiss added a comment -

          +1 for making it reusable. It should be built into javac

          Show
          Dawid Weiss added a comment - +1 for making it reusable. It should be built into javac
          Hide
          Commit Tag Bot added a comment -

          [branch_4x commit] Uwe Schindler
          http://svn.apache.org/viewvc?view=revision&revision=1412599

          LUCENE-4570: Add missing license headers

          Show
          Commit Tag Bot added a comment - [branch_4x commit] Uwe Schindler http://svn.apache.org/viewvc?view=revision&revision=1412599 LUCENE-4570 : Add missing license headers
          Hide
          Commit Tag Bot added a comment -

          [trunk commit] Uwe Schindler
          http://svn.apache.org/viewvc?view=revision&revision=1412598

          LUCENE-4570: Add missing license headers

          Show
          Commit Tag Bot added a comment - [trunk commit] Uwe Schindler http://svn.apache.org/viewvc?view=revision&revision=1412598 LUCENE-4570 : Add missing license headers
          Hide
          Uwe Schindler added a comment - - edited

          I started a google code project: http://code.google.com/p/forbidden-apis/

          This is a fork with many new additions:

          • auto-generated deprecated signature list (from rt.jar)
          • support for "bundled" and project-maintained signature lists (like the deprecated ones for various JDK versions, the well known charset/locale/... violators)
          • no direct ASM 4.1 dependency conflicting with other dependencies: The ASM library is jarjar'ed into the artifact
          • not yet: Comments for every signature thats printed in error message
          • not yet: Mäven support (Mojo) -> Selckin already started a fork in Github, but as the new project is almost a complete rewrite of the API (decouple ANT task from logic), I will need his help
          • not yet: Mäven Release, so IVY can download it

          Once there is a release (hopefully soon), this can ivy:cachepath'ed and taskdef'ed into the Lucene build

          Show
          Uwe Schindler added a comment - - edited I started a google code project: http://code.google.com/p/forbidden-apis/ This is a fork with many new additions: auto-generated deprecated signature list (from rt.jar) support for "bundled" and project-maintained signature lists (like the deprecated ones for various JDK versions, the well known charset/locale/... violators) no direct ASM 4.1 dependency conflicting with other dependencies: The ASM library is jarjar'ed into the artifact not yet: Comments for every signature thats printed in error message not yet: Mäven support (Mojo) -> Selckin already started a fork in Github, but as the new project is almost a complete rewrite of the API (decouple ANT task from logic), I will need his help not yet: Mäven Release, so IVY can download it Once there is a release (hopefully soon), this can ivy:cachepath'ed and taskdef'ed into the Lucene build
          Hide
          Dawid Weiss added a comment -

          Nice!

          Show
          Dawid Weiss added a comment - Nice!
          Hide
          Uwe Schindler added a comment -

          I already committed the comments in signatures to the googlecode repo, so the error message now contains explanation of the forbidden api (e.g. deprecated, uses default charset,...).

          Remaining issue is Mäven support, which is not important for Lucene.

          Show
          Uwe Schindler added a comment - I already committed the comments in signatures to the googlecode repo, so the error message now contains explanation of the forbidden api (e.g. deprecated, uses default charset,...). Remaining issue is Mäven support, which is not important for Lucene.
          Hide
          Uwe Schindler added a comment -

          The forbidden-api checker is now available on sonatype-snapshots with the maven-coordinates:

          groupId=de.thetaphi
          artifactId=forbiddenapis
          version=1.0-SNAPSHOT
          

          Attached is a patch for Lucene trunk, removing the forbidden api checker from checkout and use the snapshot version. To enable the download of snapshots, I added for now (until it is released) the sonatype-snapshots repo to ivy-settings.xml.

          There is some cleanup needed in the patch:

          • It somehow relies on tools compiled, otherwise some properties are not defined, to locate the txt files. This can be solved by placing the not-bundled lucene-specific signature files outside tools (where its no longer need to be). Just place solr ones in solr and lucene ones in lucene.
          • I have to review the API files and also move e.g. commons-io.txt to the checker JAR file, so we have more bundled signatures and dont need to maintain them inside lucene. This of course does not apply to specific solr/lucene ones to prevent specific test patterns.
          Show
          Uwe Schindler added a comment - The forbidden-api checker is now available on sonatype-snapshots with the maven-coordinates: groupId=de.thetaphi artifactId=forbiddenapis version=1.0-SNAPSHOT Attached is a patch for Lucene trunk, removing the forbidden api checker from checkout and use the snapshot version. To enable the download of snapshots, I added for now (until it is released) the sonatype-snapshots repo to ivy-settings.xml. There is some cleanup needed in the patch: It somehow relies on tools compiled, otherwise some properties are not defined, to locate the txt files. This can be solved by placing the not-bundled lucene-specific signature files outside tools (where its no longer need to be). Just place solr ones in solr and lucene ones in lucene. I have to review the API files and also move e.g. commons-io.txt to the checker JAR file, so we have more bundled signatures and dont need to maintain them inside lucene. This of course does not apply to specific solr/lucene ones to prevent specific test patterns.
          Hide
          Uwe Schindler added a comment - - edited

          By the way: The new checker finds use of a deprecated API, that was missing from the hand-made jdk-deprecated.txt: File.toURL(). Its used at three places in analyzers - which is a bummer, because it will prevent using those analyzers on configs where the lucene files are in a directory with e.g. umlauts or other special symbols (see deprecated message).

          Here the message:

          -check-forbidden-jdk-apis:
          [forbidden-apis] Reading bundled API signatures: jdk-unsafe-1.6
          [forbidden-apis] Reading bundled API signatures: jdk-deprecated-1.6
          [forbidden-apis] Reading API signatures: C:\Users\Uwe Schindler\Projects\lucene\trunk-lusolr3\lucene\tools\forbiddenApis\executors.txt
          [forbidden-apis] Loading classes to check...
          [forbidden-apis] Scanning for API signatures and dependencies...
          [forbidden-apis] Forbidden method invocation: java.io.File#toURL() [Deprecated in Java 1.6]
          [forbidden-apis]   in org.apache.lucene.analysis.compound.hyphenation.PatternParser (PatternParser.java:101)
          [forbidden-apis] Forbidden method invocation: java.io.File#toURL() [Deprecated in Java 1.6]
          [forbidden-apis]   in org.apache.lucene.analysis.compound.HyphenationCompoundWordTokenFilter (HyphenationCompoundWordTokenFilter.java:151)
          [forbidden-apis] Forbidden method invocation: java.io.File#toURL() [Deprecated in Java 1.6]
          [forbidden-apis]   in org.apache.lucene.analysis.compound.hyphenation.HyphenationTree (HyphenationTree.java:114)
          [forbidden-apis] Scanned 5468 (and 432 related) class file(s) for forbidden API invocations (in 2.29s), 3 error(s).
          
          Show
          Uwe Schindler added a comment - - edited By the way: The new checker finds use of a deprecated API, that was missing from the hand-made jdk-deprecated.txt: File.toURL(). Its used at three places in analyzers - which is a bummer, because it will prevent using those analyzers on configs where the lucene files are in a directory with e.g. umlauts or other special symbols (see deprecated message). Here the message: -check-forbidden-jdk-apis: [forbidden-apis] Reading bundled API signatures: jdk-unsafe-1.6 [forbidden-apis] Reading bundled API signatures: jdk-deprecated-1.6 [forbidden-apis] Reading API signatures: C:\Users\Uwe Schindler\Projects\lucene\trunk-lusolr3\lucene\tools\forbiddenApis\executors.txt [forbidden-apis] Loading classes to check... [forbidden-apis] Scanning for API signatures and dependencies... [forbidden-apis] Forbidden method invocation: java.io.File#toURL() [Deprecated in Java 1.6] [forbidden-apis] in org.apache.lucene.analysis.compound.hyphenation.PatternParser (PatternParser.java:101) [forbidden-apis] Forbidden method invocation: java.io.File#toURL() [Deprecated in Java 1.6] [forbidden-apis] in org.apache.lucene.analysis.compound.HyphenationCompoundWordTokenFilter (HyphenationCompoundWordTokenFilter.java:151) [forbidden-apis] Forbidden method invocation: java.io.File#toURL() [Deprecated in Java 1.6] [forbidden-apis] in org.apache.lucene.analysis.compound.hyphenation.HyphenationTree (HyphenationTree.java:114) [forbidden-apis] Scanned 5468 (and 432 related) class file(s) for forbidden API invocations (in 2.29s), 3 error(s).
          Hide
          Uwe Schindler added a comment -

          I fixed the violations for now...

          Show
          Uwe Schindler added a comment - I fixed the violations for now...
          Hide
          Commit Tag Bot added a comment -

          [trunk commit] Uwe Schindler
          http://svn.apache.org/viewvc?view=revision&revision=1435146

          LUCENE-4570: Fix deprecated API usage (otherwise may lead to bugs if Hyphenation filters load files from directories with non-ascii path names)

          Show
          Commit Tag Bot added a comment - [trunk commit] Uwe Schindler http://svn.apache.org/viewvc?view=revision&revision=1435146 LUCENE-4570 : Fix deprecated API usage (otherwise may lead to bugs if Hyphenation filters load files from directories with non-ascii path names)
          Hide
          Commit Tag Bot added a comment -

          [branch_4x commit] Uwe Schindler
          http://svn.apache.org/viewvc?view=revision&revision=1435148

          Merged revision(s) 1435146 from lucene/dev/trunk:
          LUCENE-4570: Fix deprecated API usage (otherwise may lead to bugs if Hyphenation filters load files from directories with non-ascii path names)

          Show
          Commit Tag Bot added a comment - [branch_4x commit] Uwe Schindler http://svn.apache.org/viewvc?view=revision&revision=1435148 Merged revision(s) 1435146 from lucene/dev/trunk: LUCENE-4570 : Fix deprecated API usage (otherwise may lead to bugs if Hyphenation filters load files from directories with non-ascii path names)
          Hide
          Uwe Schindler added a comment -

          Here a new patch for Lucene+Solr, that makes use of all new features and bundled signatures shipped with the forked forbidden-api checker:

          • it now forbids to call sun.* and com.sun.* APIs from rt.jar
          • new signatures files: jdk-system-out, commons-io-unsafe (now keyed by the version number). The commons-io version number is now in common-build.xml and used by forbidden-apis and ivy.xml (to be always consistent).

          I am shortly before releasing the forbidden API checker. Once this is done, Steven Rowe can hopefully help to add the plugin to the Maven build.

          To test this, you have to delete de.thetaphi from your ~/ivy2/cache folder, so IVY dowinloads the new snapshot version.

          Show
          Uwe Schindler added a comment - Here a new patch for Lucene+Solr, that makes use of all new features and bundled signatures shipped with the forked forbidden-api checker: it now forbids to call sun.* and com.sun.* APIs from rt.jar new signatures files: jdk-system-out, commons-io-unsafe (now keyed by the version number). The commons-io version number is now in common-build.xml and used by forbidden-apis and ivy.xml (to be always consistent). I am shortly before releasing the forbidden API checker. Once this is done, Steven Rowe can hopefully help to add the plugin to the Maven build. To test this, you have to delete de.thetaphi from your ~/ivy2/cache folder, so IVY dowinloads the new snapshot version.
          Hide
          Uwe Schindler added a comment -

          Updated patch: use complete classpath, so the checks are more correct (as every class used can be resolved)

          In general, the classpath is very hacky. Theoretically, the forbidden api checks should be done per module and after the javac runs (with uptodate). Then every module could use the standard classpath to run the checks. The next version of forbidden apis will throw errors by default if a symbol is undefined while parsing.

          Show
          Uwe Schindler added a comment - Updated patch: use complete classpath, so the checks are more correct (as every class used can be resolved) In general, the classpath is very hacky. Theoretically, the forbidden api checks should be done per module and after the javac runs (with uptodate). Then every module could use the standard classpath to run the checks. The next version of forbidden apis will throw errors by default if a symbol is undefined while parsing.
          Hide
          Uwe Schindler added a comment -

          Fix classpath. Lucene works, Solr still has problems, the thorough checks were disabled. It is currently impossible to generate a top-level classpath that contains all classes and referenced jars.

          The only fix for this is to make the forbidden-checks per module!

          Show
          Uwe Schindler added a comment - Fix classpath. Lucene works, Solr still has problems, the thorough checks were disabled. It is currently impossible to generate a top-level classpath that contains all classes and referenced jars. The only fix for this is to make the forbidden-checks per module!
          Hide
          Uwe Schindler added a comment -

          de.thetaphi:fobiddenapis:1.0 was released to sonatype release repository.

          Attached is the patch to make use of this version 1.0 in Lucene. The patch still has some TODOs for a later stage, when we should really make the forbidden-checks per module. As a workaround, Solr checks are currently not so thourough, because the full classpath is not available on top-level. We can now also add the check to the Maven build.

          I will update the documentation on the project web page tomorrow, while waiting for the release to be pushed to maven central.

          The attached patch already works, as Lucene's IVY config has Sonatype available as default repository.

          Show
          Uwe Schindler added a comment - de.thetaphi:fobiddenapis:1.0 was released to sonatype release repository. Attached is the patch to make use of this version 1.0 in Lucene. The patch still has some TODOs for a later stage, when we should really make the forbidden-checks per module. As a workaround, Solr checks are currently not so thourough, because the full classpath is not available on top-level. We can now also add the check to the Maven build. I will update the documentation on the project web page tomorrow, while waiting for the release to be pushed to maven central. The attached patch already works, as Lucene's IVY config has Sonatype available as default repository.
          Hide
          Uwe Schindler added a comment -
          Show
          Uwe Schindler added a comment - For those who want to try out/download: http://oss.sonatype.org/content/repositories/releases/de/thetaphi/forbiddenapis/1.0/
          Hide
          Uwe Schindler added a comment -

          Seems to be released on Maven Central, too: http://repo1.maven.org/maven2/de/thetaphi/forbiddenapis/1.0/

          I will commit the patch later and open a new issue to make the checks per-module.

          Show
          Uwe Schindler added a comment - Seems to be released on Maven Central, too: http://repo1.maven.org/maven2/de/thetaphi/forbiddenapis/1.0/ I will commit the patch later and open a new issue to make the checks per-module.
          Hide
          Commit Tag Bot added a comment -
          Show
          Commit Tag Bot added a comment - [trunk commit] Uwe Schindler http://svn.apache.org/viewvc?view=revision&revision=1442508 LUCENE-4570 : typo
          Hide
          Commit Tag Bot added a comment -

          [trunk commit] Uwe Schindler
          http://svn.apache.org/viewvc?view=revision&revision=1442507

          LUCENE-4570: Use the Policeman Formbidden API checker, released separately from Lucene and downloaded via Ivy

          Show
          Commit Tag Bot added a comment - [trunk commit] Uwe Schindler http://svn.apache.org/viewvc?view=revision&revision=1442507 LUCENE-4570 : Use the Policeman Formbidden API checker, released separately from Lucene and downloaded via Ivy
          Hide
          Uwe Schindler added a comment -

          I committed the attached patch to trunk and 4.x. I am now working on completing the documentation on the Google Code page: http://code.google.com/p/forbidden-apis/

          Thanks to the "default locale/charset/timezone ghostbuster"!

          Show
          Uwe Schindler added a comment - I committed the attached patch to trunk and 4.x. I am now working on completing the documentation on the Google Code page: http://code.google.com/p/forbidden-apis/ Thanks to the "default locale/charset/timezone ghostbuster"!
          Hide
          Commit Tag Bot added a comment -

          [branch_4x commit] Uwe Schindler
          http://svn.apache.org/viewvc?view=revision&revision=1442509

          Merged revision(s) 1442507-1442508 from lucene/dev/trunk:
          LUCENE-4570: Use the Policeman Forbidden API checker, released separately from Lucene and downloaded via Ivy
          LUCENE-4570: typo

          Show
          Commit Tag Bot added a comment - [branch_4x commit] Uwe Schindler http://svn.apache.org/viewvc?view=revision&revision=1442509 Merged revision(s) 1442507-1442508 from lucene/dev/trunk: LUCENE-4570 : Use the Policeman Forbidden API checker, released separately from Lucene and downloaded via Ivy LUCENE-4570 : typo
          Hide
          Steve Rowe added a comment -

          Patch adding forbidden-api calls to the Maven build.

          Currently fails when running against test classes because the MavenMojo hard-codes using the compile-scope classpath: https://code.google.com/p/forbidden-apis/source/browse/trunk/src/main/java/de/thetaphi/forbiddenapis/MavenMojo.java#139 - here's the error I get:

          [ERROR] Failed to execute goal de.thetaphi:forbiddenapis:1.1:forbiddenapis (check-forbidden-jdk-apis) on project lucene-core-tests: Check for forbidden API calls failed: java.lang.ClassNotFoundException: Loading of class org.apache.lucene.util.LuceneTestCase failed: Not found -> [Help 1]
          

          Uwe, can you look into adding a way for MavenMojo to change which classpath to use? Maybe look at the name of the phase to which an execution is bound, and if it contains the word "test" (e.g. "process-test-classes", use the test-scope classpath instead of the compile-scope classpath?

          Show
          Steve Rowe added a comment - Patch adding forbidden-api calls to the Maven build. Currently fails when running against test classes because the MavenMojo hard-codes using the compile-scope classpath: https://code.google.com/p/forbidden-apis/source/browse/trunk/src/main/java/de/thetaphi/forbiddenapis/MavenMojo.java#139 - here's the error I get: [ERROR] Failed to execute goal de.thetaphi:forbiddenapis:1.1:forbiddenapis (check-forbidden-jdk-apis) on project lucene-core-tests: Check for forbidden API calls failed: java.lang.ClassNotFoundException: Loading of class org.apache.lucene.util.LuceneTestCase failed: Not found -> [Help 1] Uwe, can you look into adding a way for MavenMojo to change which classpath to use? Maybe look at the name of the phase to which an execution is bound, and if it contains the word "test" (e.g. "process-test-classes", use the test-scope classpath instead of the compile-scope classpath?
          Hide
          Uwe Schindler added a comment -

          Hi Steve,

          it is a known problem that the Mojo currently cannot handle test classes (I think thats documented, unless I missed to add it!). So it is currently not supported to change the default phase. I will work on this in a later version - the way to go in this case is to have 2 separate Mojos (this is how other plugins does it) or like you mentioned, perhaps add a check on the current phase. The problem with the latter is: The mojo defines its dependency resolution mode (see annotation), and this cannot be changed at runtime.

          For now, I would only check the main classes (which works without any other configuration settings) and leave the tests away.

          Show
          Uwe Schindler added a comment - Hi Steve, it is a known problem that the Mojo currently cannot handle test classes (I think thats documented, unless I missed to add it!). So it is currently not supported to change the default phase. I will work on this in a later version - the way to go in this case is to have 2 separate Mojos (this is how other plugins does it) or like you mentioned, perhaps add a check on the current phase. The problem with the latter is: The mojo defines its dependency resolution mode (see annotation) , and this cannot be changed at runtime. For now, I would only check the main classes (which works without any other configuration settings) and leave the tests away.
          Show
          Uwe Schindler added a comment - I opened: https://code.google.com/p/forbidden-apis/issues/detail?id=4
          Hide
          Steve Rowe added a comment -

          Patch that enables maven usage of forbidden-apis, against 1.2-SNAPSHOT. Works for me.

          This is not committable as-is: there is a snapshot plugin repository declaration (for oss.sonatype.org) and dependency on a -SNAPSHOT version.

          Show
          Steve Rowe added a comment - Patch that enables maven usage of forbidden-apis, against 1.2-SNAPSHOT. Works for me. This is not committable as-is: there is a snapshot plugin repository declaration (for oss.sonatype.org) and dependency on a -SNAPSHOT version.
          Hide
          Uwe Schindler added a comment -

          Thanks! Looks good, I only dont like the repeated code in every module. Maybe we can improve that later.

          I am working on fixing more bugs at the moment (Äpple Java 6 on OSX was not detected as valid JDK because the directory layout of this JDK was not laid out according to the "Oracle Java standard", after fixing this I had problems with IBM J9 v7.0. Detecting booclasspath is not easy and most code available on the web is incorrect!

          Once I am ready to release I will update the dependency in Lucene and you can commit that patch with the released version.

          Show
          Uwe Schindler added a comment - Thanks! Looks good, I only dont like the repeated code in every module. Maybe we can improve that later. I am working on fixing more bugs at the moment (Äpple Java 6 on OSX was not detected as valid JDK because the directory layout of this JDK was not laid out according to the "Oracle Java standard", after fixing this I had problems with IBM J9 v7.0. Detecting booclasspath is not easy and most code available on the web is incorrect! Once I am ready to release I will update the dependency in Lucene and you can commit that patch with the released version.
          Hide
          Steve Rowe added a comment - - edited

          Looks good, I only dont like the repeated code in every module. Maybe we can improve that later.

          Yeah, I don't like it either, but Maven doesn't have a directly supported way to find the base directory in a multi-module build; this is required to locate the rule files.

          I looked around the web for solutions today, and I found two possible alternatives to the technique I use (i.e.: have each module define its own ${top-level} property pointing to the project base directory in relative terms):

          1. package resources into a jar on which each sub-module depends; the plugin can then extract resources from the jar
          2. use an embedded groovy script to set recursively find the project base directory, then set a property containing the absolutized directory.

          I think I've tried #2 in the past, but I'll try again.

          Show
          Steve Rowe added a comment - - edited Looks good, I only dont like the repeated code in every module. Maybe we can improve that later. Yeah, I don't like it either, but Maven doesn't have a directly supported way to find the base directory in a multi-module build; this is required to locate the rule files. I looked around the web for solutions today, and I found two possible alternatives to the technique I use (i.e.: have each module define its own ${top-level } property pointing to the project base directory in relative terms): package resources into a jar on which each sub-module depends; the plugin can then extract resources from the jar use an embedded groovy script to set recursively find the project base directory, then set a property containing the absolutized directory. I think I've tried #2 in the past, but I'll try again.
          Hide
          Steve Rowe added a comment -

          Patch reducing forbiddenapi plugin configuration to a minimum.

          I had to leave the previous approach (hard-coded per-module relative path to project basedir) in place to specify (re)source directories, but for the rest, including locating forbiddenapi resources, I was able to write a one-line Groovy script to set a property containing the absolute project basedir. This patch also removes redundant maven-surefire-plugin configuration in Solr.

          I tested on OS X with Maven 3.0.3 and 2.2.1, and on Win7+Cygwin with Maven 2.2.1. All work.

          Show
          Steve Rowe added a comment - Patch reducing forbiddenapi plugin configuration to a minimum. I had to leave the previous approach (hard-coded per-module relative path to project basedir) in place to specify (re)source directories, but for the rest, including locating forbiddenapi resources, I was able to write a one-line Groovy script to set a property containing the absolute project basedir. This patch also removes redundant maven-surefire-plugin configuration in Solr. I tested on OS X with Maven 3.0.3 and 2.2.1, and on Win7+Cygwin with Maven 2.2.1. All work.
          Hide
          Dawid Weiss added a comment -

          Uwe – the location of standard libraries is indeed a problem under various distros. J9 in particular has things scattered all over different JAR archives. My trick of the past was to enumerate several library classes (from different packages) and locate their bytecode based on their own class loader's getResourceAsURL call pointing to those classes' bytecode. This still won't work for certain classes that are native in J9, for example.

          I needed it for proguard but it's essentially the same problem – where the heck the library classes are.

          public class DetectRtJar {
            public static void main(String[] args) throws Exception {
              Set<File> jars = new TreeSet<File>();
          
              Class<?> [] keyClasses = {
                  java.lang.annotation.Annotation.class,
                  java.lang.management.ManagementFactory.class,
                  java.util.logging.Logger.class,
                  java.awt.Component.class,
                  java.beans.BeanDescriptor.class,
                  java.io.File.class,
                  java.lang.Object.class,
                  java.math.BigDecimal.class,
                  java.net.URL.class,
                  java.nio.Buffer.class,
                  java.security.Security.class,
                  java.sql.Array.class,
                  java.text.Collator.class,
                  java.util.List.class,
                  java.util.concurrent.ConcurrentHashMap.class,
                  java.util.zip.ZipEntry.class,
                  org.w3c.dom.Document.class,
              };
          
              ClassLoader cl = ClassLoader.getSystemClassLoader();
              for (Class<?> clazz : keyClasses) {
                URL url = cl.getResource(
                    clazz.getName().replace('.', '/') + ".class");
                if (url.getProtocol().equals("jar")) {
                  JarURLConnection juc = (JarURLConnection) url.openConnection();
                  jars.add(new File(juc.getJarFile().getName()));
                } else {
                  // Other scheme? wtf?
                  throw new RuntimeException("Unknown scheme: " + url.toString());
                }
              }
          
              StringBuilder b = new StringBuilder();
              for (File f : jars) {
                b.append("-libraryjar ").append(f.getAbsolutePath());
                b.append("(java/**)");
                b.append("\n");
              }
            }
          } 
          
          Show
          Dawid Weiss added a comment - Uwe – the location of standard libraries is indeed a problem under various distros. J9 in particular has things scattered all over different JAR archives. My trick of the past was to enumerate several library classes (from different packages) and locate their bytecode based on their own class loader's getResourceAsURL call pointing to those classes' bytecode. This still won't work for certain classes that are native in J9, for example. I needed it for proguard but it's essentially the same problem – where the heck the library classes are. public class DetectRtJar { public static void main( String [] args) throws Exception { Set<File> jars = new TreeSet<File>(); Class <?> [] keyClasses = { java.lang.annotation.Annotation.class, java.lang.management.ManagementFactory.class, java.util.logging.Logger.class, java.awt.Component.class, java.beans.BeanDescriptor.class, java.io.File.class, java.lang. Object .class, java.math.BigDecimal.class, java.net.URL.class, java.nio.Buffer.class, java.security.Security.class, java.sql.Array.class, java.text.Collator.class, java.util.List.class, java.util.concurrent.ConcurrentHashMap.class, java.util.zip.ZipEntry.class, org.w3c.dom.Document.class, }; ClassLoader cl = ClassLoader .getSystemClassLoader(); for ( Class <?> clazz : keyClasses) { URL url = cl.getResource( clazz.getName().replace('.', '/') + ".class" ); if (url.getProtocol().equals( "jar" )) { JarURLConnection juc = (JarURLConnection) url.openConnection(); jars.add( new File(juc.getJarFile().getName())); } else { // Other scheme? wtf? throw new RuntimeException( "Unknown scheme: " + url.toString()); } } StringBuilder b = new StringBuilder(); for (File f : jars) { b.append( "-libraryjar " ).append(f.getAbsolutePath()); b.append( "(java/**)" ); b.append( "\n" ); } } }
          Hide
          Uwe Schindler added a comment - - edited

          Thanks Dawid! I had the same problem with J9 like you explained (e.g. java.lang.Object is in some crazy extension dir). I solved the problem in the exact same way like you did (checking the URLConnection instance type and use more exotic class names). The final solution I comitted was easier: RuntimeMXBean.getBootClassPath() and is supported on all recent JDKs (although its documented that it may be unsupported, they have a boolean getter to detect this). I build a Set from the classpath components (JAR files and directories). When loading the classes later, i lookup the JarFile's of the class that was loaded by the checker to find out if it is a RuntimeClass or not: https://code.google.com/p/forbidden-apis/source/browse/trunk/src/main/java/de/thetaphi/forbiddenapis/Checker.java#88

          Show
          Uwe Schindler added a comment - - edited Thanks Dawid! I had the same problem with J9 like you explained (e.g. java.lang.Object is in some crazy extension dir). I solved the problem in the exact same way like you did (checking the URLConnection instance type and use more exotic class names). The final solution I comitted was easier: RuntimeMXBean.getBootClassPath() and is supported on all recent JDKs (although its documented that it may be unsupported, they have a boolean getter to detect this). I build a Set from the classpath components (JAR files and directories). When loading the classes later, i lookup the JarFile's of the class that was loaded by the checker to find out if it is a RuntimeClass or not: https://code.google.com/p/forbidden-apis/source/browse/trunk/src/main/java/de/thetaphi/forbiddenapis/Checker.java#88
          Hide
          Dawid Weiss added a comment -

          I like the bootClassPath trick, didn't know about it.

          Show
          Dawid Weiss added a comment - I like the bootClassPath trick, didn't know about it.
          Hide
          Uwe Schindler added a comment -

          Steven: I released version 1.2 including the fix to correctly support test classes in Maven. It has also fixes the "unsupported JDK" issue with JDK 6 on Mäcintrash.
          It is already available on Sonatype Releases (https://oss.sonatype.org/content/repositories/releases/de/thetaphi/forbiddenapis/1.2/), on Maven Central hopefully after a few hours. The changelog is here: http://code.google.com/p/forbidden-apis/wiki/Changes

          Show
          Uwe Schindler added a comment - Steven: I released version 1.2 including the fix to correctly support test classes in Maven. It has also fixes the "unsupported JDK" issue with JDK 6 on Mäcintrash. It is already available on Sonatype Releases ( https://oss.sonatype.org/content/repositories/releases/de/thetaphi/forbiddenapis/1.2/ ), on Maven Central hopefully after a few hours. The changelog is here: http://code.google.com/p/forbidden-apis/wiki/Changes
          Hide
          Steve Rowe added a comment -

          1.2 is on Maven Central now: http://repo1.maven.org/maven2/de/thetaphi/forbiddenapis/1.2/

          I'll test running the build with my patch after I remove -SNAPSHOT from forbiddenapi version and remove the sonatype repository.

          Show
          Steve Rowe added a comment - 1.2 is on Maven Central now: http://repo1.maven.org/maven2/de/thetaphi/forbiddenapis/1.2/ I'll test running the build with my patch after I remove -SNAPSHOT from forbiddenapi version and remove the sonatype repository.
          Hide
          Steve Rowe added a comment -

          Patch for running ForbiddenAPI checker in the Maven build - updating to 1.2 release, and removing the Sonatype Maven repository.

          Committing shortly.

          Show
          Steve Rowe added a comment - Patch for running ForbiddenAPI checker in the Maven build - updating to 1.2 release, and removing the Sonatype Maven repository. Committing shortly.
          Hide
          Steve Rowe added a comment -

          I wondered how much time running ForbiddenAPI checker, executed multiple times in each of ~40 modules, adds to the Maven build, so I ran mvn -DskipTests install, and then mvn process-test-classes, both pre- and post-patch. This excludes performing compilation, since I didn't mvn clean inbetween.

          On my Macbook pro, with Oracle Java 1.7.0_13 and Maven 3.0.3, the best (fastest) of five runs of mvn process-test-classes for pre-patch: 20s. Best of five for post-patch: 22s. So it looks like this adds 2 seconds to build time, assuming a populated OS filesystem cache.

          If I use sudo purge to clear the OS filesystem cache before every run, pre-patch best of five runs still took 20s, but post-patch best of five took 26s. So with an empty OS filesystem cache, this adds 6 seconds to the build time.

          I think this slowdown is live-with-able.

          Committing now.

          Show
          Steve Rowe added a comment - I wondered how much time running ForbiddenAPI checker, executed multiple times in each of ~40 modules, adds to the Maven build, so I ran mvn -DskipTests install , and then mvn process-test-classes , both pre- and post-patch. This excludes performing compilation, since I didn't mvn clean inbetween. On my Macbook pro, with Oracle Java 1.7.0_13 and Maven 3.0.3, the best (fastest) of five runs of mvn process-test-classes for pre-patch: 20s. Best of five for post-patch: 22s. So it looks like this adds 2 seconds to build time, assuming a populated OS filesystem cache. If I use sudo purge to clear the OS filesystem cache before every run, pre-patch best of five runs still took 20s, but post-patch best of five took 26s. So with an empty OS filesystem cache, this adds 6 seconds to the build time. I think this slowdown is live-with-able. Committing now.
          Hide
          Commit Tag Bot added a comment -

          [trunk commit] Steven Rowe
          http://svn.apache.org/viewvc?view=revision&revision=1446991

          LUCENE-4570: Added ForbiddenAPI checks to the Maven build using the ForbiddenAPI Mojo; also removed redundant maven-surefire-plugin configurations in Solr modules' POMs after putting a shared configuration in the Solr parent POM.

          Show
          Commit Tag Bot added a comment - [trunk commit] Steven Rowe http://svn.apache.org/viewvc?view=revision&revision=1446991 LUCENE-4570 : Added ForbiddenAPI checks to the Maven build using the ForbiddenAPI Mojo; also removed redundant maven-surefire-plugin configurations in Solr modules' POMs after putting a shared configuration in the Solr parent POM.
          Hide
          Commit Tag Bot added a comment -

          [branch_4x commit] Steven Rowe
          http://svn.apache.org/viewvc?view=revision&revision=1446992

          LUCENE-4570: Added ForbiddenAPI checks to the Maven build using the ForbiddenAPI Mojo; also removed redundant maven-surefire-plugin configurations in Solr modules' POMs after putting a shared configuration in the Solr parent POM. (merged trunk r1446991)

          Show
          Commit Tag Bot added a comment - [branch_4x commit] Steven Rowe http://svn.apache.org/viewvc?view=revision&revision=1446992 LUCENE-4570 : Added ForbiddenAPI checks to the Maven build using the ForbiddenAPI Mojo; also removed redundant maven-surefire-plugin configurations in Solr modules' POMs after putting a shared configuration in the Solr parent POM. (merged trunk r1446991)
          Hide
          Uwe Schindler added a comment -

          I think this slowdown is live-with-able.

          By the way, there is already an issue open to make the tests in Lucene also per-module (LUCENE-4753). The current setup is far from manageable, especially Solr's global classptah that is still broken (resulting in relaxed checks).

          I ran a Maven build on Jenkins and inspected the log files, looks good. One thing to mention: In the (current) Lucene build, we run the forbidden checks in a first step for core and test classes at once (because we have no clear separation on the top-level, this will change wth LUCENE-4753). After that we run some additional checks afterwards. Because of of this global-then-specific behaviour, I am not sure if all checks in Maven are identical. I think the executors.txt should be run for both maven main and test classes. Currently the executors check is only run for main classes, not tests. Ant build runs it on both. Maybe add the executors next to tests.txt, too.

          Show
          Uwe Schindler added a comment - I think this slowdown is live-with-able. By the way, there is already an issue open to make the tests in Lucene also per-module ( LUCENE-4753 ). The current setup is far from manageable, especially Solr's global classptah that is still broken (resulting in relaxed checks). I ran a Maven build on Jenkins and inspected the log files, looks good. One thing to mention: In the (current) Lucene build, we run the forbidden checks in a first step for core and test classes at once (because we have no clear separation on the top-level, this will change wth LUCENE-4753 ). After that we run some additional checks afterwards. Because of of this global-then-specific behaviour, I am not sure if all checks in Maven are identical. I think the executors.txt should be run for both maven main and test classes. Currently the executors check is only run for main classes, not tests. Ant build runs it on both. Maybe add the executors next to tests.txt, too.
          Hide
          Steve Rowe added a comment -

          I think the executors.txt should be run for both maven main and test classes. Currently the executors check is only run for main classes, not tests. Ant build runs it on both. Maybe add the executors next to tests.txt, too.

          Ok, I'll do that. I was trying to make the Maven setup the same as the Ant setup, and missed this difference.

          Show
          Steve Rowe added a comment - I think the executors.txt should be run for both maven main and test classes. Currently the executors check is only run for main classes, not tests. Ant build runs it on both. Maybe add the executors next to tests.txt, too. Ok, I'll do that. I was trying to make the Maven setup the same as the Ant setup, and missed this difference.
          Hide
          Commit Tag Bot added a comment -

          [branch_4x commit] Steven Rowe
          http://svn.apache.org/viewvc?view=revision&revision=1447139

          LUCENE-4570: Maven ForbiddenAPIs configuration cleanups:

          • Clean up overly long execution IDs
          • Make at least one text execution per module include internalRuntimeForbidden=true
          • Make at least one text execution per module include signatureFile executors.txt
          • Include bundledSignature commons-io-unsafe in solr test-framework forbiddenapis check
          • Note in the Solr shared test-check configuration to include bundledSignature commons-io-unsafe only in modules with commons-io on their classpath
            (merged trunk r1447138)
          Show
          Commit Tag Bot added a comment - [branch_4x commit] Steven Rowe http://svn.apache.org/viewvc?view=revision&revision=1447139 LUCENE-4570 : Maven ForbiddenAPIs configuration cleanups: Clean up overly long execution IDs Make at least one text execution per module include internalRuntimeForbidden=true Make at least one text execution per module include signatureFile executors.txt Include bundledSignature commons-io-unsafe in solr test-framework forbiddenapis check Note in the Solr shared test-check configuration to include bundledSignature commons-io-unsafe only in modules with commons-io on their classpath (merged trunk r1447138)
          Hide
          Commit Tag Bot added a comment -

          [trunk commit] Steven Rowe
          http://svn.apache.org/viewvc?view=revision&revision=1447138

          LUCENE-4570: Maven ForbiddenAPIs configuration cleanups:

          • Clean up overly long execution IDs
          • Make at least one text execution per module include internalRuntimeForbidden=true
          • Make at least one text execution per module include signatureFile executors.txt
          • Include bundledSignature commons-io-unsafe in solr test-framework forbiddenapis check
          • Note in the Solr shared test-check configuration to include bundledSignature commons-io-unsafe only in modules with commons-io on their classpath
          Show
          Commit Tag Bot added a comment - [trunk commit] Steven Rowe http://svn.apache.org/viewvc?view=revision&revision=1447138 LUCENE-4570 : Maven ForbiddenAPIs configuration cleanups: Clean up overly long execution IDs Make at least one text execution per module include internalRuntimeForbidden=true Make at least one text execution per module include signatureFile executors.txt Include bundledSignature commons-io-unsafe in solr test-framework forbiddenapis check Note in the Solr shared test-check configuration to include bundledSignature commons-io-unsafe only in modules with commons-io on their classpath
          Hide
          Commit Tag Bot added a comment -

          [branch_4x commit] Uwe Schindler
          http://svn.apache.org/viewvc?view=revision&revision=1412599

          Merged revision(s) 1412598 from lucene/dev/trunk:
          LUCENE-4570: Add missing license headers

          Show
          Commit Tag Bot added a comment - [branch_4x commit] Uwe Schindler http://svn.apache.org/viewvc?view=revision&revision=1412599 Merged revision(s) 1412598 from lucene/dev/trunk: LUCENE-4570 : Add missing license headers
          Hide
          Uwe Schindler added a comment -

          Closed after release.

          Show
          Uwe Schindler added a comment - Closed after release.

            People

            • Assignee:
              Uwe Schindler
              Reporter:
              Robert Muir
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development