Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-7727

Replace EOL'ed pegdown by flexmark-java for Java 9 compatibility

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 7.0, 6.5
    • Component/s: general/build
    • Labels:
    • Lucene Fields:
      New

      Description

      The documentation tasks use a library called "pegdown" to convert Markdown to HTML. Unfortunately, the developer of pegdown EOLed it and points the users to a faster replacement: flexmark-java (https://github.com/vsch/flexmark-java).

      This would not be important for us, if pegdown would work with Java 9, but it is also affected by the usual "setAccessible into private Java APIs" issue (see my talk at FOSDEM: https://fosdem.org/2017/schedule/event/jigsaw_challenges/).

      The migration should not be too hard, its just a bit of Groovy Code rewriting and dependency changes.

      This is the pegdown problem:

      Caused by: java.lang.RuntimeException: Could not determine whether class 'org.pegdown.Parser$$parboiled' has already been loaded
              at org.parboiled.transform.AsmUtils.findLoadedClass(AsmUtils.java:213)
              at org.parboiled.transform.ParserTransformer.transformParser(ParserTransformer.java:35)
              at org.parboiled.Parboiled.createParser(Parboiled.java:54)
              ... 50 more
      Caused by: java.lang.reflect.InaccessibleObjectException: Unable to make protected final java.lang.Class java.lang.ClassLoader.findLoadedClass(java.lang.String) accessible: module java.base does not "opens java.lang" to unnamed module @551b6736
              at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:335)
              at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:278)
              at java.base/java.lang.reflect.Method.checkCanSetAccessible(Method.java:196)
              at java.base/java.lang.reflect.Method.setAccessible(Method.java:190)
              at org.parboiled.transform.AsmUtils.findLoadedClass(AsmUtils.java:206)
              ... 52 more
      
      1. LUCENE-7727.patch
        12 kB
        Uwe Schindler
      2. LUCENE-7727.patch
        10 kB
        Uwe Schindler

        Issue Links

          Activity

          Hide
          thetaphi Uwe Schindler added a comment -

          Rewrite to use flexmark.

          I changed the task names to use "markdown" and let the implementation be hidden (which is flexmark after this patch).

          There is some room for improvements by not using the PegDown Adapter (this removes many dependencies). I will look into this tomorrow. For now build works with Java 9 and the HTML output matches.

          Show
          thetaphi Uwe Schindler added a comment - Rewrite to use flexmark. I changed the task names to use "markdown" and let the implementation be hidden (which is flexmark after this patch). There is some room for improvements by not using the PegDown Adapter (this removes many dependencies). I will look into this tomorrow. For now build works with Java 9 and the HTML output matches.
          Hide
          thetaphi Uwe Schindler added a comment - - edited

          New patch with cleanup:

          • remove of Pegdown compatibility layer (minimal dependencies)
          • auto-generate IDs (remove <a name>, so I also removed the regex for website publishing
          • extract HTML title from AST). I verified the autogen'd IDs are identical.

          I think that's ready. I will commit it later this afternoon.

          Show
          thetaphi Uwe Schindler added a comment - - edited New patch with cleanup: remove of Pegdown compatibility layer (minimal dependencies) auto-generate IDs (remove <a name> , so I also removed the regex for website publishing extract HTML title from AST). I verified the autogen'd IDs are identical. I think that's ready. I will commit it later this afternoon.
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 707d7b91e8793b4bb017e132c8a206acf85885ab in lucene-solr's branch refs/heads/master from Uwe Schindler
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=707d7b9 ]

          LUCENE-7727: Replace end-of-life Markdown parser "Pegdown" by "Flexmark" for compatibility with Java 9

          Show
          jira-bot ASF subversion and git services added a comment - Commit 707d7b91e8793b4bb017e132c8a206acf85885ab in lucene-solr's branch refs/heads/master from Uwe Schindler [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=707d7b9 ] LUCENE-7727 : Replace end-of-life Markdown parser "Pegdown" by "Flexmark" for compatibility with Java 9
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 48f8471fea823abd1a5c62ac21048a86af11de8b in lucene-solr's branch refs/heads/branch_6x from Uwe Schindler
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=48f8471 ]

          LUCENE-7727: Replace end-of-life Markdown parser "Pegdown" by "Flexmark" for compatibility with Java 9

          Show
          jira-bot ASF subversion and git services added a comment - Commit 48f8471fea823abd1a5c62ac21048a86af11de8b in lucene-solr's branch refs/heads/branch_6x from Uwe Schindler [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=48f8471 ] LUCENE-7727 : Replace end-of-life Markdown parser "Pegdown" by "Flexmark" for compatibility with Java 9
          Hide
          dsmiley David Smiley added a comment -

          Uwe, when I try to run ant precommit, I now see this error:

          resolve-markdown:
          
          BUILD FAILED
          /SmileyDev/Search/lucene-solr/lucene/common-build.xml:2415: ivy:cachepath doesn't support the nested "dependency" element.
          

          I'm wondering if you know what the cause of that may be. This is happening on one of my machines consistently, but not at all on another.

          Show
          dsmiley David Smiley added a comment - Uwe, when I try to run ant precommit , I now see this error: resolve-markdown: BUILD FAILED /SmileyDev/Search/lucene-solr/lucene/common-build.xml:2415: ivy:cachepath doesn't support the nested "dependency" element. I'm wondering if you know what the cause of that may be. This is happening on one of my machines consistently, but not at all on another.
          Hide
          thetaphi Uwe Schindler added a comment -

          Outdated Ivy? We require 2.3 minimum. Maybe clean up your ~/.ant/lib folder and run "ant ivy-bootstrap"?

          Show
          thetaphi Uwe Schindler added a comment - Outdated Ivy? We require 2.3 minimum. Maybe clean up your ~/.ant/lib folder and run "ant ivy-bootstrap"?
          Hide
          thetaphi Uwe Schindler added a comment -

          See http://ant.apache.org/ivy/history/2.3.0/use/postresolvetask.html:

          Child elements
          (Since 2.3)

          These child elements are defining an inlined ivy.xml's dependencies elements. (...)

          For flexmark we actually require multiple dependencies and this can only be handles with a single cachepath in that version. An alternative would be to have a ivy.xml file, but that's complicated for common-build.xml.

          So there is only the way to update to Ivy 2.3 as documented since years.

          Show
          thetaphi Uwe Schindler added a comment - See http://ant.apache.org/ivy/history/2.3.0/use/postresolvetask.html: Child elements (Since 2.3) These child elements are defining an inlined ivy.xml's dependencies elements. (...) For flexmark we actually require multiple dependencies and this can only be handles with a single cachepath in that version. An alternative would be to have a ivy.xml file, but that's complicated for common-build.xml. So there is only the way to update to Ivy 2.3 as documented since years.
          Hide
          dsmiley David Smiley added a comment -

          Thanks for the tips Uwe. I forgot to mention I did run ant ivy-bootstrap. So the problem was that my ~/.ant/lib/ had both ivy-2.2.0.jar and ivy-2.3.0.jar somehow for who knows how long, and this is biting me now. Perhaps Java 9 might better alert users to classpaths containing jars with classes in the same package? Something to look forward to if so.

          Show
          dsmiley David Smiley added a comment - Thanks for the tips Uwe. I forgot to mention I did run ant ivy-bootstrap . So the problem was that my ~/.ant/lib/ had both ivy-2.2.0.jar and ivy-2.3.0.jar somehow for who knows how long, and this is biting me now. Perhaps Java 9 might better alert users to classpaths containing jars with classes in the same package? Something to look forward to if so.
          Hide
          thetaphi Uwe Schindler added a comment -

          In my other projects using Ivy I have a version check on bootup of ant. Maybe we should do this, too, and print useful error message. The problem is that ANT does not have a version comparator (it only has <antversion>). So I print a warning in my projects if version is different that the expected.

          Java 9 won't bring changes to classpath. With Jigsaw modules all is better, as every modular jar has to define which packages it "exports". But for that we have to make modules out of Lucene and all dependencies (auto-modules may help). But this would also require to refactor Lucene and Solr to never use and export the same package name in different JARs.

          Show
          thetaphi Uwe Schindler added a comment - In my other projects using Ivy I have a version check on bootup of ant. Maybe we should do this, too, and print useful error message. The problem is that ANT does not have a version comparator (it only has <antversion> ). So I print a warning in my projects if version is different that the expected. Java 9 won't bring changes to classpath. With Jigsaw modules all is better, as every modular jar has to define which packages it "exports". But for that we have to make modules out of Lucene and all dependencies (auto-modules may help). But this would also require to refactor Lucene and Solr to never use and export the same package name in different JARs.

            People

            • Assignee:
              thetaphi Uwe Schindler
              Reporter:
              thetaphi Uwe Schindler
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development