Lucene - Core
  1. Lucene - Core
  2. LUCENE-1083

JDiff report of changes between different versions of Lucene

    Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 2.2
    • Fix Version/s: None
    • Component/s: general/javadocs
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      I think that a helpful addition to the release process for Lucene would be JDiff reports of the API changes between different versions. I am attaching reports of the differences between 1.9.1 and 2.2.0 and also between 2.1.0 and 2.2.0. The reports could be changed to only show the public methods. The start page is changes.html.

      This is the Ant target I added to the top-level build.xml file in the JDiff directory to produce a report:

      <target name="lucene" depends="dist">
        <taskdef name="jdiff" 
                 classname="jdiff.JDiffAntTask" 
                 classpath="${dist.dir}/antjdiff.jar" />
        <jdiff destdir="${reports.dir}/lucene" 
               verbose="on"
               stats="on"
               docchanges="on">
          <old name="1.9.1">
            <dirset dir="${examples.dir}/lucene-1.9.1/src/java" includes="org/**" />
          </old>
          <new name="2.2.0">
            <dirset dir="${examples.dir}/lucene-2.2.0/src/java" includes="org/**" />
          </new>
        </jdiff>
      </target>
      

      Disclaimer: I'm the author of JDiff

      1. 1083_2.patch
        7 kB
        Matt Doar
      2. 1083_1.patch
        7 kB
        Matt Doar
      3. core-jdiff.zip
        251 kB
        Matt Doar
      4. jdiff_lucene_210_220.zip
        2.94 MB
        Matt Doar
      5. jdiff_lucene_191_220.zip
        2.93 MB
        Matt Doar

        Activity

        Hide
        Robert Muir added a comment -

        I think this is pretty relevant.

        Its a compelling alternative to backwards compatibility tests that run with a previous version of lucene (which are costly to maintain)

        Show
        Robert Muir added a comment - I think this is pretty relevant. Its a compelling alternative to backwards compatibility tests that run with a previous version of lucene (which are costly to maintain)
        Hide
        Jan Høydahl added a comment -

        This issue has been inactive for more than 4 years. Please close if it's no longer relevant/needed, or bring it up to date if you intend to work on it. SPRING_CLEANING_2013

        Show
        Jan Høydahl added a comment - This issue has been inactive for more than 4 years. Please close if it's no longer relevant/needed, or bring it up to date if you intend to work on it. SPRING_CLEANING_2013
        Hide
        Matt Doar added a comment -

        This patch adds Doug's suggestion to use exec for svn. There is no need to update a source tree based on a tag, but I have added the target in case someone ever wanted to. If the checkout time is too long for developers, we could also make the XML file that represents a release available, and then use that, instead of regenerating it each time.

        Show
        Matt Doar added a comment - This patch adds Doug's suggestion to use exec for svn. There is no need to update a source tree based on a tag, but I have added the target in case someone ever wanted to. If the checkout time is too long for developers, we could also make the XML file that represents a release available, and then use that, instead of regenerating it each time.
        Hide
        Matt Doar added a comment -

        Ok, exec it will be, and I'll remove the jdiff.home line. The funny thing is that the "3 separate numbers" idea is Erik's suggestion, circa 2004 (http://www.ant-tasks.com/msg/18617.html)

        Show
        Matt Doar added a comment - Ok, exec it will be, and I'll remove the jdiff.home line. The funny thing is that the "3 separate numbers" idea is Erik's suggestion, circa 2004 ( http://www.ant-tasks.com/msg/18617.html )
        Hide
        Doug Cutting added a comment -

        Thanks for updating this!

        The svn task is not in ant core yet.

        Then we should use the 'exec' task until it's in broad distribution.

        + <property name="jdiff.home" location="D:/mdoar/jdiff/build/jdiff-1.1.0"/>

        That line should not exist. Folks will need to define jdiff.home outside of this build file. That includes you. Typically it will be done on the command line, with -Djdiff.home=...

        I had to define the version number as 3 separate numbers because Ant doesn't make string subst easy on properties.

        Erik, oh, great Ant wizard, any thoughts?

        Show
        Doug Cutting added a comment - Thanks for updating this! The svn task is not in ant core yet. Then we should use the 'exec' task until it's in broad distribution. + <property name="jdiff.home" location="D:/mdoar/jdiff/build/jdiff-1.1.0"/> That line should not exist. Folks will need to define jdiff.home outside of this build file. That includes you. Typically it will be done on the command line, with -Djdiff.home=... I had to define the version number as 3 separate numbers because Ant doesn't make string subst easy on properties. Erik, oh, great Ant wizard, any thoughts?
        Hide
        Matt Doar added a comment - - edited

        The attached 1083_1.patch file against r604038 does most of what you suggested. The svn task is not in ant core yet, so I couldn't use it to check out the source of the previous version. I had to define the version number as 3 separate numbers because Ant doesn't make string subst easy on properties. All the links in the JDiff reports should work, including the links to the previous and current Javadocs.

        Show
        Matt Doar added a comment - - edited The attached 1083_1.patch file against r604038 does most of what you suggested. The svn task is not in ant core yet, so I couldn't use it to check out the source of the previous version. I had to define the version number as 3 separate numbers because Ant doesn't make string subst easy on properties. All the links in the JDiff reports should work, including the links to the previous and current Javadocs.
        Hide
        Matt Doar added a comment -

        The patch as requested by Doug.

        Show
        Matt Doar added a comment - The patch as requested by Doug.
        Hide
        Doug Cutting added a comment -

        The changes to build.xml should be provided as a patch file. For details, see:

        http://wiki.apache.org/lucene-java/HowToContribute

        JDIFF_HOME and jdiff.home should be undefined by default, and the target should do nothing when these are not defined, except perhaps print a warning.

        The previous version should be defined as a subversion URL that is checked out somewhere in build/ if it doesn't already exist there, and updated otherwise. Typically the url would be to a tag. We might do something like:

        jdiff.prev.version="2.2.0"
        jdiff.prev.url.base="http://svn.apache.org/repos/asf/lucene/java/tags/release-"
        jdiff.prev.url="$

        {prev.url.base}

        $

        {jdiff.prev.version}"
        jdiff.prev.dir="${build.dir}/jdiff/prev-${jdiff.prev.version}

        "

        Does that sound reasonable?

        Show
        Doug Cutting added a comment - The changes to build.xml should be provided as a patch file. For details, see: http://wiki.apache.org/lucene-java/HowToContribute JDIFF_HOME and jdiff.home should be undefined by default, and the target should do nothing when these are not defined, except perhaps print a warning. The previous version should be defined as a subversion URL that is checked out somewhere in build/ if it doesn't already exist there, and updated otherwise. Typically the url would be to a tag. We might do something like: jdiff.prev.version="2.2.0" jdiff.prev.url.base="http://svn.apache.org/repos/asf/lucene/java/tags/release-" jdiff.prev.url="$ {prev.url.base} $ {jdiff.prev.version}" jdiff.prev.dir="${build.dir}/jdiff/prev-${jdiff.prev.version} " Does that sound reasonable?
        Hide
        Matt Doar added a comment -

        The differences between the API of the core classes of 2.2.0 and 2.3.0-dev as of revision 603799

        Show
        Matt Doar added a comment - The differences between the API of the core classes of 2.2.0 and 2.3.0-dev as of revision 603799
        Hide
        Matt Doar added a comment -

        Below is an Ant target for the core classes that works in r603799 and supports restricting the report to just protected and public methods (the jdiff ant task doesn't support that). I'll also attach the report of the differences between 2.2.0 and 2.3.0-dev that was generated with this task. The ant target still needs some work to specify where jdiff is located, and to convert the periods in the prev.version string to underscores for the URL to the previous version javadocs.

          <target name="jdiff-core" description="Generate jdiff for core classes">
          	<sequential>
              <property name="JDIFF_HOME" value="D:/mdoar/jdiff/build/jdiff-1.1.0" />
              <property name="jdiff.home" value="${JDIFF_HOME}" />
              <property name="jdiff.reportdir" value="${javadoc.dir}/core-jdiff" />
              <property name="jdiff.tmpdir" value="${javadoc.dir}/jdiff.tmp" />
              <property name="prev.version" value="2.2.0" />
              <property name="prev.src.dir" value="D:/mdoar/jdiff/examples/lucene-${prev.version}" />
        
              <mkdir dir="${jdiff.reportdir}"/>
              <mkdir dir="${jdiff.tmpdir}"/>
        
              <echo message="Generate the XML representing the previous API" />
              <javadoc access="${javadoc.access}"
                       encoding="${build.encoding}">
                  <doclet name="jdiff.JDiff"
                          path="${jdiff.home}/jdiff.jar:${jdiff.home}/xerces.jar">
                    <param name="-apidir" value="${jdiff.tmpdir}" />
                    <param name="-excludeclass" value="${javadoc.access}" />
                    <param name="-excludemember" value="${javadoc.access}" />
                    <param name="-apiname" value="${Name} ${prev.version} API" />
                  </doclet>
                <packageset dir="${prev.src.dir}/src/java"/>
                <classpath refid="javadoc.classpath"/>
              </javadoc>
        
              <echo message="Generate the XML representing the current API" />
              <javadoc access="${javadoc.access}"
                       encoding="${build.encoding}">
                  <doclet name="jdiff.JDiff"
                          path="${jdiff.home}/jdiff.jar:${jdiff.home}/xerces.jar">
                    <param name="-apidir" value="${jdiff.tmpdir}" />
                    <param name="-excludeclass" value="${javadoc.access}" />
                    <param name="-excludemember" value="${javadoc.access}" />
                    <param name="-apiname" value="${Name} ${version} API" />
                  </doclet>
                <packageset dir="src/java"/>
                <classpath refid="javadoc.classpath"/>
              </javadoc>
        
              <echo message="Generate the JDiff report comparing the APIs" />
              <javadoc
                  private="yes"
                  sourcepath="src/java"
                  destdir="${jdiff.reportdir}"
                  sourcefiles="${jdiff.home}/Null.java">
                  <doclet name="jdiff.JDiff"
                          path="${jdiff.home}/jdiff.jar:${jdiff.home}/xerces.jar">
                    <param name="-oldapi" value="${Name} ${prev.version} API" />
                    <param name="-newapi" value="${Name} ${version} API" />
                    <param name="-oldapidir" value="${jdiff.tmpdir}" />
                    <param name="-newapidir" value="${jdiff.tmpdir}" />
                    <param name="-javadocnew" value="../../core/" />
                    <!-- TODO change . to _ in prev.version-->
                    <param name="-javadocold" value="http://lucene.apache.org/java/${prev.version}/api/" /> 
                    <param name="-docchanges"/>
                    <param name="-stats"/>
                  </doclet>
                <classpath refid="javadoc.classpath"/>
              </javadoc>
        
            <!-- Copy image files. black.gif is only needed if -stats was used -->
            <copy file="${jdiff.home}/background.gif" todir="${jdiff.reportdir}" />
            <copy file="${jdiff.home}/black.gif" todir="${jdiff.reportdir}" />
            </sequential>
          </target>
        
        Show
        Matt Doar added a comment - Below is an Ant target for the core classes that works in r603799 and supports restricting the report to just protected and public methods (the jdiff ant task doesn't support that). I'll also attach the report of the differences between 2.2.0 and 2.3.0-dev that was generated with this task. The ant target still needs some work to specify where jdiff is located, and to convert the periods in the prev.version string to underscores for the URL to the previous version javadocs. <target name="jdiff-core" description="Generate jdiff for core classes"> <sequential> <property name="JDIFF_HOME" value="D:/mdoar/jdiff/build/jdiff-1.1.0" /> <property name="jdiff.home" value="${JDIFF_HOME}" /> <property name="jdiff.reportdir" value="${javadoc.dir}/core-jdiff" /> <property name="jdiff.tmpdir" value="${javadoc.dir}/jdiff.tmp" /> <property name="prev.version" value="2.2.0" /> <property name="prev.src.dir" value="D:/mdoar/jdiff/examples/lucene-${prev.version}" /> <mkdir dir="${jdiff.reportdir}"/> <mkdir dir="${jdiff.tmpdir}"/> <echo message="Generate the XML representing the previous API" /> <javadoc access="${javadoc.access}" encoding="${build.encoding}"> <doclet name="jdiff.JDiff" path="${jdiff.home}/jdiff.jar:${jdiff.home}/xerces.jar"> <param name="-apidir" value="${jdiff.tmpdir}" /> <param name="-excludeclass" value="${javadoc.access}" /> <param name="-excludemember" value="${javadoc.access}" /> <param name="-apiname" value="${Name} ${prev.version} API" /> </doclet> <packageset dir="${prev.src.dir}/src/java"/> <classpath refid="javadoc.classpath"/> </javadoc> <echo message="Generate the XML representing the current API" /> <javadoc access="${javadoc.access}" encoding="${build.encoding}"> <doclet name="jdiff.JDiff" path="${jdiff.home}/jdiff.jar:${jdiff.home}/xerces.jar"> <param name="-apidir" value="${jdiff.tmpdir}" /> <param name="-excludeclass" value="${javadoc.access}" /> <param name="-excludemember" value="${javadoc.access}" /> <param name="-apiname" value="${Name} ${version} API" /> </doclet> <packageset dir="src/java"/> <classpath refid="javadoc.classpath"/> </javadoc> <echo message="Generate the JDiff report comparing the APIs" /> <javadoc private="yes" sourcepath="src/java" destdir="${jdiff.reportdir}" sourcefiles="${jdiff.home}/Null.java"> <doclet name="jdiff.JDiff" path="${jdiff.home}/jdiff.jar:${jdiff.home}/xerces.jar"> <param name="-oldapi" value="${Name} ${prev.version} API" /> <param name="-newapi" value="${Name} ${version} API" /> <param name="-oldapidir" value="${jdiff.tmpdir}" /> <param name="-newapidir" value="${jdiff.tmpdir}" /> <param name="-javadocnew" value="../../core/" /> <!-- TODO change . to _ in prev.version--> <param name="-javadocold" value="http://lucene.apache.org/java/${prev.version}/api/" /> <param name="-docchanges"/> <param name="-stats"/> </doclet> <classpath refid="javadoc.classpath"/> </javadoc> <!-- Copy image files. black.gif is only needed if -stats was used --> <copy file="${jdiff.home}/background.gif" todir="${jdiff.reportdir}" /> <copy file="${jdiff.home}/black.gif" todir="${jdiff.reportdir}" /> </sequential> </target>
        Hide
        Matt Doar added a comment -

        As an aside, Maven repositories in general could usefully be enhanced to
        record this kind of information for a project, so that you could query them
        for "current release", "prior patch release", "prior minor release", "prior
        major release". For Lucene at 2.2.0, that would give 2.2.0,
        2.1.x(non-existent, so detaults to
        2.1.0?), 2.1.0 and 1.9.1. And then allow users to override what each value
        is for a specific release of a project.

        Show
        Matt Doar added a comment - As an aside, Maven repositories in general could usefully be enhanced to record this kind of information for a project, so that you could query them for "current release", "prior patch release", "prior minor release", "prior major release". For Lucene at 2.2.0, that would give 2.2.0, 2.1.x(non-existent, so detaults to 2.1.0?), 2.1.0 and 1.9.1. And then allow users to override what each value is for a specific release of a project.
        Hide
        Doug Cutting added a comment -

        The "prior release" is a new concept that needs to be added to the build to support this. Perhaps a property in common-build.xml that names the subversion tags of the prior major and minor releases, and the jdiff target could use these. The jdiff target should do nothing if jdiff.home is not defined. Some links to the jdiff output should be added somewhere in the docs. Folks who build releases would then be required to install jdiff and to define jdiff.home when they make releases, or else the releases will contain broken links. This should be documented on the wiki's HowToRelease page.

        Show
        Doug Cutting added a comment - The "prior release" is a new concept that needs to be added to the build to support this. Perhaps a property in common-build.xml that names the subversion tags of the prior major and minor releases, and the jdiff target could use these. The jdiff target should do nothing if jdiff.home is not defined. Some links to the jdiff output should be added somewhere in the docs. Folks who build releases would then be required to install jdiff and to define jdiff.home when they make releases, or else the releases will contain broken links. This should be documented on the wiki's HowToRelease page.
        Hide
        Matt Doar added a comment -

        Grant,

        I was imagining more that the release process for Lucene could be changed so that whoever is creating a release also runs JDiff to produce the HTML reports, which they then post on the website with the usual Javadocs, and possibly include the report in the released package, say in a directory named docs/changes next to where docs/api currently is.

        The Maven javadoc task looks like it should be able to create the reports. I guess I should see if I can come up with a working example of that.

        Show
        Matt Doar added a comment - Grant, I was imagining more that the release process for Lucene could be changed so that whoever is creating a release also runs JDiff to produce the HTML reports, which they then post on the website with the usual Javadocs, and possibly include the report in the released package, say in a directory named docs/changes next to where docs/api currently is. The Maven javadoc task looks like it should be able to create the reports. I guess I should see if I can come up with a working example of that.
        Hide
        Grant Ingersoll added a comment -

        Thanks, Matt. I assume the antjdiff.jar needs to be included somewhere? In order for this to work, you probably need to make it so it checks to see if that library exists before running it (check out the way the Clover test coverage works) and we can't include the actual JDiff libraries in Lucene b/c of licensing issues (I'm pretty sure, anyway), so ideally, your task would also download the library when executed and install it properly for users.

        Show
        Grant Ingersoll added a comment - Thanks, Matt. I assume the antjdiff.jar needs to be included somewhere? In order for this to work, you probably need to make it so it checks to see if that library exists before running it (check out the way the Clover test coverage works) and we can't include the actual JDiff libraries in Lucene b/c of licensing issues (I'm pretty sure, anyway), so ideally, your task would also download the library when executed and install it properly for users.

          People

          • Assignee:
            Unassigned
            Reporter:
            Matt Doar
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:

              Development