Lucene - Core
  1. Lucene - Core
  2. LUCENE-3337

avoid building jar files unless necessary in build

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 3.4, 4.0-ALPHA
    • Fix Version/s: 3.4, 4.0-ALPHA
    • Component/s: general/build
    • Labels:
      None
    • Lucene Fields:
      New, Patch Available

      Description

      This causes the build to be slow, we can avoid it in lots of cases (e.g. ant test)

      1. LUCENE-3337.patch
        53 kB
        Steve Rowe
      2. LUCENE-3337.patch
        20 kB
        Steve Rowe
      3. LUCENE-3337.patch
        10 kB
        Robert Muir
      4. LUCENE-3337.patch
        6 kB
        Robert Muir
      5. LUCENE-3337.patch
        3 kB
        Robert Muir

        Activity

        Hide
        Robert Muir added a comment -

        here's the start to a patch, i started with solr's core/

        Show
        Robert Muir added a comment - here's the start to a patch, i started with solr's core/
        Hide
        Robert Muir added a comment -

        updated state, working on more of the solr deps to try to improve the situation... there is still a lot more to do here though, but it seems to improve a bit, at least we are only building jar files once (but in many cases like 'test' we never need to do this in the first place)

        task trunk patch
        ant example (clean checkout) 30s 20s
        ant example (already compiled) 18s 10s
        Show
        Robert Muir added a comment - updated state, working on more of the solr deps to try to improve the situation... there is still a lot more to do here though, but it seems to improve a bit, at least we are only building jar files once (but in many cases like 'test' we never need to do this in the first place) task trunk patch ant example (clean checkout) 30s 20s ant example (already compiled) 18s 10s
        Hide
        Robert Muir added a comment -

        some other minor optimizations, a few seconds here or there.

        Show
        Robert Muir added a comment - some other minor optimizations, a few seconds here or there.
        Hide
        Steve Rowe added a comment -

        Robert, your patch is a collection of optimizations to the SOLR-2452 work. Looks good!

        My build times:

        task trunk patch
        ant example (clean checkout) 180s 119s
        ant example (already compiled) 132s 61s

        So I'm seeing the same kind of speedups as you. +1 to commit - we could use this issue to host a series of further patches.

        I plan on looking next at the utility of the *.compiled & *.uptodate property passing scheme - you mentioned on #lucene IRC that it didn't seem to be having any effect on the build.

        Show
        Steve Rowe added a comment - Robert, your patch is a collection of optimizations to the SOLR-2452 work. Looks good! My build times: task trunk patch ant example (clean checkout) 180s 119s ant example (already compiled) 132s 61s So I'm seeing the same kind of speedups as you. +1 to commit - we could use this issue to host a series of further patches. I plan on looking next at the utility of the *.compiled & *.uptodate property passing scheme - you mentioned on #lucene IRC that it didn't seem to be having any effect on the build.
        Hide
        Steve Rowe added a comment -

        I plan on looking next at the utility of the *.compiled & *.uptodate property passing scheme - you mentioned on #lucene IRC that it didn't seem to be having any effect on the build.

        The properties were being passed in as intended, but the <uptodate> tasks were still running, since their execution was not dependent on the already-defined values of *.uptodate output properties.

        The attached version of the patch, which includes Robert's latest patch, wraps the <uptodate> tasks in check-*-uptodate targets, and makes the compile-* targets depend on them, so that the <uptodate> tasks only run when the corresponding *.update property value has not previously been set.

        The only problem with this approach is that the *.jar properties aren't constructed properly, since the property passing scheme only works one-way (properties are passed into <ant> invocations, but not back out of them). I worked around this problem by defining the *.jar properties statically.

        My build times now:

        task trunk patch patch w/property passing optimizations
        ant example (clean checkout) 180s 119s 93s
        ant example (already compiled) 132s 61s 55s

        This shaves a few more seconds off my build time.

        Show
        Steve Rowe added a comment - I plan on looking next at the utility of the *.compiled & *.uptodate property passing scheme - you mentioned on #lucene IRC that it didn't seem to be having any effect on the build. The properties were being passed in as intended, but the <uptodate> tasks were still running, since their execution was not dependent on the already-defined values of *.uptodate output properties. The attached version of the patch, which includes Robert's latest patch, wraps the <uptodate> tasks in check-*-uptodate targets, and makes the compile-* targets depend on them, so that the <uptodate> tasks only run when the corresponding *.update property value has not previously been set. The only problem with this approach is that the *.jar properties aren't constructed properly, since the property passing scheme only works one-way (properties are passed into <ant> invocations, but not back out of them). I worked around this problem by defining the *.jar properties statically. My build times now: task trunk patch patch w/property passing optimizations ant example (clean checkout) 180s 119s 93s ant example (already compiled) 132s 61s 55s This shaves a few more seconds off my build time.
        Hide
        Steve Rowe added a comment -

        When I look at the output from ant -v example after applying my version of the patch, I can see that the compile-test target is called 16 times from lucene/build.xml. Seems like it could be sped up through more/better property passing. And why should building a module's jar trigger compilation of Lucene's tests?

        Show
        Steve Rowe added a comment - When I look at the output from ant -v example after applying my version of the patch, I can see that the compile-test target is called 16 times from lucene/build.xml . Seems like it could be sped up through more/better property passing. And why should building a module's jar trigger compilation of Lucene's tests?
        Hide
        Steve Rowe added a comment -

        This version of the patch moves uptodate checks and targets to build jars for other modules to lucene/contrib/contrib-build.xml. All Solr & Lucene modules are now covered. The Lucene core build now has a jar-core specialization that avoids creating the jar when it's already up-to-date - previously when running ant test from solr/, the Lucene core jar was built 41 times, because the manifest was newer each time...

        My build times now:

        task trunk Robert's patch My patch w/property passing optimizations
        ant example (clean checkout) 180s 119s 52s
        ant example (already compiled) 132s 61s 27s

        I think this should be committed; further changes can be made with more patches.

        Show
        Steve Rowe added a comment - This version of the patch moves uptodate checks and targets to build jars for other modules to lucene/contrib/contrib-build.xml . All Solr & Lucene modules are now covered. The Lucene core build now has a jar-core specialization that avoids creating the jar when it's already up-to-date - previously when running ant test from solr/ , the Lucene core jar was built 41 times, because the manifest was newer each time... My build times now: task trunk Robert's patch My patch w/property passing optimizations ant example (clean checkout) 180s 119s 52s ant example (already compiled) 132s 61s 27s I think this should be committed; further changes can be made with more patches.
        Hide
        Robert Muir added a comment -

        whoah, those timings look great!

        Show
        Robert Muir added a comment - whoah, those timings look great!
        Hide
        Robert Muir added a comment -

        I think this should be committed; further changes can be made with more patches.

        +1 to commit!

        Show
        Robert Muir added a comment - I think this should be committed; further changes can be made with more patches. +1 to commit!
        Hide
        Steve Rowe added a comment -

        Committed:

        • r1151720: trunk
        • r1151735: branch_3x
        Show
        Steve Rowe added a comment - Committed: r1151720: trunk r1151735: branch_3x
        Hide
        Yonik Seeley added a comment -

        Sweet, "ant example" the second time now comes in at 16sec for me!
        This helps big time doing ad-hoc testing/debugging using the server.

        Show
        Yonik Seeley added a comment - Sweet, "ant example" the second time now comes in at 16sec for me! This helps big time doing ad-hoc testing/debugging using the server.
        Hide
        Steve Rowe added a comment -

        On SOLR-2452, yonik wrote:

        On trunk in solr, "ant example" after touching a single source file was 31 seconds before this patch and is now 51.

        And then on this issue:

        Sweet, "ant example" the second time now comes in at 16sec for me!

        Cool, twice as fast as it was pre-SOLR-2452!

        Show
        Steve Rowe added a comment - On SOLR-2452 , yonik wrote: On trunk in solr, "ant example" after touching a single source file was 31 seconds before this patch and is now 51. And then on this issue: Sweet, "ant example" the second time now comes in at 16sec for me! Cool, twice as fast as it was pre- SOLR-2452 !
        Hide
        Steve Rowe added a comment -

        Closing this issue - further work can be done in new issues.

        Show
        Steve Rowe added a comment - Closing this issue - further work can be done in new issues.

          People

          • Assignee:
            Steve Rowe
            Reporter:
            Robert Muir
          • Votes:
            1 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development