Uploaded image for project: 'Derby'
  1. Derby
  2. DERBY-7052

Reordering and (mildly) reorganizing build.xml Ant targets around buildsource

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 10.15.1.3
    • Fix Version/s: 10.15.2.0, 10.16.0.0
    • Component/s: Build tools
    • Labels:
      None

      Description

      Module dependencies should be reflected by Ant targets dependencies.
      And Ant targets execution follows the rules stated in
          https://ant.apache.org/manual/targets.html
      so a dependency could be covered (being redundant) by applying transitivity rule
      to every dependency's dependecies (please look at the dot+pdf diagrams).
      --------------------------------------------------------------------------------
      Modules dependencies, by the docs, are :

      module depends on
      org.apache.derby.client org.apache.derby.commons
      org.apache.derby.engine org.apache.derby.commons
      org.apache.derby.locale_* org.apache.derby.commons
         
      org.apache.derby.tools org.apache.derby.client
      org.apache.derby.tools org.apache.derby.engine
         
      org.apache.derby.optionaltools org.apache.derby.tools
         
      org.apache.derby.server org.apache.derby.tools
         
      org.apache.derby.runner org.apache.derby.server
         
      org.apache.derby.engine org.osgi.framework

      but actual ant dependencies are (partial target => module map) :

       

      target depends and module depends on
      buildsource client    
      buildsource shared    
      buildsource felixStubs   org.osgi.framework
      engine shared org.apache.derby.engine org.apache.derby.commons
      client shared org.apache.derby.client org.apache.derby.commons
      tools engine org.apache.derby.tools org.apache.derby.engine
      drda engine    
      runner optional    
      optional engine    

      so we expect that building 'engine' target alone will fails due to the lack of
      'felixstubs' dependency.
      And it fails ("01 ant engine.log") quoting "requires static org.osgi.framework;"

      So the best try is to delete (on copy "build-01.xml" of "build.xml") the
      dependency
        buildsource => felixstubs
      and to put it as last dependency of 'engine' target
      Result : success ("02 ant -f build-01.xml engine.txt")

      Similarly we can expect that also 'tools' fails, since it really depends on
      'client' but this is not explicited in "build.xml".
      And it fails ("03 ant -f build-01.xml tools.log") with:
          [javac]     requires static org.apache.derby.client;

      So the best try is to switch the 'client' dependency (in copy "build-02.xml" of
      "build-01.xml") from 'buildsource' to 'tools'.

      Result: success ("04 ant -f build-02.xml tools.log").

      The same thing for 'optional' that should depends on 'tools' but depends only on
      'engine'.
      And it fails ("05 ant -f build-02.xml optional.log") with
          [javac]     requires org.apache.derby.tools;
      So we switch the 'tools' dependency from 'buildsource' to 'optional' (in copy
      "build-03.xml" of "build.02.xml")
      And it succeeds ("06 ant -f build-03.xml optional.log").

      Same as 'drda' (aka server) that require 'tools' ("07 ant -f build-03.xml drda.log")
          [javac]     requires org.apache.derby.tools;
      that succeds after adding 'tools' dependency to 'drda'
      ("08 ant -f build-04.xml drda.log")
      and also for 'runner' ("09 ant -f build-04.xml runner.log") that requires 'drda'
          [javac]     requires org.apache.derby.server;
      and succeeds after adding it into copy "build-05.xml" of "build-04.xml".
      ("10 ant -f build-05.xml runner.log")

      After graphing "build-05.xml" (in "build-05.xml.pdf") one can notice that,
      apart from redundant prerequisites (covered later), there's an extra dependency
      on 'runner', that in module hierararchy needs only 'drda'.
      After duplicating "build-05.xml" in "build-06.xml" and edited, removing
      'runner' -> 'optional' prerequisite, build of 'runner' succeeds
      ("11 ant -f build-06.xml runner.log")

      The next step is removing all the extra dependencies.
      Given the transitivity of dependencies in this set
      A => C (a depends on C)
      A => B (a depends on B)
      B => C (B depends on C)
      the first dependendency is already covered by second and third one, and ant
      will ignore it.

      After graphing "build-06.xml" (in "build-06.pdf") and focusing on the subset
      from 'buildsource' to 'shared' we can reduce it (in "build-07.xml")applying the
      previous rule and deleting these dependencies:

      • buildSource => shared
      • buildsource => drda
      • buildsource => engine
      • drda => engine
      • optional => engine

      before continuing there are some test that can be passed :

      • single target checking : a serie of deep clean + single target builds e.g. :
          ant -f build-07.xml clobber
          svn st --no-ignore
          del/rmdir items marked with "I"
          rmdir /Q /S classes.pptesting
          rmdir /Q /S jars
          del tools\java\junit.jar
          ant -f build-07.xml shared
          ... for shared, felixstub, engine, client, tools, drda, storeless, optional,
                  runner

      so the full build list, refactored in a windows batch command, is
      --------------------------------------------------------------------------------
      @echo off
      call ant clobber -file %1 -quiet -silent > nul 2>&1
      if errorlevel 1 (
          echo error %ERRORLEVEL% during first clobber
          exit /B 1
          )

      for %%t in (
              shared
              felixStubs
              engine
              client
              tools
              drda
              storeless
              optional
              runner
              buildsource
          ) do (
              call ant clobber -file %1 -quiet -silent > nul 2>&1
              rmdir /Q /S classes.pptesting > nul 2>&1
              rmdir /Q /S jars              > nul 2>&1
              del tools\java\junit.jar      > nul 2>&1
              rmdir /Q /S classes           > nul 2>&1
              rmdir /Q /S generated         > nul 2>&1
              rmdir /Q /S release           > nul 2>&1
              svn st --no-ignore | find "I    "
              echo ant -file %1 %%t
              call ant -file %1 -quiet -silent %%t
              if errorlevel 1 exit /B 1
              echo.
          )
      --------------------------------------------------------------------------------
      Executing
          "ant -f build-07.xml -quiet felixstubs"
      we get an unexpected fail (but tests exist for this) and looking at
          "13 ant -f build-07.xml felixStubs.log"
      we can notice that the root cause is an undefined variable.
      adding a dependency on 'state' to 'felixStubs' (on "build-08.xml") the build succeed
      ("14 ant -f build-08.xml felixStubs.log")
      Graphing "build-08.xml" (in "build.08.pdf") we notice that there's still a
      redundant dependency :
          shared => init
      covered by
          shared => state => showenv => init
      and, just on the right, another redundancy
          init => makeOutDirs
      already covered by
          init => setCompilerProperties => prebuild => makeOutDirs
      and, finally, a dependency
          engine => state
      already covered by
          engine => felixStubs => state
      and
          engine => shared => state
      so these can be removed.
      Building "build-09.xml" succeeds, and so the run test for all the targets
      ("15 test targets.cmd build-09.xml")
      --------------------------------------------------------------------------------
      The last tests are
        a) log comparison
        b) file comparison

      we define a cleaning sequence

      call ant clobber -f build-09.xml -quiet -silent
      rmdir /Q /S classes.pptesting
      rmdir /Q /S jars             
      rmdir /Q /S classes          
      del tools\java\junit.jar     
      svn st --no-ignore

      and, for the standard build sequence

      <cleaning sequence>
      ant -quiet all
      ant -quiet buildjars

      we generate a standard log, a verbose log
      <cleaning sequence>
      ant all       -f build.xml    >  "all - build.xml.log"
      ant buildjars -f build.xml    >> "all - build.xml.log"

      <cleaning sequence>
      ant all       -f build.xml -verbose >  "all - build.xml - verbose.log"
      ant buildjars -f build.xml -verbose >> "all - build.xml - verbose.log"

      <cleaning sequence>
      ant all       -f build-09.xml >  "all - build-09.xml.log"
      ant buildjars -f build-09.xml >> "all - build-09.xml.log"

      <cleaning sequence>
      ant all       -f build-09.xml -verbose >  "all - build-09.xml - verbose.log"
      ant buildjars -f build-09.xml -verbose >> "all - build-09.xml - verbose.log"

      all differences seems ok.
      --------------------------------------------------------------------------------
      After looking at "build-09.xml.pdf" we notice that
          buildsource => init
      is already covered by many path, eg
          buildsource => optional => tools => client => shared => state => showenv =>
              => init
      and also
          buildsource => prebuild
      is covered by
          ... => init => setCompilerProperties => prebuild
      and also
          buildsource => setCompilerProperties
      by the same path(s), so 'init', 'prebuild' and 'setCompilerProperties' can be
      deleted from 'buildsource' prerequisites in "build-10.xml".
      Tests and binary compare are ok.
      --------------------------------------------------------------------------------
      Looking at "build-10.xml.pdf" and "build-10.buildsource-expanded-01.pdf" we
      notice that 'build' target, 5th buildsource dependency, is ant-called anyway
      from 'engine' and can be deleted from 'buildsource' dependencies in
      "build-11.xml". Tests and binary compares pass.

      The only problem remaining is the 'generateSecurityPolicies' target that fails
      if built alone (due to a deep buried dependency)
      --------------------------------------------------------------------------------

      Summary of included files (maybe zipped) :

      File Content
      build@1863766.buildsource.dot actual build graphing - focused on buildsource
      build@1863766.buildsource.pdf   
      build@1863766.buildsource-expanded.dot same as above but with external build.xml expanded
      build@1863766.buildsource-expanded.pdf  
      build@1863766.xml actual build.xml
      build@1863766.xml.dot rendered in dot+pdf
      build@1863766.xml.pdf  
         
      build-11.buildsource.dot final build-11 graphing
      build-11.buildsource.pdf  
      build-11.buildsource-expanded.dot same as above but with external build.xml expanded
      build-11.buildsource-expanded.pdf  
      build-11.xml final build-11.xml
      build-11.xml.dot rendered in dot+pdf
      build-11.xml.pdf   
         
      patch.txt the tiny 65 lines patch

      Bye !

          Davide Grandi

       

        Attachments

        1. build@1863766.buildsource.pdf
          48 kB
          Davide Grandi
        2. build@1863766.buildsource-expanded.pdf
          56 kB
          Davide Grandi
        3. build@1863766.xml
          144 kB
          Davide Grandi
        4. build@1863766.xml.pdf
          102 kB
          Davide Grandi
        5. build-11.buildsource.pdf
          46 kB
          Davide Grandi
        6. build-11.buildsource-expanded.pdf
          55 kB
          Davide Grandi
        7. build-11.xml
          143 kB
          Davide Grandi
        8. build-11.xml.pdf
          99 kB
          Davide Grandi
        9. dot.zip
          16 kB
          Davide Grandi
        10. intermediate.zip
          1.39 MB
          Davide Grandi
        11. patch.txt
          3 kB
          Davide Grandi

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              davide.grandi@mclink.it Davide Grandi
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: