[DERBY-7052] Reordering and (mildly) reorganizing build.xml Ant targets around buildsource

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 10.15.1.3
• Fix Version/s: 10.15.2.0, 10.16.1.1
• Component/s: Build tools
• Assignee: Unassigned
• Reporter: Davide Grandi
• Created: 16/Aug/19 17:21
• Updated: 12/Jan/20 17:50
• Resolved: 20/Aug/19 13:15

Description

Module dependencies should be reflected by Ant targets dependencies.
And Ant targets execution follows the rules stated in
    https://ant.apache.org/manual/targets.html
so a dependency could be covered (being redundant) by applying transitivity rule
to every dependency's dependecies (please look at the dot+pdf diagrams).
--------------------------------------------------------------------------------
Modules dependencies, by the docs, are :

module depends	on
org.apache.derby.client	org.apache.derby.commons
org.apache.derby.engine	org.apache.derby.commons
org.apache.derby.locale_*	org.apache.derby.commons
org.apache.derby.tools	org.apache.derby.client
org.apache.derby.tools	org.apache.derby.engine
org.apache.derby.optionaltools	org.apache.derby.tools
org.apache.derby.server	org.apache.derby.tools
org.apache.derby.runner	org.apache.derby.server
org.apache.derby.engine	org.osgi.framework

but actual ant dependencies are (partial target => module map) :

 

target	depends	and module	depends on
buildsource	client
buildsource	shared
buildsource	felixStubs		org.osgi.framework
engine	shared	org.apache.derby.engine	org.apache.derby.commons
client	shared	org.apache.derby.client	org.apache.derby.commons
tools	engine	org.apache.derby.tools	org.apache.derby.engine
drda	engine
runner	optional
optional	engine

so we expect that building 'engine' target alone will fails due to the lack of
'felixstubs' dependency.
And it fails ("01 ant engine.log") quoting "requires static org.osgi.framework;"

So the best try is to delete (on copy "build-01.xml" of "build.xml") the
dependency
  buildsource => felixstubs
and to put it as last dependency of 'engine' target
Result : success ("02 ant -f build-01.xml engine.txt")

Similarly we can expect that also 'tools' fails, since it really depends on
'client' but this is not explicited in "build.xml".
And it fails ("03 ant -f build-01.xml tools.log") with:
    [javac]     requires static org.apache.derby.client;

So the best try is to switch the 'client' dependency (in copy "build-02.xml" of
"build-01.xml") from 'buildsource' to 'tools'.

Result: success ("04 ant -f build-02.xml tools.log").

The same thing for 'optional' that should depends on 'tools' but depends only on
'engine'.
And it fails ("05 ant -f build-02.xml optional.log") with
    [javac]     requires org.apache.derby.tools;
So we switch the 'tools' dependency from 'buildsource' to 'optional' (in copy
"build-03.xml" of "build.02.xml")
And it succeeds ("06 ant -f build-03.xml optional.log").

Same as 'drda' (aka server) that require 'tools' ("07 ant -f build-03.xml drda.log")
    [javac]     requires org.apache.derby.tools;
that succeds after adding 'tools' dependency to 'drda'
("08 ant -f build-04.xml drda.log")
and also for 'runner' ("09 ant -f build-04.xml runner.log") that requires 'drda'
    [javac]     requires org.apache.derby.server;
and succeeds after adding it into copy "build-05.xml" of "build-04.xml".
("10 ant -f build-05.xml runner.log")

After graphing "build-05.xml" (in "build-05.xml.pdf") one can notice that,
apart from redundant prerequisites (covered later), there's an extra dependency
on 'runner', that in module hierararchy needs only 'drda'.
After duplicating "build-05.xml" in "build-06.xml" and edited, removing
'runner' -> 'optional' prerequisite, build of 'runner' succeeds
("11 ant -f build-06.xml runner.log")

The next step is removing all the extra dependencies.
Given the transitivity of dependencies in this set
A => C (a depends on C)
A => B (a depends on B)
B => C (B depends on C)
the first dependendency is already covered by second and third one, and ant
will ignore it.

After graphing "build-06.xml" (in "build-06.pdf") and focusing on the subset
from 'buildsource' to 'shared' we can reduce it (in "build-07.xml")applying the
previous rule and deleting these dependencies:

• buildSource => shared
• buildsource => drda
• buildsource => engine
• drda => engine
• optional => engine

before continuing there are some test that can be passed :

• single target checking : a serie of deep clean + single target builds e.g. :
    ant -f build-07.xml clobber
    svn st --no-ignore
    del/rmdir items marked with "I"
    rmdir /Q /S classes.pptesting
    rmdir /Q /S jars
    del tools\java\junit.jar
    ant -f build-07.xml shared
    ... for shared, felixstub, engine, client, tools, drda, storeless, optional,
            runner

so the full build list, refactored in a windows batch command, is
--------------------------------------------------------------------------------
@echo off
call ant clobber -file %1 -quiet -silent > nul 2>&1
if errorlevel 1 (
    echo error %ERRORLEVEL% during first clobber
    exit /B 1
    )

for %%t in (
        shared
        felixStubs
        engine
        client
        tools
        drda
        storeless
        optional
        runner
        buildsource
    ) do (
        call ant clobber -file %1 -quiet -silent > nul 2>&1
        rmdir /Q /S classes.pptesting > nul 2>&1
        rmdir /Q /S jars              > nul 2>&1
        del tools\java\junit.jar      > nul 2>&1
        rmdir /Q /S classes           > nul 2>&1
        rmdir /Q /S generated         > nul 2>&1
        rmdir /Q /S release           > nul 2>&1
        svn st --no-ignore | find "I    "
        echo ant -file %1 %%t
        call ant -file %1 -quiet -silent %%t
        if errorlevel 1 exit /B 1
        echo.
    )
--------------------------------------------------------------------------------
Executing
    "ant -f build-07.xml -quiet felixstubs"
we get an unexpected fail (but tests exist for this) and looking at
    "13 ant -f build-07.xml felixStubs.log"
we can notice that the root cause is an undefined variable.
adding a dependency on 'state' to 'felixStubs' (on "build-08.xml") the build succeed
("14 ant -f build-08.xml felixStubs.log")
Graphing "build-08.xml" (in "build.08.pdf") we notice that there's still a
redundant dependency :
    shared => init
covered by
    shared => state => showenv => init
and, just on the right, another redundancy
    init => makeOutDirs
already covered by
    init => setCompilerProperties => prebuild => makeOutDirs
and, finally, a dependency
    engine => state
already covered by
    engine => felixStubs => state
and
    engine => shared => state
so these can be removed.
Building "build-09.xml" succeeds, and so the run test for all the targets
("15 test targets.cmd build-09.xml")
--------------------------------------------------------------------------------
The last tests are
  a) log comparison
  b) file comparison

we define a cleaning sequence
–
call ant clobber -f build-09.xml -quiet -silent
rmdir /Q /S classes.pptesting
rmdir /Q /S jars             
rmdir /Q /S classes          
del tools\java\junit.jar     
svn st --no-ignore
–
and, for the standard build sequence
–
<cleaning sequence>
ant -quiet all
ant -quiet buildjars
–
we generate a standard log, a verbose log
<cleaning sequence>
ant all       -f build.xml    >  "all - build.xml.log"
ant buildjars -f build.xml    >> "all - build.xml.log"

<cleaning sequence>
ant all       -f build.xml -verbose >  "all - build.xml - verbose.log"
ant buildjars -f build.xml -verbose >> "all - build.xml - verbose.log"

<cleaning sequence>
ant all       -f build-09.xml >  "all - build-09.xml.log"
ant buildjars -f build-09.xml >> "all - build-09.xml.log"

<cleaning sequence>
ant all       -f build-09.xml -verbose >  "all - build-09.xml - verbose.log"
ant buildjars -f build-09.xml -verbose >> "all - build-09.xml - verbose.log"

all differences seems ok.
--------------------------------------------------------------------------------
After looking at "build-09.xml.pdf" we notice that
    buildsource => init
is already covered by many path, eg
    buildsource => optional => tools => client => shared => state => showenv =>
        => init
and also
    buildsource => prebuild
is covered by
    ... => init => setCompilerProperties => prebuild
and also
    buildsource => setCompilerProperties
by the same path(s), so 'init', 'prebuild' and 'setCompilerProperties' can be
deleted from 'buildsource' prerequisites in "build-10.xml".
Tests and binary compare are ok.
--------------------------------------------------------------------------------
Looking at "build-10.xml.pdf" and "build-10.buildsource-expanded-01.pdf" we
notice that 'build' target, 5th buildsource dependency, is ant-called anyway
from 'engine' and can be deleted from 'buildsource' dependencies in
"build-11.xml". Tests and binary compares pass.

The only problem remaining is the 'generateSecurityPolicies' target that fails
if built alone (due to a deep buried dependency)
--------------------------------------------------------------------------------

Summary of included files (maybe zipped) :

File	Content
[e-mail]	actual build graphing - focused on buildsource
[e-mail]
[e-mail]	same as above but with external build.xml expanded
[e-mail]
[e-mail]	actual build.xml
[e-mail]	rendered in dot+pdf
[e-mail]
build-11.buildsource.dot	final build-11 graphing
build-11.buildsource.pdf
build-11.buildsource-expanded.dot	same as above but with external build.xml expanded
build-11.buildsource-expanded.pdf
build-11.xml	final build-11.xml
build-11.xml.dot	rendered in dot+pdf
build-11.xml.pdf
patch.txt	the tiny 65 lines patch

Bye !

    Davide Grandi