Derby
  1. Derby
  2. DERBY-3986

Stop dropping build artifacts in the subversion-controlled source tree

    Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 10.5.1.1
    • Fix Version/s: None
    • Component/s: Build tools
    • Labels:
      None

      Description

      The Derby build drops various artifacts in the source tree. Build artifacts should not muddy subversion controlled directories but should, instead, go into temporary directories created by the the build.

      The following is a (perhaps partial) list of artifacts currently dropped into subversion-controlled space:

      Sanity directives
      Generated grammars
      The English error messages
      Data type class sizes
      Toursdb
      Temporary class for verifying compiler level

      1. derby-3986-05-aa-class_size_catalog.diff
        0.7 kB
        Rick Hillegas
      2. derby-3986-04-aa-toursdb-load-tables.diff
        4 kB
        Rick Hillegas
      3. derby-3986-03-aa-release.diff
        11 kB
        Rick Hillegas
      4. filemode.diff
        0.6 kB
        Knut Anders Hatlen
      5. mk-sanity-dir.diff
        0.8 kB
        Knut Anders Hatlen
      6. chmod.diff
        0.5 kB
        Knut Anders Hatlen
      7. derby-3986-02-ah-sanity-bin-toursdb-storeless-release.diff
        26 kB
        Rick Hillegas
      8. derby-3986-02-ag-sanity-bin-toursdb-storeless-release.diff
        24 kB
        Rick Hillegas
      9. derby-3986-02-af-sanity-bin-toursdb-storeless-release.diff
        23 kB
        Rick Hillegas
      10. derby-3986-01-aa-checkCompilerLevel.diff
        0.9 kB
        Rick Hillegas

        Issue Links

          Activity

          Hide
          Rick Hillegas added a comment -

          Attaching derby-3986-05-aa-class_size_catalog.diff. This patch improves the dependency tracking for ClassSizeCatalog. Committed at subversion revision 1034093.

          Right now we are running the class_size_catalog target every time we build because we are still looking for ClassSizeCatalog over in the source tree. But this intermediate source file now lives in the generated source directory. This patch corrects the dependency tracking to look for ClassSizeCatalog in the generated tree rather than in the source tree.

          Touches the following file:

          M build.xml

          Show
          Rick Hillegas added a comment - Attaching derby-3986-05-aa-class_size_catalog.diff. This patch improves the dependency tracking for ClassSizeCatalog. Committed at subversion revision 1034093. Right now we are running the class_size_catalog target every time we build because we are still looking for ClassSizeCatalog over in the source tree. But this intermediate source file now lives in the generated source directory. This patch corrects the dependency tracking to look for ClassSizeCatalog in the generated tree rather than in the source tree. Touches the following file: M build.xml
          Hide
          Rick Hillegas added a comment -

          Attaching derby-3986-04-aa-toursdb-load-tables.diff. This fixes something broken by the 02 patch: Starting with the previous patch, the loading of the toursdb tables stopped working because the loadTables.sql script ran other scripts using file names relative to the current directory. The previous patch changed the current directory of the ij invocation, moving from the source tree into the generated tree. Committed at subversion revision 1033052.

          This patch introduces a new runijscript target and runs each of the smaller load scripts individually.

          Touches the following files:

          M java/demo/toursdb/build.xml

          Show
          Rick Hillegas added a comment - Attaching derby-3986-04-aa-toursdb-load-tables.diff. This fixes something broken by the 02 patch: Starting with the previous patch, the loading of the toursdb tables stopped working because the loadTables.sql script ran other scripts using file names relative to the current directory. The previous patch changed the current directory of the ij invocation, moving from the source tree into the generated tree. Committed at subversion revision 1033052. This patch introduces a new runijscript target and runs each of the smaller load scripts individually. Touches the following files: M java/demo/toursdb/build.xml
          Hide
          Rick Hillegas added a comment -

          Attaching derby-3986-03-aa-release.diff. This patch changes the build so that release artifacts are now dropped outside subversion-controlled space as other build artifacts are. This change should not disrupt the build for most people. Committed at subversion revision 1003607.

          This patch makes a number of changes:

          1) Creates a new artifact directory called "release" rooted off $

          {out.base}. This new directory sits parallel to classes, generated, jars, and javadoc. This directory will now be created every time you build Derby and the directory will be deleted when you issue an "ant clobber".

          2) Drops all release artifacts into the new release tree. This includes snapshots, release distributions, and the temporary directories needed to create them.

          3) Eliminates the packaging.properties file, which release managers were expected to create from packaging.tmpl. Instead, the release machinery now just uses packaging.tmpl directly. Rather than creating packaging.properties to override the default values in packaging.tmpl, now you just override the release properties the same way you override other build properties: i.e., by putting overrides in your ant.properties or specifying them on the command line.

          4) Makes environment variables visible to the release-building machinery. This makes it possible for you to phrase your ant.properties in terms of environment variables which can be overridden in your shell or script. An ant reference to an environment variable looks like this:

          ${env.FOO}

          where FOO is an environment variable set by a shell statement like this:

          export FOO=/bar/wibble/wombat



          Touches the following files:

          ---------

          M .

          Adds the new release directory to the list of files ignored by "svn status".


          ---------

          M tools/ant/properties/dirs.properties
          M tools/ant/properties/packaging.tmpl

          Defines release directories relative to ${out.base}

          .

          ---------

          M build.xml
          M tools/release/build.xml

          Wires these changes into the release machinery.

          Show
          Rick Hillegas added a comment - Attaching derby-3986-03-aa-release.diff. This patch changes the build so that release artifacts are now dropped outside subversion-controlled space as other build artifacts are. This change should not disrupt the build for most people. Committed at subversion revision 1003607. This patch makes a number of changes: 1) Creates a new artifact directory called "release" rooted off $ {out.base}. This new directory sits parallel to classes, generated, jars, and javadoc. This directory will now be created every time you build Derby and the directory will be deleted when you issue an "ant clobber". 2) Drops all release artifacts into the new release tree. This includes snapshots, release distributions, and the temporary directories needed to create them. 3) Eliminates the packaging.properties file, which release managers were expected to create from packaging.tmpl. Instead, the release machinery now just uses packaging.tmpl directly. Rather than creating packaging.properties to override the default values in packaging.tmpl, now you just override the release properties the same way you override other build properties: i.e., by putting overrides in your ant.properties or specifying them on the command line. 4) Makes environment variables visible to the release-building machinery. This makes it possible for you to phrase your ant.properties in terms of environment variables which can be overridden in your shell or script. An ant reference to an environment variable looks like this: ${env.FOO} where FOO is an environment variable set by a shell statement like this: export FOO=/bar/wibble/wombat Touches the following files: --------- M . Adds the new release directory to the list of files ignored by "svn status". --------- M tools/ant/properties/dirs.properties M tools/ant/properties/packaging.tmpl Defines release directories relative to ${out.base} . --------- M build.xml M tools/release/build.xml Wires these changes into the release machinery.
          Hide
          Knut Anders Hatlen added a comment -

          Committed revision 1003408.

          Show
          Knut Anders Hatlen added a comment - Committed revision 1003408.
          Hide
          Knut Anders Hatlen added a comment -

          One small observation regarding the "filemode" attribute error mentioned above. The change that triggered it introduced a tiny inconsistency, as can be seen here:

          <!-- bin -->
          <tarfileset dir="$

          {basedir}

          /bin" prefix="$

          {derby.bin}/bin"
          mode="755" includes="*" excludes="templates"/>
          <tarfileset dir="${generated.bin.dir}" prefix="${derby.bin}

          /bin"
          filemode="755" includes="*"/>

          The old target uses the "mode" attribute, whereas the new target uses "filemode". We should use the same for both the tarfilesets. "mode" seems to work with both Ant 1.6.5 and Ant 1.7.x, but only "filemode" is mentioned in the 1.7.x manuals, as far as I can see, so we should probably stick to that one.

          Unless there are any objections, I plan to commit the attached patch (filemode.diff) to make both the tarfilesets use "filemode".

          Show
          Knut Anders Hatlen added a comment - One small observation regarding the "filemode" attribute error mentioned above. The change that triggered it introduced a tiny inconsistency, as can be seen here: <!-- bin --> <tarfileset dir="$ {basedir} /bin" prefix="$ {derby.bin}/bin" mode="755" includes="*" excludes="templates"/> <tarfileset dir="${generated.bin.dir}" prefix="${derby.bin} /bin" filemode="755" includes="*"/> The old target uses the "mode" attribute, whereas the new target uses "filemode". We should use the same for both the tarfilesets. "mode" seems to work with both Ant 1.6.5 and Ant 1.7.x, but only "filemode" is mentioned in the 1.7.x manuals, as far as I can see, so we should probably stick to that one. Unless there are any objections, I plan to commit the attached patch (filemode.diff) to make both the tarfilesets use "filemode".
          Hide
          Knut Anders Hatlen added a comment -

          I noticed some more automated build jobs that stopped working after these changes. These jobs were running "ant release" and failed with

          (...)/tools/release/build.xml:207: Class org.apache.tools.ant.taskdefs.Tar$TarFileSet doesn't support the "filemode" attribute.

          The problem was that the changes introduced a dependency on Ant 1.7, and the build jobs used Ant 1.6.5. Changing the build environment to something newer made the problem go away. I think this is fine, since BUILDING.html already states that Ant 1.7.0 or higher is required, but I thought I'd mention it in case others come across the same issue.

          Show
          Knut Anders Hatlen added a comment - I noticed some more automated build jobs that stopped working after these changes. These jobs were running "ant release" and failed with (...)/tools/release/build.xml:207: Class org.apache.tools.ant.taskdefs.Tar$TarFileSet doesn't support the "filemode" attribute. The problem was that the changes introduced a dependency on Ant 1.7, and the build jobs used Ant 1.6.5. Changing the build environment to something newer made the problem go away. I think this is fine, since BUILDING.html already states that Ant 1.7.0 or higher is required, but I thought I'd mention it in case others come across the same issue.
          Hide
          Kathey Marsden added a comment -

          yes, user error here too. It's probably not worth changing the nighties order of things although it does make sense to clobber at the same level you built.

          Show
          Kathey Marsden added a comment - yes, user error here too. It's probably not worth changing the nighties order of things although it does make sense to clobber at the same level you built.
          Hide
          Myrna van Lunteren added a comment -

          Thanks for this improvement, Rick, looks like you thought of everything. Apologies for my lack of attention to your instructions. All works fine for me now.

          Show
          Myrna van Lunteren added a comment - Thanks for this improvement, Rick, looks like you thought of everything. Apologies for my lack of attention to your instructions. All works fine for me now.
          Hide
          Rick Hillegas added a comment -

          Hi Myrna,

          Thanks for bringing up this topic. I did edit the release targets and I did verify that the scripts are being dropped into the bin directories of the release distributions. See the STATE OF TESTING section of my long comment on 20-Sep-2010 above. The distributions look correct to me, but this is clearly an area we will have to watch closely when we release 10.7. Thanks.

          Show
          Rick Hillegas added a comment - Hi Myrna, Thanks for bringing up this topic. I did edit the release targets and I did verify that the scripts are being dropped into the bin directories of the release distributions. See the STATE OF TESTING section of my long comment on 20-Sep-2010 above. The distributions look correct to me, but this is clearly an area we will have to watch closely when we release 10.7. Thanks.
          Hide
          Myrna van Lunteren added a comment -

          I didn't study the build.xml closely, but is that something that will trip us up come release time? These files need to go into the bin dir in the bin distributions...

          Show
          Myrna van Lunteren added a comment - I didn't study the build.xml closely, but is that something that will trip us up come release time? These files need to go into the bin dir in the bin distributions...
          Hide
          Rick Hillegas added a comment - - edited

          Hi Kathey and Myrna,

          Did you perform the following steps in this order when you sync'd up to the changes introduced by this patch:

          ant clobber
          svn update

          That is the order of operations which is expected to work. The reverse order will leave a lot of cruft in the source tree. The cruft will confuse the build and produce a lot of noise for "svn status". So just to repeat, the following sequence of steps will result in an experience which is consistent with the symptoms you are seeing:

          svn update
          ant clobber

          If your environments are still broken, the following may help:

          1) Throw away your subversion client and re-initialize it from scratch. This is the technique used by Kristian and Knut to unwedge the automated builds.

          2) Or, set your client to before revision 999928, then do "ant clobber", then do "svn update"

          Thanks,
          -Rick

          Show
          Rick Hillegas added a comment - - edited Hi Kathey and Myrna, Did you perform the following steps in this order when you sync'd up to the changes introduced by this patch: ant clobber svn update That is the order of operations which is expected to work. The reverse order will leave a lot of cruft in the source tree. The cruft will confuse the build and produce a lot of noise for "svn status". So just to repeat, the following sequence of steps will result in an experience which is consistent with the symptoms you are seeing: svn update ant clobber If your environments are still broken, the following may help: 1) Throw away your subversion client and re-initialize it from scratch. This is the technique used by Kristian and Knut to unwedge the automated builds. 2) Or, set your client to before revision 999928, then do "ant clobber", then do "svn update" Thanks, -Rick
          Hide
          Knut Anders Hatlen added a comment -

          I think the issue Kathey experienced is expected if ant clobber wasn't performed before svn update. It should only happen once, since those files won't be generated in an updated checkout.

          The files Myrna found in bin are the scripts we generate at build time after DERBY-3207. They went directly into the bin directory before Rick's changes, but now they go into the generated directory too.

          Show
          Knut Anders Hatlen added a comment - I think the issue Kathey experienced is expected if ant clobber wasn't performed before svn update. It should only happen once, since those files won't be generated in an updated checkout. The files Myrna found in bin are the scripts we generate at build time after DERBY-3207 . They went directly into the bin directory before Rick's changes, but now they go into the generated directory too.
          Hide
          Myrna van Lunteren added a comment -

          Just to add my experience after these changes...
          I found that ant clobber did not fully clean up the files left over.
          Perhaps I should have done ant clobber before svn update (...)

          Without doing a manual clean up of the tree (based on svn stat), I got errors about duplicate classes (in the parser area) which makes sense.

          Oddly enough, svn stat also showed up a number of files in the bin directory, like NetworkServerControl.
          I assume I should have cleaned those up a long time ago?

          After cleaning up, in one of my environments, where I didn't set sane/insane at all, I then saw some errors indicating missing files, but running ant with -Dsane=false cleared that up, and all looks ok now for me.

          Show
          Myrna van Lunteren added a comment - Just to add my experience after these changes... I found that ant clobber did not fully clean up the files left over. Perhaps I should have done ant clobber before svn update (...) Without doing a manual clean up of the tree (based on svn stat), I got errors about duplicate classes (in the parser area) which makes sense. Oddly enough, svn stat also showed up a number of files in the bin directory, like NetworkServerControl. I assume I should have cleaned those up a long time ago? After cleaning up, in one of my environments, where I didn't set sane/insane at all, I then saw some errors indicating missing files, but running ant with -Dsane=false cleared that up, and all looks ok now for me.
          Hide
          Kathey Marsden added a comment -

          I noticed in my client and in the IBM nightly Linux build that ant clobber was not a good enough cleanup.
          The following errors occurred:

          [javac] /local1/cloudtst/dev/src/opensource/generated/java/org/apache/derby/impl/sql/compile/SQLParser.java:152: duplicate class: org.apache.derby.impl.sql.compile.SQLParser
          [javac] public class SQLParser implements SQLParserConstants {
          [javac] /local1/cloudtst/dev/src/opensource/generated/java/org/apache/derby/impl/sql/compile/SQLParserConstants.java:25: duplicate class: org.apache.derby.impl.sql.compile.SQLParserConstants
          [javac] public interface SQLParserConstants {
          [javac] /local1/cloudtst/dev/src/opensource/generated/java/org/apache/derby/impl/sql/compile/SQLParserTokenManager.java:136: duplicate class: org.apache.derby.impl.sql.compile.SQLParserTokenManager
          [javac] public class SQLParserTokenManager implements SQLParserConstants
          [javac] /local1/cloudtst/dev/src/opensource/generated/java/org/apache/derby/impl/sql/compile/TokenMgrError.java:25: duplicate class: org.apache.derby.impl.sql.compile.TokenMgrError
          [javac] public class TokenMgrError extends Error
          [javac] 4 errors
          [javac] /local1/cloudtst/dev/src/opensource/generated/java/org/apache/derby/impl/tools/ij/CharStream.java:19: duplicate class: org.apache.derby.impl.tools.ij.CharStream
          [javac] public interface CharStream {
          [javac] /local1/cloudtst/dev/src/opensource/generated/java/org/apache/derby/impl/tools/ij/SimpleCharStream.java:9: duplicate class: org.apache.derby.impl.tools.ij.SimpleCharStream
          [javac] public class SimpleCharStream
          [javac] /local1/cloudtst/dev/src/opensource/generated/java/org/apache/derby/impl/tools/ij/Token.java:8: duplicate class: org.apache.derby.impl.tools.ij.Token
          [javac] public class Token {
          [javac] /local1/cloudtst/dev/src/opensource/generated/java/org/apache/derby/impl/tools/ij/TokenMgrError.java:4: duplicate class: org.apache.derby.impl.tools.ij.TokenMgrError
          [javac] public class TokenMgrError extends Error
          [javac] /local1/cloudtst/dev/src/opensource/generated/java/org/apache/derby/impl/tools/ij/ij.java:67: duplicate class: org.apache.derby.impl.tools.ij.ij

          I removed the following files that showed as ? in svn stat and did update to get toursdb and then could build.
          java/tools/org/apache/derby/impl/tools/ij/ijConstants.java
          java/tools/org/apache/derby/impl/tools/ij/ijTokenManager.java
          java/tools/org/apache/derby/impl/tools/ij/TokenMgrError.java
          java/tools/org/apache/derby/impl/tools/ij/ij.java
          java/tools/org/apache/derby/impl/tools/ij/SimpleCharStream.java
          java/tools/org/apache/derby/impl/tools/ij/Token.java
          java/tools/org/apache/derby/impl/tools/ij/mtGrammarTokenManager.java
          java/tools/org/apache/derby/impl/tools/ij/mtGrammarConstants.java
          java/tools/org/apache/derby/impl/tools/ij/mtGrammar.java
          java/tools/org/apache/derby/impl/tools/ij/CharStream.java
          java/demo/toursdb/toursdb
          java/demo/toursdb/derby.log
          java/demo/toursdb/toursdb.out
          java/demo/toursdb/toursdb.jar
          java/engine/org/apache/derby/impl/sql/compile/SQLParserTokenManager.java
          java/engine/org/apache/derby/impl/sql/compile/SQLParser.java
          java/engine/org/apache/derby/impl/sql/compile/TokenMgrError.java
          java/engine/org/apache/derby/impl/sql/compile/SQLParserConstants.java
          java/engine/org/apache/derby/loc/messages_en.properties
          java/shared/org/apache/derby/shared/common/sanity/SanityState.java
          bin/NetworkServerControl
          bin/stopNetworkServer
          bin/sysinfo
          bin/startNetworkServer
          bin/ij
          bin/dblook

          Show
          Kathey Marsden added a comment - I noticed in my client and in the IBM nightly Linux build that ant clobber was not a good enough cleanup. The following errors occurred: [javac] /local1/cloudtst/dev/src/opensource/generated/java/org/apache/derby/impl/sql/compile/SQLParser.java:152: duplicate class: org.apache.derby.impl.sql.compile.SQLParser [javac] public class SQLParser implements SQLParserConstants { [javac] /local1/cloudtst/dev/src/opensource/generated/java/org/apache/derby/impl/sql/compile/SQLParserConstants.java:25: duplicate class: org.apache.derby.impl.sql.compile.SQLParserConstants [javac] public interface SQLParserConstants { [javac] /local1/cloudtst/dev/src/opensource/generated/java/org/apache/derby/impl/sql/compile/SQLParserTokenManager.java:136: duplicate class: org.apache.derby.impl.sql.compile.SQLParserTokenManager [javac] public class SQLParserTokenManager implements SQLParserConstants [javac] /local1/cloudtst/dev/src/opensource/generated/java/org/apache/derby/impl/sql/compile/TokenMgrError.java:25: duplicate class: org.apache.derby.impl.sql.compile.TokenMgrError [javac] public class TokenMgrError extends Error [javac] 4 errors [javac] /local1/cloudtst/dev/src/opensource/generated/java/org/apache/derby/impl/tools/ij/CharStream.java:19: duplicate class: org.apache.derby.impl.tools.ij.CharStream [javac] public interface CharStream { [javac] /local1/cloudtst/dev/src/opensource/generated/java/org/apache/derby/impl/tools/ij/SimpleCharStream.java:9: duplicate class: org.apache.derby.impl.tools.ij.SimpleCharStream [javac] public class SimpleCharStream [javac] /local1/cloudtst/dev/src/opensource/generated/java/org/apache/derby/impl/tools/ij/Token.java:8: duplicate class: org.apache.derby.impl.tools.ij.Token [javac] public class Token { [javac] /local1/cloudtst/dev/src/opensource/generated/java/org/apache/derby/impl/tools/ij/TokenMgrError.java:4: duplicate class: org.apache.derby.impl.tools.ij.TokenMgrError [javac] public class TokenMgrError extends Error [javac] /local1/cloudtst/dev/src/opensource/generated/java/org/apache/derby/impl/tools/ij/ij.java:67: duplicate class: org.apache.derby.impl.tools.ij.ij I removed the following files that showed as ? in svn stat and did update to get toursdb and then could build. java/tools/org/apache/derby/impl/tools/ij/ijConstants.java java/tools/org/apache/derby/impl/tools/ij/ijTokenManager.java java/tools/org/apache/derby/impl/tools/ij/TokenMgrError.java java/tools/org/apache/derby/impl/tools/ij/ij.java java/tools/org/apache/derby/impl/tools/ij/SimpleCharStream.java java/tools/org/apache/derby/impl/tools/ij/Token.java java/tools/org/apache/derby/impl/tools/ij/mtGrammarTokenManager.java java/tools/org/apache/derby/impl/tools/ij/mtGrammarConstants.java java/tools/org/apache/derby/impl/tools/ij/mtGrammar.java java/tools/org/apache/derby/impl/tools/ij/CharStream.java java/demo/toursdb/toursdb java/demo/toursdb/derby.log java/demo/toursdb/toursdb.out java/demo/toursdb/toursdb.jar java/engine/org/apache/derby/impl/sql/compile/SQLParserTokenManager.java java/engine/org/apache/derby/impl/sql/compile/SQLParser.java java/engine/org/apache/derby/impl/sql/compile/TokenMgrError.java java/engine/org/apache/derby/impl/sql/compile/SQLParserConstants.java java/engine/org/apache/derby/loc/messages_en.properties java/shared/org/apache/derby/shared/common/sanity/SanityState.java bin/NetworkServerControl bin/stopNetworkServer bin/sysinfo bin/startNetworkServer bin/ij bin/dblook
          Hide
          Knut Anders Hatlen added a comment -

          Committed revision 1000219.

          Show
          Knut Anders Hatlen added a comment - Committed revision 1000219.
          Hide
          Knut Anders Hatlen added a comment -

          I noticed that some automated build jobs started failing after these changes. The jobs did "ant insane" on a clean setup, and failed because the "insane" target tried to create state.properties in a directory that hadn't been created yet. The same problem can be seen by invoking "ant sane" or "ant insane" right after an "ant clobber".

          Attaching a patch (mk-sanity-dir.diff) that makes the "sane" and "insane" targets depend on "make-generated-dirs". I can now run "ant sane" and "ant insane" on a clean checkout again. Will commit shortly.

          Show
          Knut Anders Hatlen added a comment - I noticed that some automated build jobs started failing after these changes. The jobs did "ant insane" on a clean setup, and failed because the "insane" target tried to create state.properties in a directory that hadn't been created yet. The same problem can be seen by invoking "ant sane" or "ant insane" right after an "ant clobber". Attaching a patch (mk-sanity-dir.diff) that makes the "sane" and "insane" targets depend on "make-generated-dirs". I can now run "ant sane" and "ant insane" on a clean checkout again. Will commit shortly.
          Hide
          Knut Anders Hatlen added a comment -

          I noticed that the generated bin scripts don't have the executable bit set after these changes. The attached patch (chmod.diff) corrects the path in the chmod command in the makebinscript target.

          Committed revision 1000068.

          Show
          Knut Anders Hatlen added a comment - I noticed that the generated bin scripts don't have the executable bit set after these changes. The attached patch (chmod.diff) corrects the path in the chmod command in the makebinscript target. Committed revision 1000068.
          Hide
          Knut Anders Hatlen added a comment -

          I also removed the svn:ignore property on the bin directory and committed revision 1000066.

          Show
          Knut Anders Hatlen added a comment - I also removed the svn:ignore property on the bin directory and committed revision 1000066.
          Hide
          Rick Hillegas added a comment -

          I have created a new issue, DERBY-4811, to track the simplification of the clean targets.

          The following work remains to be done on this issue: Adjust the release build targets so that they stop dropping their artifacts in subversion-controlled space.

          Show
          Rick Hillegas added a comment - I have created a new issue, DERBY-4811 , to track the simplification of the clean targets. The following work remains to be done on this issue: Adjust the release build targets so that they stop dropping their artifacts in subversion-controlled space.
          Hide
          Rick Hillegas added a comment -

          Thanks, Knut. Committed derby-3986-02-ah-sanity-bin-toursdb-storeless-release.diff at subversion revision 999928.

          Show
          Rick Hillegas added a comment - Thanks, Knut. Committed derby-3986-02-ah-sanity-bin-toursdb-storeless-release.diff at subversion revision 999928.
          Hide
          Knut Anders Hatlen added a comment -

          > I am inclined to check this in soon unless you see some problem with it.

          +1 to checking in the patch, and to your proposed way forward.

          Show
          Knut Anders Hatlen added a comment - > I am inclined to check this in soon unless you see some problem with it. +1 to checking in the patch, and to your proposed way forward.
          Hide
          Rick Hillegas added a comment -

          Thanks for the additional quick feedback, Knut. Attaching a third rev of the patch: derby-3986-02-ah-sanity-bin-toursdb-storeless-release.diff. This new version adds a cleangenerated target and tidies up the other clean targets. I am inclined to check this in soon unless you see some problem with it.

          I think we should delete the useless clean targets as a separate patch. I would wait a week or so after committing the first patch, in order to give people time to adjust to the new generated directory, just in case there are problems. The cleanstate target is still used by the release machinery when building both sane and insane distributions. I recommend the following course of action for the next patch:

          o Remove cleanparsers, cleancatalog, cleantoursb

          o Leave the other clean targets: clean, cleangenerated, cleanstate, clobber

          Beyond that, we could consider another patch to remove cleangenerated and cleanstate. As a final piece of work, we could consider making clean and clobber synonyms.

          Show
          Rick Hillegas added a comment - Thanks for the additional quick feedback, Knut. Attaching a third rev of the patch: derby-3986-02-ah-sanity-bin-toursdb-storeless-release.diff. This new version adds a cleangenerated target and tidies up the other clean targets. I am inclined to check this in soon unless you see some problem with it. I think we should delete the useless clean targets as a separate patch. I would wait a week or so after committing the first patch, in order to give people time to adjust to the new generated directory, just in case there are problems. The cleanstate target is still used by the release machinery when building both sane and insane distributions. I recommend the following course of action for the next patch: o Remove cleanparsers, cleancatalog, cleantoursb o Leave the other clean targets: clean, cleangenerated, cleanstate, clobber Beyond that, we could consider another patch to remove cleangenerated and cleanstate. As a final piece of work, we could consider making clean and clobber synonyms.
          Hide
          Knut Anders Hatlen added a comment -

          > So would the following make sense:
          >
          > i) Move cleangenerated logic out of clean into its own target
          >
          > ii) Tidy up the cleanparsers and cleanstate targets

          Yes, makes sense. +1

          > Of course, I could also eliminate the cleanparsers, cleantoursdb,
          > cleancatalog, and cleanstate targets as you suggest. But these
          > targets may still be useful. They may have been created to
          > compensate for the spotty dependency tracking in our build
          > scripts. What do you think?

          I don't see much value in these targets myself, so unless someone
          speaks up and says they actually use them regularly, I'd suggest that
          we take this opportunity to trim down the size of the build
          script. I'd rather add it back later if it turns out to be needed,
          instead of keeping all the old cruft in case some of it turns out to
          be useful.

          Show
          Knut Anders Hatlen added a comment - > So would the following make sense: > > i) Move cleangenerated logic out of clean into its own target > > ii) Tidy up the cleanparsers and cleanstate targets Yes, makes sense. +1 > Of course, I could also eliminate the cleanparsers, cleantoursdb, > cleancatalog, and cleanstate targets as you suggest. But these > targets may still be useful. They may have been created to > compensate for the spotty dependency tracking in our build > scripts. What do you think? I don't see much value in these targets myself, so unless someone speaks up and says they actually use them regularly, I'd suggest that we take this opportunity to trim down the size of the build script. I'd rather add it back later if it turns out to be needed, instead of keeping all the old cruft in case some of it turns out to be useful.
          Hide
          Rick Hillegas added a comment -

          Thanks for the quick review, Knut. I am attaching a second rev of the patch which addresses your point (b): derby-3986-02-ag-sanity-bin-toursdb-storeless-release.diff

          More responses follow:

          kah> I think clean and clobber are essentially the same now. So whereas "ant insane; ant clean; ant" would previously build non-debug classes, the same sequence of commands will build debug classes after the patch has been checked in. Does that sound about right? I don't think this is a problem, but it might come as a surprise on someone.

          Thanks for catching that. It might not be what someone wants. I could move the deletion of the generated tree to a new cleangenerated target. Then clean would go back to just deleting the built classes.

          kah> While you're at it, do you think it would make sense to

          kah> a) remove the cleanparsers, cleantoursdb, cleancatalog and cleanstate targets from build.xml, since these are all taken care of implicitly by the clean target now?

          If I created a cleangenerated target, then the other clean targets would still make sense I think. I don't know who uses these targets. However, now that you bring my attention to these targets, I think that cleanparsers needs to be changed to refer to the generated tree rather than the source tree, and cleanstate needs a little touchup too.

          So would the following make sense:

          i) Move cleangenerated logic out of clean into its own target

          ii) Tidy up the cleanparsers and cleanstate targets

          Of course, I could also eliminate the cleanparsers, cleantoursdb, cleancatalog, and cleanstate targets as you suggest. But these targets may still be useful. They may have been created to compensate for the spotty dependency tracking in our build scripts. What do you think?

          kah> b) clear the svn:ignore property in

          Thanks for this suggestion. I have included this in the new version of the patch, which I'm attaching.

          Thanks,
          -Rick

          Show
          Rick Hillegas added a comment - Thanks for the quick review, Knut. I am attaching a second rev of the patch which addresses your point (b): derby-3986-02-ag-sanity-bin-toursdb-storeless-release.diff More responses follow: kah> I think clean and clobber are essentially the same now. So whereas "ant insane; ant clean; ant" would previously build non-debug classes, the same sequence of commands will build debug classes after the patch has been checked in. Does that sound about right? I don't think this is a problem, but it might come as a surprise on someone. Thanks for catching that. It might not be what someone wants. I could move the deletion of the generated tree to a new cleangenerated target. Then clean would go back to just deleting the built classes. kah> While you're at it, do you think it would make sense to kah> a) remove the cleanparsers, cleantoursdb, cleancatalog and cleanstate targets from build.xml, since these are all taken care of implicitly by the clean target now? If I created a cleangenerated target, then the other clean targets would still make sense I think. I don't know who uses these targets. However, now that you bring my attention to these targets, I think that cleanparsers needs to be changed to refer to the generated tree rather than the source tree, and cleanstate needs a little touchup too. So would the following make sense: i) Move cleangenerated logic out of clean into its own target ii) Tidy up the cleanparsers and cleanstate targets Of course, I could also eliminate the cleanparsers, cleantoursdb, cleancatalog, and cleanstate targets as you suggest. But these targets may still be useful. They may have been created to compensate for the spotty dependency tracking in our build scripts. What do you think? kah> b) clear the svn:ignore property in Thanks for this suggestion. I have included this in the new version of the patch, which I'm attaching. Thanks, -Rick
          Hide
          Knut Anders Hatlen added a comment -

          Hi Rick,

          This looks like a good improvement to me.

          I think clean and clobber are essentially the same now. So whereas "ant insane; ant clean; ant" would previously build non-debug classes, the same sequence of commands will build debug classes after the patch has been checked in. Does that sound about right? I don't think this is a problem, but it might come as a surprise on someone.

          While you're at it, do you think it would make sense to

          a) remove the cleanparsers, cleantoursdb, cleancatalog and cleanstate targets from build.xml, since these are all taken care of implicitly by the clean target now?

          b) clear the svn:ignore property in

          java/demo/toursdb
          java/tools/org/apache/derby/impl/tools/ij
          java/engine/org/apache/derby/impl/sql/compile
          java/engine/org/apache/derby/loc
          java/shared/org/apache/derby/shared/common/sanity/

          so that "svn status" will warn us if we have old files lying around by accident?

          Show
          Knut Anders Hatlen added a comment - Hi Rick, This looks like a good improvement to me. I think clean and clobber are essentially the same now. So whereas "ant insane; ant clean; ant" would previously build non-debug classes, the same sequence of commands will build debug classes after the patch has been checked in. Does that sound about right? I don't think this is a problem, but it might come as a surprise on someone. While you're at it, do you think it would make sense to a) remove the cleanparsers, cleantoursdb, cleancatalog and cleanstate targets from build.xml, since these are all taken care of implicitly by the clean target now? b) clear the svn:ignore property in java/demo/toursdb java/tools/org/apache/derby/impl/tools/ij java/engine/org/apache/derby/impl/sql/compile java/engine/org/apache/derby/loc java/shared/org/apache/derby/shared/common/sanity/ so that "svn status" will warn us if we have old files lying around by accident?
          Hide
          Rick Hillegas added a comment -

          The regression tests passed cleanly for me on jars built using this patch.

          Show
          Rick Hillegas added a comment - The regression tests passed cleanly for me on jars built using this patch.
          Hide
          Rick Hillegas added a comment -

          Attaching derby-3986-02-af-sanity-bin-toursdb-storeless-release.diff. This patch intends to put build artifacts in a directory tree separate from the subversion-controlled source tree. This writeup addresses the following topics:

          o Motivation

          o High level description

          o New tree for generated artifacts

          o Remaining work

          o State of testing

          o How to apply the patch

          o Patch details

          ------------------ MOTIVATION ----------------

          Right now the Derby build drops artifacts into the subversion-controlled source tree. This makes it awkward to run multiple builds using different environments, since each new build clobbers the output of other builds. After this patch is applied, it should be possible to build Derby in different environments without clobbering the results of other builds. In particular, the following use-cases should be supported:

          o Building Derby in parallel using different rev levels of Java

          o Building Derby on a hypervisor whose guest operating systems share a read-only subversion client but compile into os-specific output directories.

          ------------------ HIGH LEVEL DESCRIPTION ----------------

          This patch introduces two high level concepts:

          1) A new artifact directory called "generated", which lives parallel to the classes, jars, and javadoc directories already created by the build machinery. This new directory is the root of a tree where the build puts generated java files and other artifacts which do not fit tidily into the existing classes, jars, or javadoc directories.

          2) A new variable $

          {out.base} which you can customize in your ant.properties or on the command line. ${out.base}

          is the base directory for all build artifacts. It defaults to be $

          {basedir}

          . That means that if you do not customize $

          {out.base}, then most of your build artifacts (including classes, jars, and javadoc) will continue to be dropped into the same locations where they are dropped today. If you do customize ${out.base}

          , then all of the following artifacts will be dropped into the $

          {out.base} directory:

          changenumber.properties
          classes
          classes.pptesting
          classes.storeless
          generated
          jars
          javadoc




          ------------------ NEW TREE FOR GENERATED ARTIFACTS ----------------

          After applying this patch and building Derby, you will see the following new subtree:

          generated
          generated/bin
          generated/bin/dblook
          generated/bin/ij
          generated/bin/NetworkServerControl
          generated/bin/startNetworkServer
          generated/bin/stopNetworkServer
          generated/bin/sysinfo
          generated/java
          generated/java/org
          generated/java/org/apache
          generated/java/org/apache/derby
          generated/java/org/apache/derby/iapi
          generated/java/org/apache/derby/iapi/services
          generated/java/org/apache/derby/iapi/services/cache
          generated/java/org/apache/derby/iapi/services/cache/ClassSizeCatalog.java
          generated/java/org/apache/derby/impl
          generated/java/org/apache/derby/impl/sql
          generated/java/org/apache/derby/impl/sql/compile
          generated/java/org/apache/derby/impl/sql/compile/SQLParser.java
          generated/java/org/apache/derby/impl/sql/compile/SQLParserConstants.java
          generated/java/org/apache/derby/impl/sql/compile/SQLParserTokenManager.java
          generated/java/org/apache/derby/impl/sql/compile/TokenMgrError.java
          generated/java/org/apache/derby/impl/tools
          generated/java/org/apache/derby/impl/tools/ij
          generated/java/org/apache/derby/impl/tools/ij/CharStream.java
          generated/java/org/apache/derby/impl/tools/ij/ij.java
          generated/java/org/apache/derby/impl/tools/ij/ijConstants.java
          generated/java/org/apache/derby/impl/tools/ij/ijTokenManager.java
          generated/java/org/apache/derby/impl/tools/ij/mtGrammar.java
          generated/java/org/apache/derby/impl/tools/ij/mtGrammarConstants.java
          generated/java/org/apache/derby/impl/tools/ij/mtGrammarTokenManager.java
          generated/java/org/apache/derby/impl/tools/ij/SimpleCharStream.java
          generated/java/org/apache/derby/impl/tools/ij/Token.java
          generated/java/org/apache/derby/impl/tools/ij/TokenMgrError.java
          generated/java/org/apache/derby/loc
          generated/java/org/apache/derby/loc/messages_en.properties
          generated/java/org/apache/derby/shared
          generated/java/org/apache/derby/shared/common
          generated/java/org/apache/derby/shared/common/sanity
          generated/java/org/apache/derby/shared/common/sanity/SanityState.java
          generated/java/org/apache/derby/shared/common/sanity/state.properties
          generated/toursdb
          generated/toursdb/derby.log
          generated/toursdb/toursdb
          generated/toursdb/toursdb.jar
          generated/toursdb/toursdb.out
          generated/toursdb/toursdb/...



          ------------------ REMAINING WORK ----------------

          More work may be needed to relocate the special artifacts created by the release build targets. Those targets may not behave correctly if you customize ${out.base}

          .

          ------------------ STATE OF TESTING ----------------

          I have tested building a subversion client on both a host operating system and a guest operating system, with each os building into its own artifact tree.

          I have also built a release on my host operating system using the default $

          {out.base}. The following artifacts (now built into the generated tree) appear in the correct locations in the bin distributions:

          - merged scripts in the bin directory
          - toursdb database in demo/databases

          I am running regression tests against the host os build to verify that everything was generated correctly.


          ------------------ HOW TO APPLY THE PATCH ----------------

          To apply this patch, do the following:

          o First, "ant clean" your subversion client. This removes most of the generated artifacts from your client (there are some stragglers like changenumber.properties which will still hang around). Technically, this step should not be necessary and if you omit this step, the build probably will function correctly after you apply the patch. However, I have not tested the case of applying this patch to a dirty client, so I advise you to "ant clean" first.

          o Then apply the patch as you would normally do.


          ------------------ PATCH DETAILS ----------------

          Touches the following files:

          ------

          M .

          Updates the svn:ignore property so that the new generated directory does not confuse "ant status".


          ------

          M tools/ant/properties/dirs.properties

          Adds the ${out.base}

          variable and makes it the root of the artifact directories.

          ------

          M java/tools/org/apache/derby/impl/tools/build.xml

          Builds the tools grammar into the generated tree.

          ------

          M java/engine/org/apache/derby/impl/sql/build.xml

          Builds the SQL grammar into the generated tree. Deletes the 3 artifacts which are overridden by checked-in sources:

          CharStream.java
          ParseException.java
          Token.java

          As a follow-on effort, we may want to consider eliminating the checked-in sources.

          ------

          M java/demo/build.xml
          M java/demo/toursdb/build.xml

          Drops the toursdb database into the generated directory.

          ------

          M java/engine/org/apache/derby/loc/build.xml

          Drops the English messages into the generated directory.

          ------

          M java/shared/build.xml

          Drops the sanity state into the generated directory.

          ------

          M tools/release/build.xml

          Now looks into the generated tree for toursdb and the merged bin scripts.

          ------

          M build.xml

          Changes to the master build script to pull all of this together.

          Show
          Rick Hillegas added a comment - Attaching derby-3986-02-af-sanity-bin-toursdb-storeless-release.diff. This patch intends to put build artifacts in a directory tree separate from the subversion-controlled source tree. This writeup addresses the following topics: o Motivation o High level description o New tree for generated artifacts o Remaining work o State of testing o How to apply the patch o Patch details ------------------ MOTIVATION ---------------- Right now the Derby build drops artifacts into the subversion-controlled source tree. This makes it awkward to run multiple builds using different environments, since each new build clobbers the output of other builds. After this patch is applied, it should be possible to build Derby in different environments without clobbering the results of other builds. In particular, the following use-cases should be supported: o Building Derby in parallel using different rev levels of Java o Building Derby on a hypervisor whose guest operating systems share a read-only subversion client but compile into os-specific output directories. ------------------ HIGH LEVEL DESCRIPTION ---------------- This patch introduces two high level concepts: 1) A new artifact directory called "generated", which lives parallel to the classes, jars, and javadoc directories already created by the build machinery. This new directory is the root of a tree where the build puts generated java files and other artifacts which do not fit tidily into the existing classes, jars, or javadoc directories. 2) A new variable $ {out.base} which you can customize in your ant.properties or on the command line. ${out.base} is the base directory for all build artifacts. It defaults to be $ {basedir} . That means that if you do not customize $ {out.base}, then most of your build artifacts (including classes, jars, and javadoc) will continue to be dropped into the same locations where they are dropped today. If you do customize ${out.base} , then all of the following artifacts will be dropped into the $ {out.base} directory: changenumber.properties classes classes.pptesting classes.storeless generated jars javadoc ------------------ NEW TREE FOR GENERATED ARTIFACTS ---------------- After applying this patch and building Derby, you will see the following new subtree: generated generated/bin generated/bin/dblook generated/bin/ij generated/bin/NetworkServerControl generated/bin/startNetworkServer generated/bin/stopNetworkServer generated/bin/sysinfo generated/java generated/java/org generated/java/org/apache generated/java/org/apache/derby generated/java/org/apache/derby/iapi generated/java/org/apache/derby/iapi/services generated/java/org/apache/derby/iapi/services/cache generated/java/org/apache/derby/iapi/services/cache/ClassSizeCatalog.java generated/java/org/apache/derby/impl generated/java/org/apache/derby/impl/sql generated/java/org/apache/derby/impl/sql/compile generated/java/org/apache/derby/impl/sql/compile/SQLParser.java generated/java/org/apache/derby/impl/sql/compile/SQLParserConstants.java generated/java/org/apache/derby/impl/sql/compile/SQLParserTokenManager.java generated/java/org/apache/derby/impl/sql/compile/TokenMgrError.java generated/java/org/apache/derby/impl/tools generated/java/org/apache/derby/impl/tools/ij generated/java/org/apache/derby/impl/tools/ij/CharStream.java generated/java/org/apache/derby/impl/tools/ij/ij.java generated/java/org/apache/derby/impl/tools/ij/ijConstants.java generated/java/org/apache/derby/impl/tools/ij/ijTokenManager.java generated/java/org/apache/derby/impl/tools/ij/mtGrammar.java generated/java/org/apache/derby/impl/tools/ij/mtGrammarConstants.java generated/java/org/apache/derby/impl/tools/ij/mtGrammarTokenManager.java generated/java/org/apache/derby/impl/tools/ij/SimpleCharStream.java generated/java/org/apache/derby/impl/tools/ij/Token.java generated/java/org/apache/derby/impl/tools/ij/TokenMgrError.java generated/java/org/apache/derby/loc generated/java/org/apache/derby/loc/messages_en.properties generated/java/org/apache/derby/shared generated/java/org/apache/derby/shared/common generated/java/org/apache/derby/shared/common/sanity generated/java/org/apache/derby/shared/common/sanity/SanityState.java generated/java/org/apache/derby/shared/common/sanity/state.properties generated/toursdb generated/toursdb/derby.log generated/toursdb/toursdb generated/toursdb/toursdb.jar generated/toursdb/toursdb.out generated/toursdb/toursdb/... ------------------ REMAINING WORK ---------------- More work may be needed to relocate the special artifacts created by the release build targets. Those targets may not behave correctly if you customize ${out.base} . ------------------ STATE OF TESTING ---------------- I have tested building a subversion client on both a host operating system and a guest operating system, with each os building into its own artifact tree. I have also built a release on my host operating system using the default $ {out.base}. The following artifacts (now built into the generated tree) appear in the correct locations in the bin distributions: - merged scripts in the bin directory - toursdb database in demo/databases I am running regression tests against the host os build to verify that everything was generated correctly. ------------------ HOW TO APPLY THE PATCH ---------------- To apply this patch, do the following: o First, "ant clean" your subversion client. This removes most of the generated artifacts from your client (there are some stragglers like changenumber.properties which will still hang around). Technically, this step should not be necessary and if you omit this step, the build probably will function correctly after you apply the patch. However, I have not tested the case of applying this patch to a dirty client, so I advise you to "ant clean" first. o Then apply the patch as you would normally do. ------------------ PATCH DETAILS ---------------- Touches the following files: ------ M . Updates the svn:ignore property so that the new generated directory does not confuse "ant status". ------ M tools/ant/properties/dirs.properties Adds the ${out.base} variable and makes it the root of the artifact directories. ------ M java/tools/org/apache/derby/impl/tools/build.xml Builds the tools grammar into the generated tree. ------ M java/engine/org/apache/derby/impl/sql/build.xml Builds the SQL grammar into the generated tree. Deletes the 3 artifacts which are overridden by checked-in sources: CharStream.java ParseException.java Token.java As a follow-on effort, we may want to consider eliminating the checked-in sources. ------ M java/demo/build.xml M java/demo/toursdb/build.xml Drops the toursdb database into the generated directory. ------ M java/engine/org/apache/derby/loc/build.xml Drops the English messages into the generated directory. ------ M java/shared/build.xml Drops the sanity state into the generated directory. ------ M tools/release/build.xml Now looks into the generated tree for toursdb and the merged bin scripts. ------ M build.xml Changes to the master build script to pull all of this together.
          Hide
          Knut Anders Hatlen added a comment -

          Not only are the generated grammars dropped into the svn-controlled space, but at least two of the generated files are even checked in (impl/sql/compile/ParseException.java and impl/tools/ij/ParseException.java) so that they are not re-generated. I tried to remove them and did "ant clobber" + "ant" to regenerate them, and then some of the syntax errors actually became more helpful.

          For example, the error message for this statement

          ij> create table t;

          changed from

          ERROR 42X01: Syntax error: Encountered "<EOF>" at line 1, column 14.

          to the more helpful

          ERROR 42X01: Syntax error: Encountered "<EOF>" at line 1, column 14.
          Was expecting one of:
          "as" ...
          "(" ...
          .

          Some of the error messages did however become very long after this change, like the error for "select * from t1 inner join t2" which listed more than 150 alternatives.

          Show
          Knut Anders Hatlen added a comment - Not only are the generated grammars dropped into the svn-controlled space, but at least two of the generated files are even checked in (impl/sql/compile/ParseException.java and impl/tools/ij/ParseException.java) so that they are not re-generated. I tried to remove them and did "ant clobber" + "ant" to regenerate them, and then some of the syntax errors actually became more helpful. For example, the error message for this statement ij> create table t; changed from ERROR 42X01: Syntax error: Encountered "<EOF>" at line 1, column 14. to the more helpful ERROR 42X01: Syntax error: Encountered "<EOF>" at line 1, column 14. Was expecting one of: "as" ... "(" ... . Some of the error messages did however become very long after this change, like the error for "select * from t1 inner join t2" which listed more than 150 alternatives.
          Hide
          Myrna van Lunteren added a comment -

          seems to me all currently available patches have been committed...

          Show
          Myrna van Lunteren added a comment - seems to me all currently available patches have been committed...
          Hide
          Rick Hillegas added a comment -

          Committed derby-3986-01-aa-checkCompilerLevel.diff at subversion revision 727725.

          Show
          Rick Hillegas added a comment - Committed derby-3986-01-aa-checkCompilerLevel.diff at subversion revision 727725.
          Hide
          Rick Hillegas added a comment -

          Attaching derby-3986-01-aa-checkCompilerLevel.diff. This patch changes the way that we determine whether the compiler is at least at the Java 5 level. Instead of doing a dummy compile and checking for the output, we just check to see if the classpath contains a class added in Java 5.

          Touches the following files:

          M build.xml

          Show
          Rick Hillegas added a comment - Attaching derby-3986-01-aa-checkCompilerLevel.diff. This patch changes the way that we determine whether the compiler is at least at the Java 5 level. Instead of doing a dummy compile and checking for the output, we just check to see if the classpath contains a class added in Java 5. Touches the following files: M build.xml

            People

            • Assignee:
              Unassigned
              Reporter:
              Rick Hillegas
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:

                Development