Bug 6606

Summary: META-BUG problems with delegating classloaders
Product: Ant Reporter: Stefan Bodewig <bodewig>
Component: CoreAssignee: Ant Notifications List <notifications>
Status: REOPENED ---    
Severity: enhancement CC: ant, apetrushko, bobby, dk, francis.andre, jglick, jimes, jochen.wiedmann, johnny_hujol, jrlawson, kreiger, martin.moorhead, Martin.vGagern, mfagan1, nicolas.mailhot, pete.cant, tommy
Priority: P3    
Version: 1.4   
Target Milestone: ---   
Hardware: All   
OS: All   
Bug Depends on: 38799    
Bug Blocks:    

Description Stefan Bodewig 2002-02-21 10:09:56 UTC
This META-BUG is here to collect all reports of the "<style> ignores <classpath>"
or "<junit> ignores <classpath>" type, which are symptoms of a common root
problem.

I'll try to describe the problem here and show how to work around the problem, the
goal is to gather feedback on this description and add it to the FAQ and the
manual after that.
Comment 1 Stefan Bodewig 2002-02-21 10:13:14 UTC
*** Bug 5947 has been marked as a duplicate of this bug. ***
Comment 2 Stefan Bodewig 2002-02-21 10:33:04 UTC
*** Bug 1118 has been marked as a duplicate of this bug. ***
Comment 3 Stefan Bodewig 2002-02-21 10:36:13 UTC
*** Bug 3934 has been marked as a duplicate of this bug. ***
Comment 4 Stefan Bodewig 2002-02-21 11:19:24 UTC
OK, first of all let's state that Ant adds all .jar files from ANT_HOME/lib
to CLASSPATH, therefore when I say "in CLASSPATH" in the rest, it means "either
in your CLASSPATH environment or ANT_HOME/lib".

This bug collects a common type of problem: A task needs an external library and
it has a nested classpath element so that you can point it to this external
library, but that doesn't work unless you put the external library into the
CLASSPATH.

The root of the problem is that the class that needs the external library
is on the CLASSPATH.

When you specify a nested <classpath> in Ant, Ant creates a new class loader
that uses the path you have specified.  It then tries to load additional
classes from this classloader.

In most cases - for example the two cases above - Ant doesn't load the external
library directly, it is the loaded class that does so.

In the case of <junit> it is the task implementation itself and in the case
of <style> it is the implementation of the
org.apache.tools.ant.taskdefs.XSLTLiaison class.

Ant's class loader implementation uses Java's delegation model, see
http://java.sun.com/products/jdk/1.2/docs/api/java/lang/ClassLoader.html
the paragraph

> The ClassLoader class uses a delegation model to search for classes and 
> resources. Each instance of ClassLoader has an associated parent class loader.
> When called upon to find a class or resource, a ClassLoader instance will 
> delegate the search for the class or resource to its parent class loader
> before attempting to find the class or resource itself. The virtual machine's
> built-in class loader, called the bootstrap class loader, does not itself have
> a parent but may serve as the parent of a ClassLoader instance.

This means, Ant's class loader will consult the bootstrap class loader first,
which tries to load classes from CLASSPATH.  The bootstrap class loader
doesn't know anything about Ant's class loader or even the path you have specified.

If the bootstrap class loader can load the class Ant has asked it to load,
this class will try to load the external library from CLASSPATH as well - it
doesn't know anything else - and will not find it unless the library is in
CLASSPATH as well.

To solve this, you have two major options:

(1) put all external libaries you need in CLASSPATH as well

this is not what you want, otherwise you wouldn't have found this bug report.

(2) remove the class that loads the external library from the CLASSPATH.

The easiest way to do this is to remove optional.jar from ANT_HOME.  If you do
so, you will have to <taskdef> all optional tasks and use nested <classpath>
elements in the <taskdef> tasks that point to the new location of optional.jar.
Also, don't forget to add the new location of optional.jar to the <classpath>
of your <style> or <junit> task.

If you want to avoid to <taskdef> all optional tasks you need, the only other
option is to remove the classes that should not be loaded via the bootstrap
class loader from optional.jar and put them into a separate archive. Add this
separate archive to the <classpath> of your <style> or <junit> task - and make
sure the separate archive is not in CLASSPATH.

In the case of <junit> you'd have to remove all classes that are in the
org/apache/tools/ant/taskdefs/optional/junit directory, in the <style> case
it is one of the *Liaison classes in org/apache/tools/ant/taskdefs/optional.

If you use the option to break up optional.jar for <junit>, you still have to
use a <taskdef> with a nested <classpath> to define the junit task.

Does this description make any sense?  Would it help people who face the same
problem you have seen?  If not, how can we improve it?
Comment 5 Stefan Bodewig 2002-02-21 11:54:16 UTC
*** Bug 4065 has been marked as a duplicate of this bug. ***
Comment 6 Jim Scarborough 2002-02-21 12:18:50 UTC
Bug 5947 offers what I think is a better workaround for JUnit, namely the fork=yes option.  True, it's not exactly what I want, but neither is it as complex (in my way of thinking) as disassembling and manually reassembling the ant optional framework.  I did come across an aside in the docs mentioning that the classpath wouldn't work unless we fork, but better documentation would help with the understanding so folks like me don't spend hours trying to figure out why the classpath tag isn't working in this excellent new build tool that I'm just beginning to figure out :-)  A warning from JUnit when it encounters a classpath tag in a non-fork task that wasn't taskdef'ed would also help to bring attention to the problem and help folks with the workaround without resolving it.How to resolve fully?  I'll have to cogitate more on that one.
Comment 7 Stefan Bodewig 2002-02-21 12:25:26 UTC
Thanks Jim,

yes, fork="true" helps in the case of <junit>, I forgot about it.
Comment 8 Steve Loughran 2002-02-22 07:12:12 UTC
*** Bug 5825 has been marked as a duplicate of this bug. ***
Comment 9 nicolas.mailhot 2002-02-22 11:04:11 UTC
Thanks for this very clear description of the problem from ant's point of view.
Now let me expose some of the very concrete problems we run because of this bug.

We use ant scripts to build our product on various software platform (some
linux, some forte/netbeans on windows with the integrated ant). Since our
external dependencies shift a lot they are downloaded on the build computer by
ant at the beginning of the build process. This is done via xslt analysis of xml
descriptors bundled with each of our modules. Installation documentation is also
processed with xslt at build time.

Putting all libraries in the classpath is therefore not possible since most of
the external jars used in a typical build do not exist on the build system
before ant start, and if they do they might not have the correct version for the
version of the product we are building (moreover the maturity and behaviour of
xslt java engines changes a lot from version to version and provider to provider
and using the wrong xslt jar with a version of our ant scripts will cause it to
fail)

Micromanaging ant installations on all the build systems would be a nightmare :
some are bundled with other products (forte) and anyway there is a very high
risk they would have to be changed on *all* system every time the main build
system changed a little or we tried to build an old version.

I appreciate this is a difficult problem and I do not expect ant to be fixed at
once but I really think it is worth it (and the number of bugs open on this
subjets kind of suggests it) 
Comment 10 Stefan Bodewig 2002-03-13 12:10:45 UTC
*** Bug 7081 has been marked as a duplicate of this bug. ***
Comment 11 Stefan Bodewig 2002-04-18 14:49:19 UTC
This now is http://jakarta.apache.org/ant/faq.html#delegating-classloader
Comment 12 jochen.wiedmann 2002-04-26 15:22:33 UTC
Stefan, your suggested solution "remove the class that loads the external 
library from the CLASSPATH" is not sufficient. It fails, for example, in
the case of Mappers. The problem is quite the same: The regexp packages
are not found, unless they are in lib. And, of course, I cannot remove
the Mapper classes from the classpath.

There has to be another solution for the problem. For example, TomCat
had quite the same thing with JAR's in WEB-INF/lib. For example, if
I have a WEB-INF/classes/jndi.properties then it wasn't found before
TomCat 3.2.4. I had to put it into TomCat's system class path. Nowadays
this works.


Comment 13 jochen.wiedmann 2002-06-12 07:32:16 UTC
I have a suggestion, which fixes most of the problems. More precise,
it someone depends on the "right" behaviour of the software components
in use. But the most important software component is, of course, Ant.

First, let me explain my view of the problem: Any Java class has
an associated ClassLoader. The ClassLoader has loaded the class,
thus it is able to load other classes from the same JAR file (or
files). If two classes A and B have different ClassLoaders and B
tries to load a class from A's JAR file with Class.forName(...),
then this doesn't necessarily work.

This is exactly the problem with Ant. For example, if I load a
class MyTask with

  <taskdef name="mytask" ...><classpath>...</classpath></taskdef>

then the class MyTask has the right ClassLoader, but the Ant
classes don't. So, if for example the MyTask uses a class MyType
implementing a nested element of MyTask, then the IntrospectionHelper
will do a Class.forName("MyType"). This fails, because Class.forName
uses the ClassLoader of the IntrospectionHelper.

The proposed solution works as follows:

  - Never use Class.forName(). Replace it with a helper method
    like

      public static Class myForName(String name)
          throws ClassNotFoundException {
        try {
          return Class.forName(name);
        } catch (ClassNotFoundException e) {
          ClassLoader cl = Thread.currentThread().getContextClassLoader();
          if (cl == null) {
            throw e;
          }
          return cl.loadClass(name);
        }

  - Make sure, that the threads ContextClassLoader is right. For
    example, use a helper class ThreadContextClassLoader, which
    grabs the current ContextClassLoader and an additional
    ClassLoader and uses Class.forName(), original thread
    ContextClassLoader and the additional ClassLoader, in that
    order and enclose the perform() method with

      Task task;
      ThreadClassLoader tcl =
        new ThreadClassLoader(task.getClass().getClassLoader());
      ClassLoader cl = Thread.currentThread().getClassLoader();
      try {
        Thread.currentThread.setContextClassLoader(tcl);
        task.perform;
      } finally {
        Thread.currentThread.setContextClassLoader(cl);
      }

I offer to create a patch, if someone (Stefan?) discusses details
with me and advices me, which version to patch exactly.


Regards,

Jochen
Comment 14 Stefan Bodewig 2002-06-12 07:40:44 UTC
One problem with you approach is, that the context classloader is not available
in JDK 1.1.

We already have at least half of the infrastructure for your proposal in place,
i.e. AntClassLoader knows how to set and unset itself as the context classloader
(via reflection) and the oat.ant.util.LoaderUtil class can retrieve the context
class loader via reflection.

All that we'd need now:

(1) set the context class loader in the right places
(2) use it consistently
(3) ensure we don't break backwards compatibility

I'll be happy to assist here, the context would be no earlier than Ant 1.6 of
course.
Comment 15 Conor MacNeill 2002-06-12 08:25:21 UTC
The statement below is not correct.

> then the class MyTask has the right ClassLoader, but the Ant
> classes don't. So, if for example the MyTask uses a class MyType
> implementing a nested element of MyTask, then the IntrospectionHelper
> will do a Class.forName("MyType"). This fails, because Class.forName
> uses the ClassLoader of the IntrospectionHelper.

Introspection Helper gets the class of the nested type from the class being
introspected through reflection. This class will have been loaded by MyTask's
loader when the MyTask class was resolved by the VM. Class.forName is not used
in IntrospectionHelper unless you are trying to contruct a Class argument which
would be very uncommon (probably never).

Just FYI...
Comment 16 Stephane Bailliez 2002-08-15 09:56:59 UTC
*** Bug 9841 has been marked as a duplicate of this bug. ***
Comment 17 Stephane Bailliez 2002-08-16 15:26:19 UTC
*** Bug 10931 has been marked as a duplicate of this bug. ***
Comment 18 Stefan Bodewig 2002-11-28 14:40:35 UTC
*** Bug 14931 has been marked as a duplicate of this bug. ***
Comment 19 Stefan Bodewig 2003-08-06 11:13:32 UTC
*** Bug 22170 has been marked as a duplicate of this bug. ***
Comment 20 Tommy Svensson 2003-08-06 17:04:54 UTC
Since my bug report (22170) has been marked as a duplicate of this one, I guess
its better that I put my comment in this instead.

I think I might know what the problem is. Since reporting my bug I have made my
own ClassLoader for my project, and I got exactly the same exception as Ant
does! It complains that it cannot find "Path.class". What I didn't notice at
first was that the class it complains about not finding is not having any
package! In my case the Path class does belong to a package. But somewhere in my
code there is a 

  if (this.interfaceClass.getPackage() != null) ...

getPackage() did actually return null when this.interfaceClass had been loaded
by my class loader causing "Path.class" without a package to be looked for a bit
down using Class.forName(). This ofcourse failed with the ClassNotFoundException.

What I did was to add a call to definePackage(...) with the class package before
calling defineClass() in findClass(). When the class is then defined the package
 the class file refers to is also defined and the defined class can reference it.
After this fix my class loader works perfectly.

If I bring up AntClassLoader.java in my editor and do a find on "definePackage"
nothing is found! So AntClassLoader never calls definePackage()! That is the
source of my problem!

Best Regards, Tommy Svensson
Comment 21 Stefan Bodewig 2005-03-11 16:22:41 UTC
*** Bug 33440 has been marked as a duplicate of this bug. ***
Comment 22 Tobias K. Tobiasen 2005-05-30 21:39:55 UTC
I have a patch that makes junit work without any changes to the ant installation.
The only requirement is that you fork a new JVM for the junit test.
How did I do this? I changed the ant-junit.jar so it does not load/reference any
junit classes before it forks a new JVM. That way the classloader problems
disappear since junit classes are not loaded inside ant's JVM.

If you want to give it a testspin, get the jar file here:
1) Download ant-junit.jar from here http://tobiasen.dk/ant/ant-junit.jar
2) Drop it in $ANT_HOME/lib
3) Run your tests, remember to set fork="yes" in your tests and have the
junit.jar in the classpath of the test. 
<junit fork="yes" printsummary="yes">
  <classpath>
	include junit.jar here...
  </classpath>
...

I have compiled the ant-junit.jar using jdk 1.4.2_08 and against ant 1.6.4.

I would like to submit the patch, but I have no experience in submitting
patches. Please tell me how. 
The patch is quite big, since almost all classes in
org/apache/tools/ant/taskdefs/optional/junit are changed and I have introduced
some new interfaces. 
Comment 23 Antoine Levy-Lambert 2005-05-30 22:37:27 UTC
http://ant.apache.org/ant_task_guidelines.html to read about how to create
patches ....
Comment 24 Jesse Glick 2005-05-31 23:37:38 UTC
(In reply to comment #22)
> I changed the ant-junit.jar so it does not load/reference any
> junit classes before it forks a new JVM. That way the classloader problems
> disappear since junit classes are not loaded inside ant's JVM.

This could be quite useful. E.g. the NetBeans IDE uses

http://www.netbeans.org/download/dev/javadoc/org-apache-tools-ant-module/org/apache/tools/ant/module/spi/AutomaticExtraClasspathProvider.html

from the module bundling junit.jar in order to force it to be added to Ant's
primary classpath. Even though the build scripts that the IDE generates will
have an explicit path for junit.jar - which is used e.g. when compiling unit
tests - it cannot currently run <junit> unless this has been done. A patch such
as is described here would make it possible to do omit junit.jar from Ant's
primary classpath, and would mean that a command-line build incl. unit testing
would succeed so long as the path was set up correctly, without requiring the
user to run Ant with junit.jar in $CLASSPATH (or ant/lib/ or whatever).
Comment 25 Jesse Barnum 2005-06-07 15:32:00 UTC
Why not simply remove the optional ant tasks from the classpath in the default install of ant? This 
seems pretty reasonable:

* For those that don't mind modifying their ant home directory, it's not really any extra work: since you 
can't use tasks like <junit> without manually adding junit.jar to the ant lib folder anyway, it doesn't 
seem like much more work to drag in two jars (ant-junit.jar & junit.jar) rather than one (junit.jar) into 
the lib folder.

* For this (like me) who want to create a user-friendly build process that does not require users to 
modify their ant installs, I can redistribute the relevant jars with my project and load them within my 
build.xml file.

I want my users to just be able to download my package and type 'ant' to have everything work. The 
only way I see to do this right now is to copy the source for the JUnit optional task into my own 
package, so that it is not loaded by the core classloader. This is a pretty crummy workaround.
Comment 26 Johnny Hujol 2005-09-29 16:26:04 UTC
*** Bug 36862 has been marked as a duplicate of this bug. ***
Comment 27 Sam Wilson 2005-11-02 20:52:38 UTC
(In reply to comment #25)
> Why not simply remove the optional ant tasks from the classpath in the default
install of ant? This 
> seems pretty reasonable:
> 
> * For those that don't mind modifying their ant home directory, it's not
really any extra work: since you 
> can't use tasks like <junit> without manually adding junit.jar to the ant lib
folder anyway, it doesn't 
> seem like much more work to drag in two jars (ant-junit.jar & junit.jar)
rather than one (junit.jar) into 
> the lib folder.
> 
> * For this (like me) who want to create a user-friendly build process that
does not require users to 
> modify their ant installs, I can redistribute the relevant jars with my
project and load them within my 
> build.xml file.
> 
> I want my users to just be able to download my package and type 'ant' to have
everything work. The 
> only way I see to do this right now is to copy the source for the JUnit
optional task into my own 
> package, so that it is not loaded by the core classloader. This is a pretty
crummy workaround.

I think this is the best all-around solution to the problem, with one addition:

Put the optional ant tasks into a separate folder (optional/lib, opt/lib or some
such) and then add a default property called ant.optional.library.dir that is
similar to the ant.library.dir except that it points to this new optional
library directory.

This way, if you want to use an optional task you can simply include the
external jars in your distribution and then write a taskdef whose classpath
includes the appropriate optional.jar file and its dependencies.

While this does create a little more work for anyone wanting to use the optional
tasks, the volume of help requests would be simplified by adding an sample
taskdef to the doc page of each optional task.

As it stands right now, the only workable solution to this problem makes ant
much less useful to people trying to build code rather than people trying to
write ant scripts. I would imagine that usability should focus first on people
trying to build code before it focuses on people trying to write ant scripts
because (typically) the person writing the ant script will be somewhat more
technical than the other. Besides, it makes the ant script more self documenting
with respect to its dependencies.
Comment 28 Steve Loughran 2005-11-03 12:22:50 UTC
> this does create a little more work for anyone wanting to use the optional
tasks, 

i.e. it would break every single build file on the planet that uses any optional
task.

Sam, we cannot do that. we cannot break every build file. We need a better
solution. For reference, my project's build process is documented here:

http://cvs.sourceforge.net/viewcvs.py/*checkout*/smartfrog/core/antbuild/doc/third_generation_build_process.sxw

We have a directory of libraries under SCM, this directory contains the releases
of add on tasks that we require for the build. You need either to declare this
with a -lib option in ANT_OPTS, or pull the files into ANT_HOME/lib
${user.home}/.ant/lib

Life would be simpler if we could somehow declare in a per-project basis an
extra dir of stuff to load. All that stuff would need to be unloaded at the end
of the build of course, or you have just contaminated a hosting JVM. I just dont
see an easy way of doing this. 
Comment 29 Tobias K. Tobiasen 2005-11-03 13:30:24 UTC
> Sam, we cannot do that. we cannot break every build file. We need a better
> solution. 
I want to stress that there is a _simple_ solution for this problem. No need to
break any existing build files or rearrange jar files. This is true for the
junit task anyway.
How? Simply fork junit tests into a separate JVM and make sure only the new JVM
uses junit classes. There is really no need for the JUnitTask.java task to
import junit.framework.Test, it is just sloppy code. I have made this change and
it can be done in less than a day! 
The only drawback is that people that do not fork tests into a separate JVM
still needs the current workaround.

@See http://issues.apache.org/bugzilla/show_bug.cgi?id=6606#c22
Comment 30 Jesse Glick 2006-02-27 20:42:29 UTC
I have filed a separate bug #38799 to cover the specific case of junit.jar in
<junit>, since this metabug seems too broad to close directly.
Comment 31 Martin von Gagern 2010-01-18 09:26:28 UTC
I've got two cross references, both patches available.

Bug #47003 makes AntClassLoader in combination with the <classloader> task much more powerful, by ensuring that the ant core class loader will always be an AntClassLoader, which in turn allows addition of class path elements. The result is that you can simply use <classloader> to extend the core ant classpath at need, and don't have to worry about a complicated classloader hierarchy.

Bug #47002 takes yet another shot at <junit> and the missing TraXLiaison. This time without forking, but instead through yet another nested <classpath> element.

Unfortunately both patches have seen no comments or review yet. Still waiting...