Thanks for the updated patch Xuan Gong!
I had bit of time to sit down to go over the patch and think about related points.
(1) URL.setURLStreamHandlerFactory() (and supporting non-local paths)
Regarding setting the URLStreamHandlerFactory, you can call URL.setURLStreamHandlerFactory() at most once on a JVM, and any attempt to set it again within the same process will throw an error:
Sets an application's URLStreamHandlerFactory. This method can be called at most once in a given Java Virtual Machine.
In the patch, this is being called inside a for loop. This will throw an Error for any subsequent initialization of aux services. If this was needed to be able to handle non-local paths (like hdfs), then we would need to find a different way than this method to handle it.
On a related note, how important is it to support non-local classpaths? If implementing it is not trivial, you might want to separate that work into a separate JIRA and address that. I'd be curious to hear how important that part of the feature is.
(2) other types of classloading
The patch will ensure that the aux service class itself will be loaded by the application classloader and any class that needs to be loaded in a normal manner as part of executing the aux service class. However, there are other types of classloading that can happen. Two major types you need to consider are classloading via Configuration.getClass() and reflection using thread context classloader (via Thread.currentThread().getContextClassLoader()).
For example, if the aux service code depends on another class property (owned by the aux service) in the configuration, that will be invoked via Configuration.getClass(), and it will still use the system classloader to load that class. Then it's very likely that you'll get a ClassNotFoundException.
The thread context classloader represents another similar problem. The moment the aux service code hits a code path that does Class.forName() that loads classes via the thread context classloader, and it needs to load an aux service-related class (that is not present in the main NM classpath), you will get a ClassNotFoundException.
If you look at the existing uses of the ApplicationClassLoader, you'll see we usually try to demarcate the code regions that need to run under the ApplicationClassLoader, and set and unset both the Configuration classloader and the thread context classloader. You might need to do the same thing with executing the aux service code. Luckily, I believe there are well-defined entry points and exit points for the aux service code, so hopefully it is not too difficult to do it completely.
(3) configuration property
The configuration property "...classloader.location" doesn't seem quite natural. It is really about the classpath. How about simply "...classpath" instead?
Also, it would be good to document that this is a comma-separated classpath somewhere.
(4) unit test
l.366: I'm quite confused by the comment and the code. If I'm reading this correctly, it is setting the classname configuration property with the TEST_DIR location, which is pretty much a bogus value. So it's understandable this won't work. But it's not really setting the classpath to a different location. Does it verify or confirm anything about this feature?
l.383: should we delete TEST_DIR here? It seems like the directory is deleted in another unit test. I don't think it's created again for each of the unit tests. It doesn't seem like it's necessary to delete this directory at all here. Did I miss something?
Overall, you might want to take a look at the TestApplicationClassLoader test to see how you can formulate the test. Note that all org.apache.hadoop.* classes are considered system classes and are exempt from the loading from the ApplicationClassLoader, so playing with the system classes is needed to test this (unless you can create test classes outside org.apache.hadoop).