Currently, the creation of the S4R package involves unpackaging all dependencies, then putting them altogether in a single file along with the S4 application classes.
This is very inefficient (and therefore potentially long, resulting in long development cycles) because there is a lot of file copying involved. In addition it may be brittle due to name conflicts.
We should improve the process by:
- including external jars in the S4R archive without unpackaging them, typically in the /lib dir
- updating the classloading mechanism in S4. Ideas could be taken from Hadoop's RunJar class. We should avoid fancy stuff here, and the current multi-classloader is not really needed if we have a single app per S4 node.
- "shading" platform classes and dependencies, in order to avoid any conflict between S4 platform dependencies and application dependencies?