Details
-
Bug
-
Status: Reopened
-
Major
-
Resolution: Fixed
-
None
-
None
-
None
Description
Samza team could also support a problem for long classpath by implementing something that would reduce the classpath length.
Some potential ideas to fix:
- Change the generated classpath (done in Samza's bash scripts) to be a list of `<directory>/*jar` entries instead of a list of every individual jar. Wildcards in the classpath are supported in jdk >= 6.
This should be a relatively simple change and can be implemented by getting the parent directory of all jar files into a list, sorting and making the list unique, then append a wildcard to the end to pick up all jars under each directory in the list (ask Jon Bringhurst if you have any questions).
See https://github.com/apache/samza/blob/92ae4c628abb3d113520ec47ca82f08c480123ad/samza-shell/src/main/bash/run-class.sh#L63 for where the code change for this should probably be. - Use the manifest in a special "uber-jar" to list all dependencies in the classpath. Then just include that one "uber-jar" in the classpath on the cli. This would probably be tough to do, and seems like it would be annoying to manage, but apparently IntelliJ uses this strategy.
Note that this is not an issue with the number of arguments (argc limit, which is based on a multiple of the ulimit -s stack size), it is a problem with the length of a single argument (strlen(*argv) limit based on MAX_ARG_STRLEN, which is a multiple of the PAGE_SIZE). So, the only way to get around this limit is to recompile the kernel with a different page size (which obviously isn't practical).
