[HADOOP-12747] support wildcard in libjars argument

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.9.0, 3.0.0-alpha1
• Component/s: util
• Target Version/s: 2.9.0
• Hadoop Flags: Reviewed
• Release Note: It is now possible to specify multiple jar files for the libjars argument using a wildcard. For example, you can specify "-libjars 'libs/*'" as a shorthand for all jars in the libs directory.
• Assignee: Sangjin Lee
• Reporter: Sangjin Lee
• Created: 28/Jan/16 18:46
• Updated: 29/Jul/19 15:47
• Resolved: 09/Aug/16 01:02

Description

There is a problem when a user job adds too many dependency jars in their command line. The HADOOP_CLASSPATH part can be addressed, including using wildcards (*). But the same cannot be done with the -libjars argument. Today it takes only fully specified file paths.

We may want to consider supporting wildcards as a way to help users in this situation. The idea is to handle it the same way the JVM does it: * expands to the list of jars in that directory. It does not traverse into any child directory.

Also, it probably would be a good idea to do it only for libjars (i.e. don't do it for -files and -archives).

Issue Links

• causes: MAPREDUCE-7225 Fix broken current folder expansion during MR job start
• is related to: YARN-4958 The file localization process should allow for wildcards to reduce the application footprint in the state store