[HADOOP-8899] Classpath exceeds maximum OS limit

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1-win
• Fix Version/s: 1-win
• Target Version/s: 1-win
• Hadoop Flags: Reviewed
• Assignee: Ahmed El Baz
• Reporter: Ahmed El Baz
• Created: 08/Oct/12 20:44
• Updated: 24/Jan/13 19:00
• Resolved: 22/Oct/12 21:44

Description

We hit errors in TT due to long classpath value. One example is Oozie trying to start a hive job, and it fails with the following error:
java.io.IOException: Command exceeds the OS command length limit: 8192, command: "set CLASSPATH="….

The classpath includes Hadoop Core + Hive Jars which are in the distributed cache. This is causing the classpath to be too long giving the error above. A viable long term fix is to generate a temporary JAR file in the task cache directory which includes all elements in the long classpath, and then provide this as the "-classpath" argument for the JVM to be spawned. This is gated for Windows only.

Issue Links

• is part of: HADOOP-8645 Stabilize branch-1-win
• is related to: MAPREDUCE-4959 bundling classpath into jar manifest on Windows does not expand environment variables or wildcards
• relates to: YARN-316 YARN container launch may exceed maximum Windows command line length due to long classpath