[MAPREDUCE-4322] Fix command-line length abort issues on Windows

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1-win
• Component/s: tasktracker

Environment

Windows, downstream applications with long aggregate classpaths

• Assignee: Ivan Mitic
• Reporter: John Gordon
• Created: 06/Jun/12 21:42
• Updated: 05/Jul/12 07:03
• Resolved: 05/Jul/12 07:03

Description

When a task is started on the tasktracker, it creates a small batch file to invoke java and runs that batch. Within the batch file, the invocation of Java currently has -classpath ${CLASSPATH} inline to the command. That line often exceeds 8000 characters. This is ok for most linux distributions because the line limit env variable is often set much higher than this. However, for Windows this cause cmd to abort execution. This surfaces in Hadoop as an unknown failure mode for the task.

I think the easiest and most natural way to fix this is to push the -classpath option into a config file to take the longest variable part of the line and put it somewhere that scales better.

Issue Links

• relates to: HADOOP-8079 Proposal for enhancements to Hadoop for Windows Server and Windows Azure development and runtime environments