[HADOOP-1864] Support for big jar file (>2G)

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Won't Fix
• Affects Version/s: 0.14.1
• Labels: pull-request-available
• Assignee: Unassigned
• Reporter: Yiping Han
• Created: 08/Sep/07 00:21
• Updated: 23/Feb/23 18:24
• Resolved: 25/Oct/07 22:23

Description

We have huge size binary that need to be distributed onto tasktracker nodes in Hadoop streaming mode. We've tried both -file option and -cacheArchive option. It seems the tasktracker node cannot unjar jar files bigger than 2G. We are considering split our binaries into multiple jars, but with -file, it seems we cannot do it. Also, we would prefer -cacheArchive option for performance issue, but it seems -cacheArchive does not allow more than appearance in the streaming options. Even if -cacheArchive support multiple jars, we still need a way to put the jars into a single directory tree, instead of using multiple symbolic links.

So, in general, we need a feasible and efficient way to update large size (>2G) binaries for Hadoop streaming. Don't know if there is an existing solution that we either didn't find or took it wrong. Or there should be some extra work to provide a solution?

Issue Links

• relates to: HADOOP-2089 Multiple caheArchive does not work in Hadoop streaming
• links to: GitHub Pull Request #5429