Issue Details (XML | Word | Printable)

Key: HADOOP-3280
Type: Bug Bug
Status: Closed Closed
Resolution: Fixed
Priority: Blocker Blocker
Assignee: Arun C Murthy
Reporter: Rick Cox
Votes: 0
Watchers: 2
Operations

If you were logged in you would be able to see more operations.
Hadoop Common

virtual address space limits break streaming apps

Created: 18/Apr/08 05:07 PM   Updated: 08/Jul/09 05:05 PM
Return to search
Component/s: None
Affects Version/s: None
Fix Version/s: 0.17.0

Time Tracking:
Not Specified

File Attachments:
  File Name Date Attached Attached By Size
Text File Licensed for inclusion in ASF works HADOOP-3280_0_20080418.patch 2008-04-18 06:35 PM Arun C Murthy 2 kB
Text File Licensed for inclusion in ASF works HADOOP-3280_1_20080423.patch 2008-04-24 03:49 AM Arun C Murthy 7 kB
Text File Licensed for inclusion in ASF works HADOOP-3280_2_20080424.patch 2008-04-25 02:22 AM Arun C Murthy 18 kB
Text File Licensed for inclusion in ASF works HADOOP-3280_3_20080425.patch 2008-04-25 10:23 PM Arun C Murthy 18 kB
Text File Licensed for inclusion in ASF works patch-3280-1.txt 2008-04-24 11:41 AM Amareshwari Sriramadasu 18 kB
Text File Licensed for inclusion in ASF works patch-3280-2.txt 2008-04-25 05:30 AM Amareshwari Sriramadasu 18 kB
Text File Licensed for inclusion in ASF works patch-3280-3.txt 2008-04-25 05:43 AM Amareshwari Sriramadasu 18 kB
Text File Licensed for inclusion in ASF works patch-3280.txt 2008-04-22 12:44 PM Amareshwari Sriramadasu 15 kB
Text File Licensed for inclusion in ASF works patch-3280.txt 2008-04-22 11:44 AM Amareshwari Sriramadasu 14 kB

Hadoop Flags: Reviewed, Incompatible change
Release Note: This patch adds the mapred.child.ulimit to limit the virtual memory for children processes to the given value.
Resolution Date: 29/Apr/08 10:46 PM


 Description  « Hide
HADOOP-2765 added a mandatory, hard virtual address space limit to streaming apps based on the Java process's -Xmx setting.

This makes it impossible to run a 64-bit streaming app that needs large address spaces under a 32-bit JVM, even if one is otherwise willing to dramatically increase the -Xmx setting without cause. Also, unlike Java's -Xmx limit, the virtual address space limit for an arbitrary UNIX process does not necessarily correspond to RAM usage, so it's likely to be a relatively difficult to configure limit.

2765 was originally opened to allow an optional wrapper script around streaming tasks, one use case for which was setting a ulimit. That approach seems much less intrusive and more flexible than the final implementation. The ulimit can also be trivially set by the streaming task itself without any support from Hadoop.

Marking this as an 0.17 blocker because it will break deployed apps and there is no workaround available.

 All   Comments   Work Log   Change History   Subversion Commits      Sort Order: Ascending order - Click to sort in descending order
Arun C Murthy made changes - 18/Apr/08 06:35 PM
Field Original Value New Value
Attachment HADOOP-3280_0_20080418.patch [ 12380529 ]
Amareshwari Sriramadasu made changes - 22/Apr/08 05:07 AM
Assignee Amareshwari Sriramadasu [ amareshwari ]
Amareshwari Sriramadasu made changes - 22/Apr/08 11:44 AM
Attachment patch-3280.txt [ 12380685 ]
Amareshwari Sriramadasu made changes - 22/Apr/08 11:44 AM
Status Open [ 1 ] Patch Available [ 10002 ]
Amareshwari Sriramadasu made changes - 22/Apr/08 12:44 PM
Status Patch Available [ 10002 ] Open [ 1 ]
Amareshwari Sriramadasu made changes - 22/Apr/08 12:44 PM
Attachment patch-3280.txt [ 12380688 ]
Amareshwari Sriramadasu made changes - 22/Apr/08 12:45 PM
Status Open [ 1 ] Patch Available [ 10002 ]
Owen O'Malley made changes - 22/Apr/08 06:50 PM
Status Patch Available [ 10002 ] Open [ 1 ]
Arun C Murthy made changes - 24/Apr/08 03:28 AM
Assignee Amareshwari Sriramadasu [ amareshwari ] Arun C Murthy [ acmurthy ]
Arun C Murthy made changes - 24/Apr/08 03:49 AM
Attachment HADOOP-3280_1_20080423.patch [ 12380822 ]
Amareshwari Sriramadasu made changes - 24/Apr/08 11:41 AM
Attachment patch-3280-1.txt [ 12380834 ]
Amareshwari Sriramadasu made changes - 24/Apr/08 11:41 AM
Status Open [ 1 ] Patch Available [ 10002 ]
Arun C Murthy made changes - 25/Apr/08 02:19 AM
Status Patch Available [ 10002 ] Open [ 1 ]
Arun C Murthy made changes - 25/Apr/08 02:22 AM
Attachment HADOOP-3280_2_20080424.patch [ 12380888 ]
Arun C Murthy made changes - 25/Apr/08 02:22 AM
Status Open [ 1 ] Patch Available [ 10002 ]
Amareshwari Sriramadasu made changes - 25/Apr/08 05:28 AM
Status Patch Available [ 10002 ] Open [ 1 ]
Amareshwari Sriramadasu made changes - 25/Apr/08 05:30 AM
Attachment patch-3280-2.txt [ 12380894 ]
Amareshwari Sriramadasu made changes - 25/Apr/08 05:31 AM
Status Open [ 1 ] Patch Available [ 10002 ]
Amareshwari Sriramadasu made changes - 25/Apr/08 05:43 AM
Attachment patch-3280-3.txt [ 12380895 ]
Amareshwari Sriramadasu made changes - 25/Apr/08 05:44 AM
Status Patch Available [ 10002 ] Open [ 1 ]
Amareshwari Sriramadasu made changes - 25/Apr/08 05:45 AM
Status Open [ 1 ] Patch Available [ 10002 ]
Arun C Murthy made changes - 25/Apr/08 10:23 PM
Status Patch Available [ 10002 ] Open [ 1 ]
Arun C Murthy made changes - 25/Apr/08 10:23 PM
Attachment HADOOP-3280_3_20080425.patch [ 12380972 ]
Arun C Murthy made changes - 25/Apr/08 10:23 PM
Status Open [ 1 ] Patch Available [ 10002 ]
Owen O'Malley made changes - 29/Apr/08 10:46 PM
Status Patch Available [ 10002 ] Resolved [ 5 ]
Resolution Fixed [ 1 ]
Release Note This patch adds the mapred.child.ulimit to limit the virtual memory for children processes to the given value.
Hadoop Flags [Incompatible change, Reviewed]
Nigel Daley made changes - 21/May/08 08:06 PM
Status Resolved [ 5 ] Closed [ 6 ]
Owen O'Malley made changes - 08/Jul/09 05:05 PM
Component/s contrib/streaming [ 12310972 ]

Powered by a free Atlassian JIRA open source license for Apache Software Foundation. Try JIRA - bug tracking software for your team.
Atlassian JIRA the Professional Issue Tracker. (Enterprise Edition, Version: 3.13.2-#335) - Bug/feature request - Atlassian news - Contact Administrators