Hadoop Common
  1. Hadoop Common
  2. HADOOP-3280

virtual address space limits break streaming apps

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.17.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      This patch adds the mapred.child.ulimit to limit the virtual memory for children processes to the given value.

      Description

      HADOOP-2765 added a mandatory, hard virtual address space limit to streaming apps based on the Java process's -Xmx setting.

      This makes it impossible to run a 64-bit streaming app that needs large address spaces under a 32-bit JVM, even if one is otherwise willing to dramatically increase the -Xmx setting without cause. Also, unlike Java's -Xmx limit, the virtual address space limit for an arbitrary UNIX process does not necessarily correspond to RAM usage, so it's likely to be a relatively difficult to configure limit.

      2765 was originally opened to allow an optional wrapper script around streaming tasks, one use case for which was setting a ulimit. That approach seems much less intrusive and more flexible than the final implementation. The ulimit can also be trivially set by the streaming task itself without any support from Hadoop.

      Marking this as an 0.17 blocker because it will break deployed apps and there is no workaround available.

      1. patch-3280-3.txt
        18 kB
        Amareshwari Sriramadasu
      2. patch-3280-2.txt
        18 kB
        Amareshwari Sriramadasu
      3. patch-3280-1.txt
        18 kB
        Amareshwari Sriramadasu
      4. patch-3280.txt
        14 kB
        Amareshwari Sriramadasu
      5. patch-3280.txt
        15 kB
        Amareshwari Sriramadasu
      6. HADOOP-3280_3_20080425.patch
        18 kB
        Arun C Murthy
      7. HADOOP-3280_2_20080424.patch
        18 kB
        Arun C Murthy
      8. HADOOP-3280_1_20080423.patch
        7 kB
        Arun C Murthy
      9. HADOOP-3280_0_20080418.patch
        2 kB
        Arun C Murthy

        Activity

        Owen O'Malley made changes -
        Component/s contrib/streaming [ 12310972 ]
        Nigel Daley made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Owen O'Malley made changes -
        Hadoop Flags [Incompatible change, Reviewed]
        Release Note This patch adds the mapred.child.ulimit to limit the virtual memory for children processes to the given value.
        Resolution Fixed [ 1 ]
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Arun C Murthy made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Arun C Murthy made changes -
        Attachment HADOOP-3280_3_20080425.patch [ 12380972 ]
        Arun C Murthy made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Amareshwari Sriramadasu made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Amareshwari Sriramadasu made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Amareshwari Sriramadasu made changes -
        Attachment patch-3280-3.txt [ 12380895 ]
        Amareshwari Sriramadasu made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Amareshwari Sriramadasu made changes -
        Attachment patch-3280-2.txt [ 12380894 ]
        Amareshwari Sriramadasu made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Arun C Murthy made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Arun C Murthy made changes -
        Attachment HADOOP-3280_2_20080424.patch [ 12380888 ]
        Arun C Murthy made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Amareshwari Sriramadasu made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Amareshwari Sriramadasu made changes -
        Attachment patch-3280-1.txt [ 12380834 ]
        Arun C Murthy made changes -
        Attachment HADOOP-3280_1_20080423.patch [ 12380822 ]
        Arun C Murthy made changes -
        Assignee Amareshwari Sriramadasu [ amareshwari ] Arun C Murthy [ acmurthy ]
        Owen O'Malley made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Amareshwari Sriramadasu made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Amareshwari Sriramadasu made changes -
        Attachment patch-3280.txt [ 12380688 ]
        Amareshwari Sriramadasu made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Amareshwari Sriramadasu made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Amareshwari Sriramadasu made changes -
        Attachment patch-3280.txt [ 12380685 ]
        Amareshwari Sriramadasu made changes -
        Assignee Amareshwari Sriramadasu [ amareshwari ]
        Arun C Murthy made changes -
        Field Original Value New Value
        Attachment HADOOP-3280_0_20080418.patch [ 12380529 ]
        Rick Cox created issue -

          People

          • Assignee:
            Arun C Murthy
            Reporter:
            Rick Cox
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development