Hadoop Common
  1. Hadoop Common
  2. HADOOP-3104

MultithreadMapRunner keeps consuming records even if trheads are not available

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 0.16.1
    • Fix Version/s: 0.16.2
    • Component/s: None
    • Labels:
      None
    • Environment:

      all

      Description

      The ExecutorService execute() method does not block when there are not threads available, it queues up the runnables until there are threads.

      The problem is that all key/values are read and kept in memory for the task, with large datasets this will create a OOM exception.

      Have to figure out how to use the execute in blocking fashion.

      1. patch3104.txt
        7 kB
        Alejandro Abdelnur
      2. patch3104.txt
        5 kB
        Alejandro Abdelnur
      3. patch3104-2.txt
        7 kB
        Owen O'Malley
      4. patch3104-3.txt
        7 kB
        Owen O'Malley

        Activity

        No work has yet been logged on this issue.

          People

          • Assignee:
            Alejandro Abdelnur
            Reporter:
            Alejandro Abdelnur
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development