Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1943

Implement limits on per-job JobConf, Counters, StatusReport, Split-Sizes

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.20.203.0
    • Component/s: None
    • Labels:
      None

      Description

      We have come across issues in production clusters wherein users abuse counters, statusreport messages and split sizes. One such case was when one of the users had 100 million counters. This leads to jobtracker going out of memory and being unresponsive. In this jira I am proposing to put sane limits on the status report length, the number of counters and the size of block locations returned by the input split.

      1. MAPREDUCE-1943-0.20-yahoo.patch
        14 kB
        Mahadev konar
      2. MAPREDUCE-1943-0.20-yahoo.patch
        11 kB
        Mahadev konar
      3. MAPREDUCE-1943-yahoo-hadoop-0.20S.patch
        23 kB
        Mahadev konar
      4. MAPREDUCE-1943-yahoo-hadoop-0.20S-fix.patch
        18 kB
        Mahadev konar
      5. MAPREDUCE-1943-yahoo-hadoop-0.20S-fix.patch
        18 kB
        Mahadev konar
      6. MAPREDUCE-1943-yahoo-hadoop-0.20S-fix.patch
        20 kB
        Mahadev konar
      7. MAPREDUCE-1943-yahoo-hadoop-0.20S-fix.patch
        13 kB
        Mahadev konar
      8. MAPREDUCE-1943-yahoo-hadoop-0.20S-fix.patch
        6 kB
        Mahadev konar
      9. MAPREDUCE-1943-yahoo-hadoop-0.20S-fix.patch
        4 kB
        Mahadev konar

        Issue Links

          Activity

          Mahadev konar created issue -
          Mahadev konar made changes -
          Field Original Value New Value
          Fix Version/s 0.22.0 [ 12314184 ]
          Mahadev konar made changes -
          Attachment MAPREDUCE-1521-0.20-yahoo.patch [ 12449495 ]
          Mahadev konar made changes -
          Attachment MAPREDUCE-1521-0.20-yahoo.patch [ 12449495 ]
          Mahadev konar made changes -
          Attachment MAPREDUCE-1943-0.20-yahoo.patch [ 12449496 ]
          Mahadev konar made changes -
          Attachment MAPREDUCE-1943-0.20-yahoo.patch [ 12449699 ]
          Mahadev konar made changes -
          Attachment MAPREDUCE-1943-yahoo-hadoop-0.20S.patch [ 12449985 ]
          Mahadev konar made changes -
          Fix Version/s 0.22.0 [ 12314184 ]
          Mahadev konar made changes -
          Mahadev konar made changes -
          Mahadev konar made changes -
          Mahadev konar made changes -
          Mahadev konar made changes -
          Attachment MAPREDUCE-1943-yahoo-hadoop-0.20S-fix.patch [ 12454319 ]
          Mahadev konar made changes -
          Attachment MAPREDUCE-1943-yahoo-hadoop-0.20S-fix.patch [ 12454319 ]
          Mahadev konar made changes -
          Mahadev konar made changes -
          Nigel Daley made changes -
          Fix Version/s 0.22.0 [ 12314184 ]
          Tom White made changes -
          Fix Version/s 0.20.203.0 [ 12316151 ]
          Owen O'Malley made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Owen O'Malley made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Arun C Murthy made changes -
          Link This issue is cloned as MAPREDUCE-3469 [ MAPREDUCE-3469 ]
          Sangjin Lee made changes -
          Link This issue relates to MAPREDUCE-5186 [ MAPREDUCE-5186 ]

            People

            • Assignee:
              Mahadev konar
              Reporter:
              Mahadev konar
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development