Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1943

Implement limits on per-job JobConf, Counters, StatusReport, Split-Sizes

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.20.203.0
    • None
    • None

    Description

      We have come across issues in production clusters wherein users abuse counters, statusreport messages and split sizes. One such case was when one of the users had 100 million counters. This leads to jobtracker going out of memory and being unresponsive. In this jira I am proposing to put sane limits on the status report length, the number of counters and the size of block locations returned by the input split.

      Attachments

        1. MAPREDUCE-1943-0.20-yahoo.patch
          11 kB
          Mahadev Konar
        2. MAPREDUCE-1943-0.20-yahoo.patch
          14 kB
          Mahadev Konar
        3. MAPREDUCE-1943-yahoo-hadoop-0.20S.patch
          23 kB
          Mahadev Konar
        4. MAPREDUCE-1943-yahoo-hadoop-0.20S-fix.patch
          4 kB
          Mahadev Konar
        5. MAPREDUCE-1943-yahoo-hadoop-0.20S-fix.patch
          6 kB
          Mahadev Konar
        6. MAPREDUCE-1943-yahoo-hadoop-0.20S-fix.patch
          13 kB
          Mahadev Konar
        7. MAPREDUCE-1943-yahoo-hadoop-0.20S-fix.patch
          20 kB
          Mahadev Konar
        8. MAPREDUCE-1943-yahoo-hadoop-0.20S-fix.patch
          18 kB
          Mahadev Konar
        9. MAPREDUCE-1943-yahoo-hadoop-0.20S-fix.patch
          18 kB
          Mahadev Konar

        Issue Links

          is cloned by

          Improvement - An improvement or enhancement to an existing feature or task. MAPREDUCE-3469 Port to 0.22 - Implement limits on per-job JobConf, Counters, StatusReport, Split-Sizes

          • Major - Major loss of function.
          • Resolved
          relates to

          Bug - A problem which impairs or prevents the functions of the product. MAPREDUCE-5186 mapreduce.job.max.split.locations causes some splits created by CombineFileInputFormat to fail

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Activity

            People

              mahadev Mahadev Konar
              mahadev Mahadev Konar
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: