Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5773

Provide dedicated MRAppMaster syslog length limit

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 2.3.0
    • Fix Version/s: 2.4.0
    • Component/s: mr-am
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      Currently mapreduce.task.userlog.limit.kb controls both the length of task attempt logs and MA-AM attempt logs. Obviously, MR-AM log is not userlog.
      We are interested in keeping MR-AM log either in its entirety or in much larger size than task logs for debugging.

      MAPREDUCE-5672 introduced CRLA. We already use a dedicated setting for how many backups of rolled files we keep for MR-AM. However, for large jobs with tens of megabyte AM log, it means that you have a long series of files. A natural improvement is to have a configuration yarn.app.mapreduce.am.container.log.limit.kb . It allows to either disable rolling the log for altogether while keeping it for the task attempts, or to use much a larger limit to make rolling less frequent.

        Attachments

        1. MAPREDUCE-5773.v01.patch
          7 kB
          Gera Shegalov

          Activity

            People

            • Assignee:
              jira.shegalov Gera Shegalov
              Reporter:
              jira.shegalov Gera Shegalov
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: