Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5773

Provide dedicated MRAppMaster syslog length limit

VotersStop watchingWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Blocker
    • Resolution: Fixed
    • 2.3.0
    • 2.4.0
    • mr-am
    • None
    • Reviewed

    Description

      Currently mapreduce.task.userlog.limit.kb controls both the length of task attempt logs and MA-AM attempt logs. Obviously, MR-AM log is not userlog.
      We are interested in keeping MR-AM log either in its entirety or in much larger size than task logs for debugging.

      MAPREDUCE-5672 introduced CRLA. We already use a dedicated setting for how many backups of rolled files we keep for MR-AM. However, for large jobs with tens of megabyte AM log, it means that you have a long series of files. A natural improvement is to have a configuration yarn.app.mapreduce.am.container.log.limit.kb . It allows to either disable rolling the log for altogether while keeping it for the task attempts, or to use much a larger limit to make rolling less frequent.

      Attachments

        1. MAPREDUCE-5773.v01.patch
          7 kB
          Gera Shegalov

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            jira.shegalov Gera Shegalov
            jira.shegalov Gera Shegalov
            Votes:
            0 Vote for this issue
            Watchers:
            3 Stop watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment