Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1648

Use rolling to limit tasklogs

Log workAgile BoardRank to TopRank to BottomAttach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Won't Fix
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: tasktracker
    • Labels:

      Description

      There are at least two types of task-logs: syslog and stdlog

      Task-Jvm outputs syslog by log4j with TaskLogAppender, TaskLogAppender looks just like "tail -c", it stores last N byte/line logs in memory(via queue), and do real output only if all logs is commit and Appender is going to close.

      The common problem of TaskLogAppender and 'tail -c' is keep everything in memory and user can't see any log output while task is in progress.

      So I'm going to try RollingFileAppender instead of TaskLogAppender, use MaxFileSize&MaxBackupIndex to limit log file size.

      RollingFileAppender is also suitable for stdout/stderr, just redirect stdout/stderr to log4j via LoggingOutputStream, no client code have to be changed, and RollingFileAppender seems better than 'tail -c' too.

        Attachments

        Issue Links

          Activity

          $i18n.getText('security.level.explanation', $currentSelection) Viewable by All Users
          Cancel

            People

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment