Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-5193

For long running services, aggregate logs when a container completes instead of when the app completes

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Won't Fix
    • None
    • None
    • None
    • None

    Description

      For a long running service, containers will typically not complete very often. However, when a container completes - it would be useful to aggregate the logs right then, instead of waiting for the app to complete.
      This will allow the command line log tool to lookup containers for an app from the log file index itself, instead of having to go and talk to YARN. Talking to YARN really only works if ATS is enabled, and YARN is configured to publish container information to ATS (That may not always be the case - since this can overload ATS quite fast).

      There's some added benefits like cleaning out local disk space early, instead of waiting till the app completes. (There's probably a separate jira somewhere about cleanup of container for long running services anyway)

      cc Vinod Kumar Vavilapalli, Xuan Gong

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            sseth Siddharth Seth
            Votes:
            1 Vote for this issue
            Watchers:
            11 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment