Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-2443 Handling logs of long-running services on YARN
  3. YARN-2548

Find a more scalable way to handle logs for long running service

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • nodemanager, resourcemanager
    • None

    Description

      After YARN-2468, the container logs will be aggregated separately based on the time. It will increate the total number of log files. It is fine for small cluster. But for the larger cluster, it will make too-many-files problem even worse.

      We need to find a more scalable way to handle those logs. Aggregate all container logs in a single file is an option, but we need to find a different way, other than TFile(do not support append), to do it.

      Attachments

        Issue Links

          is related to

          New Feature - A new feature of the product, which has yet to be developed. YARN-2942 Aggregated Log Files should be combined

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              xgong Xuan Gong
              xgong Xuan Gong
              Votes:
              1 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated: