Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-2942

Aggregated Log Files should be combined

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Won't Fix
    • Affects Version/s: 2.6.0
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Turning on log aggregation allows users to easily store container logs in HDFS and subsequently view them in the YARN web UIs from a central place. Currently, there is a separate log file for each Node Manager. This can be a problem for HDFS if you have a cluster with many nodes as you’ll slowly start accumulating many (possibly small) files per YARN application. The current “solution” for this problem is to configure YARN (actually the JHS) to automatically delete these files after some amount of time.

      We should improve this by compacting the per-node aggregated log files into one log file per application.

        Attachments

        1. CompactedAggregatedLogsProposal_v1.pdf
          93 kB
          Robert Kanter
        2. YARN-2942-preliminary.001.patch
          13 kB
          Robert Kanter
        3. CompactedAggregatedLogsProposal_v2.pdf
          113 kB
          Robert Kanter
        4. YARN-2942-preliminary.002.patch
          44 kB
          Robert Kanter
        5. YARN-2942.001.patch
          89 kB
          Robert Kanter
        6. YARN-2942.002.patch
          91 kB
          Robert Kanter
        7. YARN-2942.003.patch
          92 kB
          Robert Kanter
        8. CombinedAggregatedLogsProposal_v3.pdf
          138 kB
          Robert Kanter
        9. ConcatableAggregatedLogsProposal_v4.pdf
          191 kB
          Robert Kanter
        10. ConcatableAggregatedLogsProposal_v5.pdf
          195 kB
          Robert Kanter
        11. CombinedAggregatedLogsProposal_v6.pdf
          127 kB
          Robert Kanter
        12. CombinedAggregatedLogsProposal_v7.pdf
          153 kB
          Robert Kanter
        13. ConcatableAggregatedLogsProposal_v8.pdf
          124 kB
          Robert Kanter

          Issue Links

            Activity

              People

              • Assignee:
                rkanter Robert Kanter
                Reporter:
                rkanter Robert Kanter
              • Votes:
                0 Vote for this issue
                Watchers:
                36 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: