Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-431 [Umbrella] Complete/Stabilize YARN application log-handling
  3. YARN-85

Allow per job log aggregation configuration

        Details

        • Type: Sub-task
        • Status: Open
        • Priority: Critical
        • Resolution: Unresolved
        • Affects Version/s: None
        • Fix Version/s: None
        • Component/s: nodemanager
        • Labels:
          None

          Description

          Currently, if log aggregation is enabled for a cluster - logs for all jobs will be aggregated - leading to a whole bunch of files on hdfs which users may not want.
          Users should be able to control this along with the aggregation policy - failed only, all, etc.

            Issue Links

            blocks

            Bug - A problem which impairs or prevents the functions of the product. YARN-386 [Umbrella] YARN API Changes

            • Major - Major loss of function.
            • Open
            is related to

            Sub-task - The sub-task of the issue YARN-221 NM should provide a way for AM to tell it not to aggregate logs.

            • Major - Major loss of function.
            • Resolved

            Bug - A problem which impairs or prevents the functions of the product. MAPREDUCE-3143 Complete aggregation of user-logs spit out by containers onto DFS

            • Major - Major loss of function.
            • Closed

              Activity

              Ascending order - Click to sort in descending order
              Hide
              Permalink
              vinodkv Vinod Kumar Vavilapalli added a comment -

              +1, the infrastructure for this is all there, including tests IIRC, we just need to expose it to the users.

              Show
              vinodkv Vinod Kumar Vavilapalli added a comment - +1, the infrastructure for this is all there, including tests IIRC, we just need to expose it to the users.
              Hide
              Permalink
              sseth Siddharth Seth added a comment -

              In the initial patch, am planning to only support per job enable/disable aggregation. Along with this - an option to keep the logs on the NM for some time if the job disables aggregation.
              Something like FAILED_ONLY will not work very well rightnow - since container exit status is not necessarily an indication of whether a task completes successfully or not.

              Show
              sseth Siddharth Seth added a comment - In the initial patch, am planning to only support per job enable/disable aggregation. Along with this - an option to keep the logs on the NM for some time if the job disables aggregation. Something like FAILED_ONLY will not work very well rightnow - since container exit status is not necessarily an indication of whether a task completes successfully or not.
              Hide
              Permalink
              revans2 Robert Joseph Evans added a comment -

              Once YARN-221 goes in we should be able to expose a lot more functionality, like aggregate only a sampling of successful jobs, aggregate only failed jobs, etc.

              Show
              revans2 Robert Joseph Evans added a comment - Once YARN-221 goes in we should be able to expose a lot more functionality, like aggregate only a sampling of successful jobs, aggregate only failed jobs, etc.
              Hide
              Permalink
              ctrezzo Chris Trezzo added a comment -

              I have submitted a patch on YARN-221 and I am also starting to work on the MapReduce level changes to provide configuration for log aggregation on a per job basis. I will hopefully post a patch soon.

              Show
              ctrezzo Chris Trezzo added a comment - I have submitted a patch on YARN-221 and I am also starting to work on the MapReduce level changes to provide configuration for log aggregation on a per job basis. I will hopefully post a patch soon.
              Hide
              Permalink
              lohit Lohit Vijayarenu added a comment -

              Patch looks good to me. Can anyone else also take a look at the patch.

              Show
              lohit Lohit Vijayarenu added a comment - Patch looks good to me. Can anyone else also take a look at the patch.
              Hide
              Permalink
              mingma Ming Ma added a comment -

              Regarding Seth's comment of "container exit status is not necessarily an indication of whether a task completes successfully or not", https://issues.apache.org/jira/browse/MAPREDUCE-5465 should fix the issue.

              Show
              mingma Ming Ma added a comment - Regarding Seth's comment of "container exit status is not necessarily an indication of whether a task completes successfully or not", https://issues.apache.org/jira/browse/MAPREDUCE-5465 should fix the issue.

                People

                • Assignee:
                  Unassigned
                  Reporter:
                  sseth Siddharth Seth
                • Votes:
                  1 Vote for this issue
                  Watchers:
                  9 Start watching this issue

                  Dates

                  • Created:
                    Updated:

                    Development