Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6415

Create a tool to combine aggregated logs into HAR files

          Details

          • Type: New Feature
          • Status: Resolved
          • Priority: Major
          • Resolution: Fixed
          • Affects Version/s: 2.8.0
          • Fix Version/s: 2.8.0, 3.0.0-alpha1
          • Component/s: None
          • Labels:
            None
          • Target Version/s:
          • Hadoop Flags:
            Reviewed

            Description

            While we wait for YARN-2942 to become viable, it would still be great to improve the aggregated logs problem. We can write a tool that combines aggregated log files into a single HAR file per application, which should solve the too many files and too many blocks problems. See the design document for details.

            See YARN-2942 for more context.

              Attachments

              1. MAPREDUCE-6415.003.patch
                49 kB
                Robert Kanter
              2. MAPREDUCE-6415.002.patch
                50 kB
                Robert Kanter
              3. MAPREDUCE-6415.002.patch
                50 kB
                Robert Kanter
              4. MAPREDUCE-6415.001.patch
                48 kB
                Robert Kanter
              5. MAPREDUCE-6415_prelim_002.patch
                32 kB
                Robert Kanter
              6. MAPREDUCE-6415_prelim_001.patch
                25 kB
                Robert Kanter
              7. MAPREDUCE-6415_branch-2.003.patch
                50 kB
                Robert Kanter
              8. MAPREDUCE-6415_branch-2.002.patch
                50 kB
                Robert Kanter
              9. MAPREDUCE-6415_branch-2.001.patch
                48 kB
                Robert Kanter
              10. MAPREDUCE-6415_branch-2_prelim_002.patch
                32 kB
                Robert Kanter
              11. MAPREDUCE-6415_branch-2_prelim_001.patch
                25 kB
                Robert Kanter
              12. HAR-ableAggregatedLogs_v1.pdf
                111 kB
                Robert Kanter

                Issue Links

                depends upon

                Improvement - An improvement or enhancement to an existing feature or task. YARN-3950 Add unique YARN_SHELL_ID environment variable to DistributedShell

                • Major - Major loss of function.
                • Resolved
                relates to

                Bug - A problem which impairs or prevents the functions of the product. MAPREDUCE-7027 HadoopArchiveLogs shouldn't delete the original logs if the HAR creation fails

                • Critical - Crashes, loss of data, severe memory leak.
                • Resolved

                Bug - A problem which impairs or prevents the functions of the product. MAPREDUCE-6480 archive-logs tool may miss applications

                • Major - Major loss of function.
                • Resolved

                Bug - A problem which impairs or prevents the functions of the product. MAPREDUCE-6494 Permission issue when running archive-logs tool as different users

                • Major - Major loss of function.
                • Resolved

                Bug - A problem which impairs or prevents the functions of the product. MAPREDUCE-6495 Docs for archive-logs tool

                • Major - Major loss of function.
                • Resolved

                Bug - A problem which impairs or prevents the functions of the product. MAPREDUCE-6503 archive-logs tool should use HADOOP_PREFIX instead of HADOOP_HOME

                • Major - Major loss of function.
                • Resolved

                Bug - A problem which impairs or prevents the functions of the product. MAPREDUCE-6550 archive-logs tool changes log ownership to the Yarn user when using DefaultContainerExecutor

                • Major - Major loss of function.
                • Resolved

                Bug - A problem which impairs or prevents the functions of the product. MAPREDUCE-7023 TestHadoopArchiveLogs.testCheckFilesAndSeedApps fails on rerun

                • Minor - Minor loss of function, or other problem where easy workaround is present.
                • Resolved

                New Feature - A new feature of the product, which has yet to be developed. YARN-2942 Aggregated Log Files should be combined

                • Major - Major loss of function.
                • Resolved

                Improvement - An improvement or enhancement to an existing feature or task. MAPREDUCE-6970 archive-logs tool should throttle container requests

                • Major - Major loss of function.
                • Open

                Improvement - An improvement or enhancement to an existing feature or task. YARN-4086 Allow Aggregated Log readers to handle HAR files

                • Major - Major loss of function.
                • Resolved

                Improvement - An improvement or enhancement to an existing feature or task. YARN-4946 RM should not consider an application as COMPLETED when log aggregation is not in a terminal state

                • Major - Major loss of function.
                • Resolved

                  Activity

                    People

                    • Assignee:
                      rkanter Robert Kanter
                      Reporter:
                      rkanter Robert Kanter
                    • Votes:
                      1 Vote for this issue
                      Watchers:
                      18 Start watching this issue

                      Dates

                      • Created:
                        Updated:
                        Resolved: