Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6415

Create a tool to combine aggregated logs into HAR files

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.8.0
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      While we wait for YARN-2942 to become viable, it would still be great to improve the aggregated logs problem. We can write a tool that combines aggregated log files into a single HAR file per application, which should solve the too many files and too many blocks problems. See the design document for details.

      See YARN-2942 for more context.

        Attachments

        1. MAPREDUCE-6415.003.patch
          49 kB
          Robert Kanter
        2. MAPREDUCE-6415.002.patch
          50 kB
          Robert Kanter
        3. MAPREDUCE-6415.002.patch
          50 kB
          Robert Kanter
        4. MAPREDUCE-6415.001.patch
          48 kB
          Robert Kanter
        5. MAPREDUCE-6415_prelim_002.patch
          32 kB
          Robert Kanter
        6. MAPREDUCE-6415_prelim_001.patch
          25 kB
          Robert Kanter
        7. MAPREDUCE-6415_branch-2.003.patch
          50 kB
          Robert Kanter
        8. MAPREDUCE-6415_branch-2.002.patch
          50 kB
          Robert Kanter
        9. MAPREDUCE-6415_branch-2.001.patch
          48 kB
          Robert Kanter
        10. MAPREDUCE-6415_branch-2_prelim_002.patch
          32 kB
          Robert Kanter
        11. MAPREDUCE-6415_branch-2_prelim_001.patch
          25 kB
          Robert Kanter
        12. HAR-ableAggregatedLogs_v1.pdf
          111 kB
          Robert Kanter

          Issue Links

          depends upon

          Improvement - An improvement or enhancement to an existing feature or task. YARN-3950 Add unique YARN_SHELL_ID environment variable to DistributedShell

          • Major - Major loss of function.
          • Resolved
          relates to

          Bug - A problem which impairs or prevents the functions of the product. MAPREDUCE-7236 HadoopArchiveLogs will use token to create proxy user when kerberos on

          • Major - Major loss of function.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. MAPREDUCE-7027 HadoopArchiveLogs shouldn't delete the original logs if the HAR creation fails

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. MAPREDUCE-6480 archive-logs tool may miss applications

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. MAPREDUCE-6494 Permission issue when running archive-logs tool as different users

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. MAPREDUCE-6495 Docs for archive-logs tool

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. MAPREDUCE-6503 archive-logs tool should use HADOOP_PREFIX instead of HADOOP_HOME

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. MAPREDUCE-6550 archive-logs tool changes log ownership to the Yarn user when using DefaultContainerExecutor

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. MAPREDUCE-7023 TestHadoopArchiveLogs.testCheckFilesAndSeedApps fails on rerun

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. MAPREDUCE-7235 NPE in HadoopArchiveLogs#filterAppsByAggregatedStatus

          • Major - Major loss of function.
          • Patch Available

          New Feature - A new feature of the product, which has yet to be developed. YARN-2942 Aggregated Log Files should be combined

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. MAPREDUCE-6970 archive-logs tool should throttle container requests

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. YARN-4086 Allow Aggregated Log readers to handle HAR files

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. YARN-4946 RM should not consider an application as COMPLETED when log aggregation is not in a terminal state

          • Major - Major loss of function.
          • Resolved

            Activity

              People

              • Assignee:
                rkanter Robert Kanter
                Reporter:
                rkanter Robert Kanter
              • Votes:
                1 Vote for this issue
                Watchers:
                19 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: