[MAPREDUCE-6415] Create a tool to combine aggregated logs into HAR files

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.8.0
• Fix Version/s: 2.8.0, 3.0.0-alpha1
• Target Version/s: 2.8.0
• Hadoop Flags: Reviewed
• Assignee: Robert Kanter
• Reporter: Robert Kanter
• Created: 24/Jun/15 23:19
• Updated: 28/Aug/19 12:02
• Resolved: 10/Sep/15 00:56

Description

While we wait for YARN-2942 to become viable, it would still be great to improve the aggregated logs problem. We can write a tool that combines aggregated log files into a single HAR file per application, which should solve the too many files and too many blocks problems. See the design document for details.

See YARN-2942 for more context.

Issue Links

• depends upon: YARN-3950 Add unique YARN_SHELL_ID environment variable to DistributedShell
• relates to: MAPREDUCE-7236 HadoopArchiveLogs will use token to create proxy user when kerberos on
• relates to: MAPREDUCE-7027 HadoopArchiveLogs shouldn't delete the original logs if the HAR creation fails
• relates to: MAPREDUCE-6480 archive-logs tool may miss applications
• relates to: MAPREDUCE-6494 Permission issue when running archive-logs tool as different users
• relates to: MAPREDUCE-6495 Docs for archive-logs tool
• relates to: MAPREDUCE-6503 archive-logs tool should use HADOOP_PREFIX instead of HADOOP_HOME
• relates to: MAPREDUCE-6550 archive-logs tool changes log ownership to the Yarn user when using DefaultContainerExecutor
• relates to: MAPREDUCE-7023 TestHadoopArchiveLogs.testCheckFilesAndSeedApps fails on rerun
• relates to: MAPREDUCE-7235 NPE in HadoopArchiveLogs#filterAppsByAggregatedStatus
• relates to: YARN-2942 Aggregated Log Files should be combined
• relates to: MAPREDUCE-6970 archive-logs tool should throttle container requests
• relates to: YARN-4086 Allow Aggregated Log readers to handle HAR files
• relates to: YARN-4946 RM should not consider an application as COMPLETED when log aggregation is not in a terminal state