[YARN-4904] YARN Log tooling enhancements

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Xuan Gong
• Reporter: Xuan Gong
• Created: 31/Mar/16 23:09
• Updated: 12/Jul/16 20:38

Sub-Tasks

• "yarn logs" command should not require the appOwner argument: Resolved
• Improve "yarn logs" command-line to optionally show log metadata also: Resolved
• Yarn logs should take a -out option to write to a directory: Resolved
• Yarn logs should support a option to output logs as compressed archive: Resolved
• ATS/NM should support a link to dowload/get the logs in text format: Resolved
• Improve yarn logs to fetch container logs without specifying nodeId: Resolved
• Re-direction of a particular log file within in a container in NM UI does not redirect properly to Log Server ( history ) on container completion: Resolved
• Refactory ContainersLogsBlock, AggregatedLogsBlock and container log webservice introduced in AHS to minimize the duplication.: Open
• Add support for zipping/unzipping logs while in transit for the NM logs web-service: Open
• Replace RPC calls with WebService calls in LogsCLI: Resolved
• Improve "yarn log" command-line to read the last K bytes for the log files: Resolved
• Improve "yarn log" command-line "logFiles" option to support regex: Resolved
• Avoid setting CONTENT-DISPOSITION header in the container-logs web-service: Resolved
• "yarn logs" for running containers should print an explicit footer saying that the log may be incomplete: Resolved
• Get Container logs for the Running application from Yarn Logs CommandLine: Resolved
• Rename the “download=true” option for getLogs in NMWebServices and AHSWebServices: Resolved
• LogReader not closed in AHSWebServices and FileInputStream not closed in NMWebServices while reading container logs: Resolved
• Close LogReader in in AHSWebServices#getStreamingOutput and FileInputStream in NMWebServices#getLogs: Resolved
• Improve yarn logs to get Container List: Resolved
• Cannot obtain logs using YARN CLI -am for either KILLED or RUNNING AM: Resolved
• Close FileInputStream in NMWebServices#getLogs in branch-2.8: Resolved
• Container line in yarn logs output for a live application should include the hostname for the container: Resolved
• Logs for a completed container are not available in the yarn logs output for a live application: Resolved
• remove AHS enable check from LogsCLI#fetchAMContainerLogs: Resolved
• yarn logs command: no need to specify -applicationId when specifying containerId: Resolved
• obtaining app logs for last 'n' bytes using CLI gives 'java.io.IOException': Resolved
• TestLogsCLI#testFetchApplictionLogs fails in trunk/branch-2: Resolved
• app logs with 'n' bytes via CLI fails with NumberFormatException: Resolved
• Yarn CLI to obtain App logs for last 'n' bytes fails with 'java.io.IOException' and for 'n' bytes fails with NumberFormatException: Resolved
• Obtaining logs for completed container says 'file belongs to a running container ' at the end: Resolved
• For AM containers, or for containers of running-apps, "yarn logs" incorrectly only (tries to) shows syslog file-type by default: Resolved
• Improve Yarn logs command to get container logs based on Node Id: Patch Available
• Parameter 'size' in the webservices "/containerlogs/$containerid/$filename" and in AHSWebServices is semantically confusing: Open
• Differentiate exactly match with regex in yarn log CLI: Resolved
• Unable to access logs for a running application if YARN_ACL_ENABLE is enabled: Resolved
• Yarn live log aggregation does not throw if command line arg is wrong: Resolved
• Add a new web service to list the files on a container in AHSWebService: Resolved
• Improve webservice to list aggregated log files: Resolved
• improve YARN webservice to output aggregated container logs: Resolved
• Improve AHS webservice to accept NM address as a parameter to get container logs: Resolved
• re-direct NM Web Service to get container logs for finished applications: Resolved
• Add a service to fetch a given list of log files, to a single archive: Open
• Obtaining app logs for Running application fails with "Unable to parse json from webservice. Error:": Resolved
• yarn logs cli should provide logs for a completed container even when application is still running: Resolved
• Potential infinite redirection on YARN log redirection web service: Resolved
• YARN logs CLI needs alway check containerLogsInfo/containerLogInfo before parse the JSON object from NMWebService: Resolved
• Add a size threshold beyond which yarn logs will require a force option: Resolved
• Add retry logic in LogsCLI when fetch running application logs: Resolved
• Document Log CLI in YarnCommands.md: Open