[YARN-10025] Various improvements in YARN log servlets

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.2.1
• Component/s: yarn
• Target Version/s: 3.5.0
• Assignee: Adam Antal
• Reporter: Adam Antal
• Created: 13/Dec/19 09:13
• Updated: 04/Jan/24 08:34

Description

There are multiple ways how we can enhance the current log servlets in YARN.

Issue Links

• is related to: YARN-4993 Refactory ContainersLogsBlock, AggregatedLogsBlock and container log webservice introduced in AHS to minimize the duplication.
• is related to: YARN-6080 Create WebServiceUtils to have common functions used in RMWebService, NMWebService and AHSWebService

Sub-Tasks

• Add ability to filter the Cluster Applications API request by name: Resolved
• Pull out common code pieces from ATS v1.5 and v2: Resolved
• Integrate the new abstract log servlet to the JobHistory server: Resolved
• Provide utility to ask whether an application is in final status: Resolved
• Support listing of aggregated logs for containers belonging to an application attempt: Resolved
• Add option to UIv2 to get container logs from the new JHS API: Resolved
• Endpoint for obtaining direct URL for the logs: Resolved
• Add lazy initialization of LogAggregationFileControllerFactory in LogServlet: Resolved
• Create an endpoint for remote application log directory path query: Resolved
• Create a general purpose log request with additional query parameters: Resolved
• Remove code duplicates from log servlet utility classes: Open
• Yarn logs CLI filtering by application attempt: Resolved
• Add ability to provide required app information manually for log servlet: Open
• Add ability for ATS (log servlet) to read logs of running apps: Open
• Tail -f style CLI tool for extracting logs of running containers: Open
• Document the new JHS API: Open
• Implement regex querying of logs: Open
• Create simple copy log aggregation file controller: Resolved
• HsWebServices containerlogs does not honor ACLs for completed jobs: Resolved
• HsWebServices getContainerReport uses loginUser instead of remoteUser to access ApplicationClientProtocol: Open