[SLIDER-1185] Increase Slider applications (live/dead) debuggability by providing container (live and dead) diagnostics from cmd-line and YARN status

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Slider 0.91
• Fix Version/s: Slider 0.92
• Component/s: appmaster, client
• Assignee: Gour Saha
• Reporter: Gour Saha
• Created: 05/Jan/17 16:20
• Updated: 10/Mar/17 07:05
• Resolved: 06/Mar/17 20:59

Description

Today, the options to debug a failing Slider application are painful. One option is to traverse several links in RM UI starting from the application link and going all the way down to the container logs. An app-owner might have access to a gateway, but still logs might not be available until the app dies if the cluster is not enabled with rolling log aggregation.

Slider provides capability to create apps with friendly names and hence to a certain extent hides the YARN application id. It is not difficult to find the YARN application id, but then app owners are more used to referring their apps by their well-known names. All interactions from the command line using the Slider client requires only the app name.

It would be great to provide container diagnostics (live and dead) like absolute links to container logs in RM UI (links for live and dead containers will be different), additional YARN-level diagnostics (specifically for failed/killed containers), etc. With the absolute log links, an app owner can directly jump to the container logs without having to hunt through RM UI. All these information should be made available from Slider client so that app-owners can query directly from cmd-line using app names. Consumers of Slider client as an SDK will be able to call appropriate APIs and get these diagnostic information. For example, Ambari Slider Views can now show these diagnostics directly in the Ambari UI, relieving the app owners of the pain to traverse the RM UI. Eventually, when the app dies/completes these container diagnostics should be published to YARN status, such that debugging of a failed application becomes easy as well.

At a high level, I am thinking of a cmd-line like -
slider diagnostics --name <app-name> --containers

Issue Links

• is cloned by: SLIDER-1216 [Phase 2] Increase Slider applications (live/dead) debuggability by providing container (live and dead) diagnostics from cmd-line and YARN status

Sub-Tasks

• Create app diagnostics resource with placeholder for containers (live/dead): Resolved
• Create site documentation for the new diagnostics feature detailed in SLIDER-1187: Open
• Make container completion time available in container diagnostics: Resolved
• If an app fails due to "Too many recent failures" - provide the list of containers which counted towards this: Resolved
• Provide information on pending allocations, and last allocation time, yarn headroom resource available: Resolved
• container.state is set to STATE_LIVE for containers belonging to apps which have been killed: Resolved
• Provide information on whether a slider app was killed / stopped via a request: Resolved
• Expose diagnostics information of a running app in ApplicationMaster UI: Resolved
• Remove "Slider AppMaster is accessible" log message from INFO level in getDiagnostics call: Resolved
• Update container log links to log-server URL only for a completed application: Resolved
• Coverity scan issue identified with SLIDER-1214's commit: Resolved