[SLIDER-1187] Create app diagnostics resource with placeholder for containers (live/dead)

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Slider 0.91
• Fix Version/s: Slider 0.92
• Component/s: appmaster, client
• Assignee: Gour Saha
• Reporter: Gour Saha
• Created: 17/Jan/17 17:08
• Updated: 06/Mar/17 21:23
• Resolved: 25/Jan/17 01:39

Description

This is a sample JSON structure of the proposed diagnostics resource -

{
  "finalStatus": "SUCCEEDED", 
  "finalMessage": "stop command issued", 
  "containers": [
    {
      "containerId": "container_e3374_1485226679409_0016_01_000004", 
      "component": "COMMAND_LOGGER", 
      "appVersion": "1.0.0", 
      "state": 3, 
      "exitCode": -1000, 
      "diagnostics": "", 
      "createTime": 1485285533968, 
      "startTime": 1485285533989, 
      "host": "cn008.l42scl.hortonworks.com", 
      "hostURL": "http://cn008.l42scl.hortonworks.com:8042", 
      "logLink": "http://cn007.l42scl.hortonworks.com:19888/jobhistory/logs/cn008.l42scl.hortonworks.com:45454/container_e3374_1485226679409_0016_01_000004/ctx/root"
    }, 
    {
      "containerId": "container_e3374_1485226679409_0016_01_000003", 
      "component": "COMMAND_LOGGER", 
      "appVersion": "1.0.0", 
      "state": 3, 
      "exitCode": -1000, 
      "diagnostics": "", 
      "createTime": 1485285120456, 
      "startTime": 1485285120723, 
      "host": "cn005.l42scl.hortonworks.com", 
      "hostURL": "http://cn005.l42scl.hortonworks.com:8042", 
      "logLink": "http://cn007.l42scl.hortonworks.com:19888/jobhistory/logs/cn005.l42scl.hortonworks.com:45454/container_e3374_1485226679409_0016_01_000003/ctx/root"
    }, 
    {
      "containerId": "container_e3374_1485226679409_0016_01_000002", 
      "component": "COMMAND_LOGGER", 
      "appVersion": "1.0.0", 
      "state": 4, 
      "exitCode": -100, 
      "diagnostics": "Container released by application", 
      "createTime": 1485285120464, 
      "startTime": 1485285120522, 
      "host": "cn008.l42scl.hortonworks.com", 
      "hostURL": "http://cn008.l42scl.hortonworks.com:8042", 
      "logLink": "http://cn007.l42scl.hortonworks.com:19888/jobhistory/logs/cn008.l42scl.hortonworks.com:45454/container_e3374_1485226679409_0016_01_000002/ctx/root"
    }
  ]
}

API consumers will need to call SliderClient#actionDiagnosticContainers API to get the ApplicationDiagnostics object. This object has 3 attributes -

1. finalStatus - app-level status which is empty for a running app (of type org.apache.hadoop.yarn.api.records.FinalApplicationStatus)
2. finalMessage - app-level summary message which is populated after the app dies
3. containers - a set of all currently running and all previously failed containers (type org.apache.slider.api.types.ContainerInformation)
   Note, it also contains an additional helper method getContainer(String containerId) which will return the ContainerInformation for a specific container if the container-id is known.

ContainerInformation (for each running or dead container) contains several attributes which gets updated as and when a container transitions through various stages - like newly created, running, dead, etc. Following are the attributes -

• containerId
• component
• appVersion
• released (true/false)
• state (of type org.apache.slider.api.StateValues)
• exitCode (of type org.apache.hadoop.yarn.api.records.ContainerExitStatus)
• diagnostics (container level diagnostics message)
• createTime
• startTime
• host
• hostURL
• placement
• output (empty so don't use)
• logLink (container log link for a live as well as a dead container)

For an app which is still RUNNING -

ApplicationDiagnostics object can be retrieved at any point in the app's lifetime by calling the SliderClient#actionDiagnosticContainers(ActionDiagnosticArgs diagnosticArgs) API with only the name field in ActionDiagnosticArgs set to the application name. It can be retrieved on the command-line by calling the diagnostics command with the following arguments -

slider diagnostics --name <app-name> --containers

On the command-line it is dumped in JSON format.

For an app which is FAILED/KILLED -

The ApplicationDiagnostics object is set as YARN application diagnostics and can be retrieved by YARN API or through application command line like -

yarn application -status <application_id>

Note, the ApplicationDiagnostics object (in JSON format) can also be viewed in RM UI of the application in the Diagnostics: field.

To retrieve using YARN Client API, this JSON string can be retrieved by calling YarnClient#getApplicationReport(ApplicationId appId) to get the ApplicationReport and then subsequently calling ApplicationReport#getDiagnostics. This JSON string can then be easily converted to the Slider ApplicationDiagnostics object by calling the static method ApplicationDiagnostics#fromJson(String json).