YARN Timeline Server.

[ Go Back ]

Overview

YARN solves the problem of applications' historic information in a generic fashion, which is the timeline server. There are two parts to it:

  • Serving generic information about completed applications. Generic information includes application level data like queue name, user information etc in the ApplicationSubmissionContext, list of application attempts that run for the application, information about each application attempt, List of containers run under each application attempt, and information about each container.
  • Serving per-framework information about completed applications. per-framework information is completely specific to the framework. For example, MapReduce framework can include things like number of map tasks, reduce tasks, counters etc. Application developers can publish the per-framework information to the timeline server via TimelineClient within ApplicationMaster and the application's containers.

Current Status

Timeline sever is still in progress. We are planning to add more features to it. Currently, security check is still not available for accessing applications' historic information. The generic information and the per-framework information should be collected and presented in a more integrated manner. Finally, the per-framework information is only available via RESTful APIs, using JSON type content.

Simple Configuration

Users need to configure the timeline server before starting it. The simplest configuration you should do in yarn-site.xml is to set the host of them timeline server:

<property>
  <description>The hostname of the timeline service web application.</description>
  <name>yarn.timeline-service.hostname</name>
  <value>0.0.0.0</value>
</property>

Advanced Configuration

In addition to the host, it is also free for users to configure the port of the RPC interface and the web interface, and the number of RPC handler threads.

<property>
  <description>This is default address for the timeline server to start the RPC
  server.</description>
  <name>yarn.timeline-service.address</name>
  <value>${yarn.timeline-service.hostname}:10200</value>
</property>

<property>
  <description>The http address of the timeline service web application.</description>
  <name>yarn.timeline-service.webapp.address</name>
  <value>${yarn.timeline-service.hostname}:8188</value>
</property>

<property>
  <description>The https address of the timeline service web application.</description>
  <name>yarn.timeline-service.webapp.https.address</name>
  <value>${yarn.timeline-service.hostname}:8190</value>
</property>

<property>
  <description>Handler thread count to serve the client RPC requests.</description>
  <name>yarn.timeline-service.handler-thread-count</name>
  <value>10</value>
</property>

Generic Data Service Configuration

Users can set whether the generic data service is enabled or not, choose the store class for the generic data. There are more configurations related to generic data service, and users can refer to yarn-default.xml for all of them.

<property>
  <description>Indicate to ResourceManager as well as clients whether
  history-service is enabled or not. If enabled, ResourceManager starts
  recording historical data that ApplicationHistory service can consume.
  Similarly, clients can redirect to the history service when applications
  finish if this is enabled.</description>
  <name>yarn.timeline-service.generic-application-history.enabled</name>
  <value>false</value>
</property>

<property>
  <description>Store class name for history store, defaulting to file system
  store</description>
  <name>yarn.timeline-service.generic-application-history.store-class</name>
  <value>org.apache.hadoop.yarn.server.applicationhistoryservice.FileSystemApplicationHistoryStore</value>
</property>

Per-framework Data Service Configuration

Users can set whether per-framework data service is enabled or not, choose the store class for the per-framework data, and tune the retention of the per-framework data. There are more configurations related to per-framework data service, and users can refer to yarn-default.xml for all of them.

<property>
  <description>Indicate to clients whether timeline service is enabled or not.
  If enabled, clients will put entities and events to the timeline server.
  </description>
  <name>yarn.timeline-service.enabled</name>
  <value>true</value>
</property>

<property>
  <description>Store class name for timeline store.</description>
  <name>yarn.timeline-service.store-class</name>
  <value>org.apache.hadoop.yarn.server.applicationhistoryservice.timeline.LeveldbTimelineStore</value>
</property>

<property>
  <description>Enable age off of timeline store data.</description>
  <name>yarn.timeline-service.ttl-enable</name>
  <value>true</value>
</property>

<property>
  <description>Time to live for timeline store data in milliseconds.</description>
  <name>yarn.timeline-service.ttl-ms</name>
  <value>604800000</value>
</property>

Running Timeline Server

Given all the aforementioned configurations are set properly, users can start the timeline server with the following command:

  $ yarn historyserver

Or users can start the timeline server as a daemon:

  $ yarn-daemon.sh start historyserver

Accessing Generic Data via Command Line

Now users are able to access applications' generic historic data via the command line:

  $ yarn application -status <Application ID>
  $ yarn applicationattempt -list <Application ID>
  $ yarn applicationattempt -status <Application Attempt ID>
  $ yarn container -list <Application Attempt ID>
  $ yarn container -status <Container ID>

Publishing Per-framework Data

Developers can define what information they want to record for their applications, compose TimelineEntity and TimelineEvent objects, and put the entities and events to the timeline server via TimelineClient. Bellow is an example:

  // Create and start the timeline client
  TimelineClient client = TimelineClient.createTimelineClient();
  client.init(conf);
  client.start();

  TimelineEntity entity = null;
  // Compose the entity
  try {
    TimelinePutResponse response = client.putEntities(entity);
  } catch (IOException e) {
    // Handle the exception
  } catch (YarnException e) {
    // Handle the exception
  }

  // Stop the timeline client
  client.stop();