[ Go Back ]
YARN solves the problem of applications' historic information in a generic fashion, which is the timeline server. There are two parts to it:
Timeline sever is still in progress. We are planning to add more features to it. Currently, security check is still not available for accessing applications' historic information. The generic information and the per-framework information should be collected and presented in a more integrated manner. Finally, the per-framework information is only available via RESTful APIs, using JSON type content.
Users need to configure the timeline server before starting it. The simplest configuration you should do in yarn-site.xml is to set the host of them timeline server:
<property> <description>The hostname of the timeline service web application.</description> <name>yarn.timeline-service.hostname</name> <value>0.0.0.0</value> </property>
In addition to the host, it is also free for users to configure the port of the RPC interface and the web interface, and the number of RPC handler threads.
<property>
<description>This is default address for the timeline server to start the RPC
server.</description>
<name>yarn.timeline-service.address</name>
<value>${yarn.timeline-service.hostname}:10200</value>
</property>
<property>
<description>The http address of the timeline service web application.</description>
<name>yarn.timeline-service.webapp.address</name>
<value>${yarn.timeline-service.hostname}:8188</value>
</property>
<property>
<description>The https address of the timeline service web application.</description>
<name>yarn.timeline-service.webapp.https.address</name>
<value>${yarn.timeline-service.hostname}:8190</value>
</property>
<property>
<description>Handler thread count to serve the client RPC requests.</description>
<name>yarn.timeline-service.handler-thread-count</name>
<value>10</value>
</property>Users can set whether the generic data service is enabled or not, choose the store class for the generic data. There are more configurations related to generic data service, and users can refer to yarn-default.xml for all of them.
<property> <description>Indicate to ResourceManager as well as clients whether history-service is enabled or not. If enabled, ResourceManager starts recording historical data that ApplicationHistory service can consume. Similarly, clients can redirect to the history service when applications finish if this is enabled.</description> <name>yarn.timeline-service.generic-application-history.enabled</name> <value>false</value> </property> <property> <description>Store class name for history store, defaulting to file system store</description> <name>yarn.timeline-service.generic-application-history.store-class</name> <value>org.apache.hadoop.yarn.server.applicationhistoryservice.FileSystemApplicationHistoryStore</value> </property>
Users can set whether per-framework data service is enabled or not, choose the store class for the per-framework data, and tune the retention of the per-framework data. There are more configurations related to per-framework data service, and users can refer to yarn-default.xml for all of them.
<property> <description>Indicate to clients whether timeline service is enabled or not. If enabled, clients will put entities and events to the timeline server. </description> <name>yarn.timeline-service.enabled</name> <value>true</value> </property> <property> <description>Store class name for timeline store.</description> <name>yarn.timeline-service.store-class</name> <value>org.apache.hadoop.yarn.server.applicationhistoryservice.timeline.LeveldbTimelineStore</value> </property> <property> <description>Enable age off of timeline store data.</description> <name>yarn.timeline-service.ttl-enable</name> <value>true</value> </property> <property> <description>Time to live for timeline store data in milliseconds.</description> <name>yarn.timeline-service.ttl-ms</name> <value>604800000</value> </property>
Given all the aforementioned configurations are set properly, users can start the timeline server with the following command:
$ yarn historyserver
Or users can start the timeline server as a daemon:
$ yarn-daemon.sh start historyserver
Now users are able to access applications' generic historic data via the command line:
$ yarn application -status <Application ID> $ yarn applicationattempt -list <Application ID> $ yarn applicationattempt -status <Application Attempt ID> $ yarn container -list <Application Attempt ID> $ yarn container -status <Container ID>
Developers can define what information they want to record for their applications, compose TimelineEntity and TimelineEvent objects, and put the entities and events to the timeline server via TimelineClient. Bellow is an example:
// Create and start the timeline client
TimelineClient client = TimelineClient.createTimelineClient();
client.init(conf);
client.start();
TimelineEntity entity = null;
// Compose the entity
try {
TimelinePutResponse response = client.putEntities(entity);
} catch (IOException e) {
// Handle the exception
} catch (YarnException e) {
// Handle the exception
}
// Stop the timeline client
client.stop();