diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml index dc41a16..325c7a1 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml @@ -1105,7 +1105,7 @@ This is default address for the timeline server to start the RPC server. yarn.timeline-service.address - 0.0.0.0:10200 + ${yarn.timeline-service.hostname}:10200 diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/TimelineServer.apt.vm b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/TimelineServer.apt.vm new file mode 100644 index 0000000..6f1982a --- /dev/null +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/TimelineServer.apt.vm @@ -0,0 +1,214 @@ +~~ Licensed under the Apache License, Version 2.0 (the "License"); +~~ you may not use this file except in compliance with the License. +~~ You may obtain a copy of the License at +~~ +~~ http://www.apache.org/licenses/LICENSE-2.0 +~~ +~~ Unless required by applicable law or agreed to in writing, software +~~ distributed under the License is distributed on an "AS IS" BASIS, +~~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +~~ See the License for the specific language governing permissions and +~~ limitations under the License. See accompanying LICENSE file. + + --- + YARN Timeline Server. + --- + --- + ${maven.build.timestamp} + +YARN Timeline Server. + + \[ {{{./index.html}Go Back}} \] + +%{toc|section=1|fromDepth=0|toDepth=3} + +* Overview + + YARN solves the problem of applications' historic information in a generic + fashion, which is the timeline server. There are two parts to it: + + * Serving generic information about completed applications. Generic + information includes application level data like queue name, user + information etc in the ApplicationSubmissionContext, list of + application attempts that run for the application, information about each + application attempt, List of containers run under each application attempt, + and information about each container. + + * Serving per-framework information about completed applications. + per-framework information is completely specific to the framework. For + example, MapReduce framework can include things like number of map tasks, + reduce tasks, counters etc. Application developers can publish the + per-framework information to the timeline server via TimelineClient within + ApplicationMaster and the application's containers. + +* Current Status + + Timeline sever is still in progress. We are planning to add more features to + it. Currently, security check is still not available for accessing + applications' historic information. The generic information and the + per-framework information should be collected and presented in a more + integrated manner. Finally, the per-framework information is only available + via RESTful APIs, using JSON type content. + +* Simple Configuration + + Users need to configure the timeline server before starting it. The simplest + configuration you should do in <<>> is to set the host of them + timeline server: + ++---+ + + The hostname of the timeline service web application. + yarn.timeline-service.hostname + 0.0.0.0 + ++---+ + +* Advanced Configuration + + In addition to the host, it is also free for users to configure the port of + the RPC interface and the web interface, and the number of RPC handler + threads. + ++---+ + + This is default address for the timeline server to start the RPC + server. + yarn.timeline-service.address + ${yarn.timeline-service.hostname}:10200 + + + + The http address of the timeline service web application. + yarn.timeline-service.webapp.address + ${yarn.timeline-service.hostname}:8188 + + + + The https address of the timeline service web application. + yarn.timeline-service.webapp.https.address + ${yarn.timeline-service.hostname}:8190 + + + + Handler thread count to serve the client RPC requests. + yarn.timeline-service.handler-thread-count + 10 + ++---+ + +* Generic Data Service Configuration + + Users can set whether the generic data service is enabled or not, choose the + store class for the generic data. There are more configurations related to + generic data service, and users can refer to <<>> for all + of them. + ++---+ + + Indicate to ResourceManager as well as clients whether + history-service is enabled or not. If enabled, ResourceManager starts + recording historical data that ApplicationHistory service can consume. + Similarly, clients can redirect to the history service when applications + finish if this is enabled. + yarn.timeline-service.generic-application-history.enabled + false + + + + Store class name for history store, defaulting to file system + store + yarn.timeline-service.generic-application-history.store-class + org.apache.hadoop.yarn.server.applicationhistoryservice.FileSystemApplicationHistoryStore + ++---+ + +* Per-framework Data Service Configuration + + Users can set whether per-framework data service is enabled or not, choose + the store class for the per-framework data, and tune the retention of the + per-framework data. There are more configurations related to per-framework + data service, and users can refer to <<>> for all of them. + ++---+ + + Indicate to clients whether timeline service is enabled or not. + If enabled, clients will put entities and events to the timeline server. + + yarn.timeline-service.enabled + true + + + + Store class name for timeline store. + yarn.timeline-service.store-class + org.apache.hadoop.yarn.server.applicationhistoryservice.timeline.LeveldbTimelineStore + + + + Enable age off of timeline store data. + yarn.timeline-service.ttl-enable + true + + + + Time to live for timeline store data in milliseconds. + yarn.timeline-service.ttl-ms + 604800000 + ++---+ + +* Running Timeline Server + + Given all the aforementioned configurations are set properly, users can start + the timeline server with the following command: + ++---+ + $ yarn historyserver ++---+ + + Or users can start the timeline server as a daemon: + ++---+ + $ yarn-daemon.sh start historyserver ++---+ + +* Accessing Generic Data via Command Line + + Now users are able to access applications' generic historic data via the + command line: + ++---+ + $ yarn application -status + $ yarn applicationattempt -list + $ yarn applicationattempt -status + $ yarn container -list + $ yarn container -status ++---+ + +* Publishing Per-framework Data + + Developers can define what information they want to record for their + applications, compose <<>> and <<>> objects, + and put the entities and events to the timeline server via + <<>>. Bellow is an example: + ++---+ + // Create and start the timeline client + TimelineClient client = TimelineClient.createTimelineClient(); + client.init(conf); + client.start(); + + TimelineEntity entity = null; + // Compose the entity + try { + TimelinePutResponse response = client.putEntities(entity); + } catch (IOException e) { + // Handle the exception + } catch (YarnException e) { + // Handle the exception + } + + // Stop the timeline client + client.stop(); ++---+ diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/index.apt.vm b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/index.apt.vm index 1988c0b..9f07b19 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/index.apt.vm +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/index.apt.vm @@ -43,7 +43,7 @@ MapReduce NextGen aka YARN aka MRv2 * {{{./YARN.html}NextGen MapReduce}} - * {{{./WritingYarnApplications.html}Writing Yarn Applications}} + * {{{./WritingYarnApplications.html}Writing YARN Applications}} * {{{./CapacityScheduler.html}Capacity Scheduler}} @@ -51,6 +51,8 @@ MapReduce NextGen aka YARN aka MRv2 * {{{./WebApplicationProxy.html}Web Application Proxy}} + * {{{./TimelineServer.html}YARN Timeline Server}} + * {{{../../hadoop-project-dist/hadoop-common/CLIMiniCluster.html}CLI MiniCluster}} * {{{../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduce_Compatibility_Hadoop1_Hadoop2.html}Backward Compatibility between Apache Hadoop 1.x and 2.x for MapReduce}}