diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServiceV2.md b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServiceV2.md index 7435201..fecc8e0 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServiceV2.md +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServiceV2.md @@ -514,6 +514,8 @@ predicates, an empty list will be returned. "daterange=20150711-20150714" returns flows active between these 2 dates.
"daterange=20150711-" returns flows active on and after 20150711.
"daterange=-20150711" returns flows active on and before 20150711.
+1. `fromid` - If specified, retrieve the next set of flows from the given fromid. The set of entities retrieved is inclusive of specified fromid. + fromid should be taken from the value associated with FROM_ID info key in flow entity response which was sent earlier. #### Example JSON Response: @@ -559,6 +561,7 @@ predicates, an empty list will be returned. "info": { "SYSTEM_INFO_CLUSTER": "test-cluster", "UID": "test-cluster!sjlee!ds-date", + "FROM_ID": "test-cluster!1460419200000!sjlee!ds-date", "SYSTEM_INFO_FLOW_NAME": "ds-date", "SYSTEM_INFO_DATE": 1460419200000, "SYSTEM_INFO_USER": "sjlee" @@ -610,6 +613,8 @@ predicates, an empty list will be returned. 1. `fields` - Specifies which fields to retrieve. For querying flow runs, only `ALL` or `METRICS` are valid fields. Other fields will lead to HTTP 400 (Bad Request) response. If not specified, in response, id, type, createdtime and info fields will be returned. +1. `fromid` - If specified, retrieve the next set of flow run entities from the given fromid. The set of entities retrieved is inclusive of specified fromid. + fromid should be taken from the value associated with FROM_ID info key in flow entity response which was sent earlier. #### Example JSON Response: @@ -622,6 +627,7 @@ predicates, an empty list will be returned. "createdtime": 1460420587974, "info": { "UID": "test-cluster!sjlee!ds-date!1460420587974", + "FROM_ID": "test-cluster!sjlee!ds-date!1460420587974", "SYSTEM_INFO_FLOW_RUN_ID": 1460420587974, "SYSTEM_INFO_FLOW_NAME": "ds-date", "SYSTEM_INFO_FLOW_RUN_END_TIME": 1460420595198, @@ -638,6 +644,7 @@ predicates, an empty list will be returned. "createdtime": 1460420305659, "info": { "UID": "test-cluster!sjlee!ds-date!1460420305659", + "FROM_ID": "test-cluster!sjlee!ds-date!1460420305659", "SYSTEM_INFO_FLOW_RUN_ID": 1460420305659, "SYSTEM_INFO_FLOW_NAME": "ds-date", "SYSTEM_INFO_FLOW_RUN_END_TIME": 1460420311966, @@ -709,6 +716,7 @@ while querying individual flow runs. "isrelatedto": {}, "info": { "UID":"yarn-cluster!varun!QuasiMonteCarlo!1465246348599", + "FROM_ID":"yarn-cluster!varun!QuasiMonteCarlo!1465246348599", "SYSTEM_INFO_FLOW_RUN_END_TIME":1465246378051, "SYSTEM_INFO_FLOW_NAME":"QuasiMonteCarlo", "SYSTEM_INFO_USER":"varun", @@ -813,6 +821,8 @@ none of the apps match the predicates, an empty list will be returned. or metricstoretrieve is specified. Ignored otherwise. The maximum possible value for metricslimit can be maximum value of Integer. If it is not specified or has a value less than 1, and metrics have to be retrieved, then metricslimit will be considered as 1 i.e. latest single value of metric(s) will be returned. +1. `fromid` - If specified, retrieve the next set of application entities from the given fromid. The set of entities retrieved is inclusive of specified fromid. + fromid should be taken from the value associated with FROM_ID info key in flow entity response which was sent earlier. #### Example JSON Response: @@ -827,6 +837,7 @@ none of the apps match the predicates, an empty list will be returned. "configs": { }, "info": { "UID": "yarn-cluster!application_1465246237936_0001" + "FROM_ID": "yarn-cluster!varun!QuasiMonteCarlo!1465246348599!application_1465246237936_0001", }, "relatesto": { } }, @@ -840,6 +851,7 @@ none of the apps match the predicates, an empty list will be returned. "configs": { }, "info": { "UID": "yarn-cluster!application_1464983628730_0005" + "FROM_ID": "yarn-cluster!varun!QuasiMonteCarlo!1465246348599!application_1464983628730_0005", }, "relatesto": { } } @@ -940,6 +952,8 @@ match the predicates, an empty list will be returned. or metricstoretrieve is specified. Ignored otherwise. The maximum possible value for metricslimit can be maximum value of Integer. If it is not specified or has a value less than 1, and metrics have to be retrieved, then metricslimit will be considered as 1 i.e. latest single value of metric(s) will be returned. +1. `fromid` - If specified, retrieve the next set of application entities from the given fromid. The set of entities retrieved is inclusive of specified fromid. + fromid should be taken from the value associated with FROM_ID info key in flow entity response which was sent earlier. #### Example JSON Response: @@ -952,6 +966,7 @@ match the predicates, an empty list will be returned. "createdtime": 1460419580171, "info": { "UID": "test-cluster!sjlee!ds-date!1460419580171!application_1460419579913_0002" + "FROM_ID": "test-cluster!sjlee!ds-date!1460419580171!application_1460419579913_0002", }, "configs": {}, "isrelatedto": {}, @@ -1140,6 +1155,8 @@ If none of the entities match the predicates, an empty list will be returned. or metricstoretrieve is specified. Ignored otherwise. The maximum possible value for metricslimit can be maximum value of Integer. If it is not specified or has a value less than 1, and metrics have to be retrieved, then metricslimit will be considered as 1 i.e. latest single value of metric(s) will be returned. +1. `fromid` - If specified, retrieve the next set of generic entities from the given fromid. The set of entities retrieved is inclusive of specified fromid. + fromid should be taken from the value associated with FROM_ID info key in flow entity response which was sent earlier. #### Example JSON Response: @@ -1154,6 +1171,7 @@ If none of the entities match the predicates, an empty list will be returned. "configs": { }, "info": { "UID": "yarn-cluster!application_1465246237936_0001!YARN_APPLICATION_ATTEMPT!appattempt_1465246237936_0001_000001" + "FROM_ID": "yarn-cluster!sjlee!ds-date!1460419580171!application_1465246237936_0001!YARN_APPLICATION_ATTEMPT!0!appattempt_1465246237936_0001_000001" }, "relatesto": { } }, @@ -1167,6 +1185,7 @@ If none of the entities match the predicates, an empty list will be returned. "configs": { }, "info": { "UID": "yarn-cluster!application_1465246237936_0001!YARN_APPLICATION_ATTEMPT!appattempt_1465246237936_0001_000002" + "FROM_ID": "yarn-cluster!sjlee!ds-date!1460419580171!application_1465246237936_0001!YARN_APPLICATION_ATTEMPT!0!appattempt_1465246237936_0001_000002" }, "relatesto": { } } @@ -1233,6 +1252,7 @@ container ID. Similarly, application attempt can be queried by specifying entity or metricstoretrieve is specified. Ignored otherwise. The maximum possible value for metricslimit can be maximum value of Integer. If it is not specified or has a value less than 1, and metrics have to be retrieved, then metricslimit will be considered as 1 i.e. latest single value of metric(s) will be returned. +1. `entityidprefix` Defines the id prefix for the entity to be fetched. If specified, then entity retrieval will be faster. #### Example JSON Response: @@ -1245,7 +1265,8 @@ container ID. Similarly, application attempt can be queried by specifying entity "isrelatedto": { }, "configs": { }, "info": { - "UID": "yarn-cluster!application_1465246237936_0001!YARN_APPLICATION_ATTEMPT!appattempt_1465246237936_0001_000001" + "UID": "yarn-cluster!application_1465246237936_0001!YARN_APPLICATION_ATTEMPT!0!appattempt_1465246237936_0001_000001" + "FROM_ID": "yarn-cluster!sjlee!ds-date!1460419580171!application_1465246237936_0001!YARN_APPLICATION_ATTEMPT!0!appattempt_1465246237936_0001_000001" }, "relatesto": { } } @@ -1256,3 +1277,41 @@ container ID. Similarly, application attempt can be queried by specifying entity 1. If any problem occurs in parsing request, HTTP 400 (Bad Request) is returned. 1. If flow context information cannot be retrieved or entity for the given entity id cannot be found, HTTP 404 (Not Found) is returned. 1. For non-recoverable errors while retrieving data, HTTP 500 (Internal Server Error) is returned. + +### Query generic entity types + +With this API, you can query set of available entity types for a given app id. If the REST endpoint without the cluster name is used, the cluster specified by the configuration yarn.resourcemanager.cluster-id in yarn-site.xml is taken. If userid, flow name and flow run id which are optional query parameters are not specified, they will be queried based on app id and cluster id from the flow context information stored in underlying storage implementation. + +#### HTTP request: + + GET /ws/v2/timeline/apps/{appid}/entity-types + + or + + GET /ws/v2/timeline/clusters/{clusterid}/apps/{appid}/entity-types + +#### Query Parameters Supported: + +1. `userid` - If specified, entity must belong to this user. This query param must be specified along with flowname and flowrunid query params, otherwise it will be ignored. + If userid, flowname and flowrunid are not specified then timeline reader will fetch flow context information based on cluster and appid while executing the query. +1. `flowname` - If specified, entity must belong to this flow name. This query param must be specified along with userid and flowrunid query params, otherwise it will be ignored. + If userid, flowname and flowrunid are not specified, we would have to fetch flow context information based on cluster and appid while executing the query. +1. `flowrunid` - If specified, entity must belong to this flow run id. This query param must be specified along with userid and flowname query params, otherwise it will be ignored. + If userid, flowname and flowrunid are not specified, we would have to fetch flow context information based on cluster and appid while executing the query. + +#### Example JSON Response: + + { + YARN_APPLICATION_ATTEMPT, + YARN_CONTAINER, + MAPREDUCE_JOB, + MAPREDUCE_TASK, + MAPREDUCE_TASK_ATTEMPT + } + +#### Response Codes + +1. If successful, a HTTP 200 (OK) response is returned. +1. If any problem occurs in parsing request, HTTP 400 (Bad Request) is returned. +1. If flow context information cannot be retrieved or entity for the given entity id cannot be found, HTTP 404 (Not Found) is returned. +1. For non-recoverable errors while retrieving data, HTTP 500 (Internal Server Error) is returned.