Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-5577

[Atsv2] Document object passing in infofilters with an example

    Details

    • Hadoop Flags:
      Reviewed

      Description

      In HierarchicalTimelineEntity, setparent/addChild allows to set parent/child entities at INFO level. The key is an string and value as an object.
      Like below, for YARN_CONTAINER entity parent entity set for application.

      "SYSTEM_INFO_PARENT_ENTITY": {
             "type": "YARN_APPLICATION",
             "id": "application_1471931266232_0024"
           }
      

      But to use infofilter on entity type YARN_CONTAINER for an specific applicationId, IIUC there is no way to pass object as value in infofilter.

      To make easier retrieval either

      1. publish parent/child entity id and type as string rather that object like below
        "SYSTEM_INFO_PARENT_ENTITY_TYPE": "YARN_APPLICATION"
        "SYSTEM_INFO_PARENT_ENTITY_ID":"application_1471931266232_0024"
        

        OR

      2. Add ability to provide object as filter with below format like infofilters=SYSTEM_INFO_PARENT_ENTITY eq ((type eq YARN_APPLICATION) AND (id eq application_1471931266232_0024))

      I believe 2nd approach will be well applicable for any entities. But I am not sure does HBase supports such a custom filters while scanning a table.

      1st approaches will be much easier to change.

      1. YARN-5577.patch
        4 kB
        Rohith Sharma K S

        Activity

        Hide
        Naganarasimha Naganarasimha G R added a comment -

        +1 for first approach,
        But Sangjin Lee , Vrushali C, Joep Rottinghuis, any idea why we choose to store it as object ?

        Show
        Naganarasimha Naganarasimha G R added a comment - +1 for first approach, But Sangjin Lee , Vrushali C , Joep Rottinghuis , any idea why we choose to store it as object ?
        Hide
        rohithsharma Rohith Sharma K S added a comment -

        I found the way to achieve filtering of an object in infofilter. Infofilters uses an JasonParser to parse the value. So this value can be object in the form of Json.
        For example : To list all the container entities belongs to app-attempt, query can be

        GET /ws/v2/timeline/apps/application_1471931266232_0024/entities/YARN_CONTAINER?fields=INFO&infofilters=SYSTEM_INFO_PARENT_ENTITY eq {"type":"YARN_APPLICATION_ATTEMPT","id":"appattempt_1471931266232_0024_000002"}
        

        Value should be in the form of Json without any space i.e

        {"type":"YARN_APPLICATION_ATTEMPT","id":"appattempt_1471931266232_0024_000002"}

        This has to be captured in the AtsV2 documentation along with an example which is missing now for infofilters.

        Show
        rohithsharma Rohith Sharma K S added a comment - I found the way to achieve filtering of an object in infofilter. Infofilters uses an JasonParser to parse the value. So this value can be object in the form of Json. For example : To list all the container entities belongs to app-attempt, query can be GET /ws/v2/timeline/apps/application_1471931266232_0024/entities/YARN_CONTAINER?fields=INFO&infofilters=SYSTEM_INFO_PARENT_ENTITY eq {"type":"YARN_APPLICATION_ATTEMPT","id":"appattempt_1471931266232_0024_000002"} Value should be in the form of Json without any space i.e { "type" : "YARN_APPLICATION_ATTEMPT" , "id" : "appattempt_1471931266232_0024_000002" } This has to be captured in the AtsV2 documentation along with an example which is missing now for infofilters.
        Hide
        rohithsharma Rohith Sharma K S added a comment -

        Updated the documentation patch for passing an object in infofilters along with an example.

        Show
        rohithsharma Rohith Sharma K S added a comment - Updated the documentation patch for passing an object in infofilters along with an example.
        Hide
        rohithsharma Rohith Sharma K S added a comment -

        Summary also updated as per the patch provided.

        Show
        rohithsharma Rohith Sharma K S added a comment - Summary also updated as per the patch provided.
        Hide
        varun_saxena Varun Saxena added a comment -

        Updating documentation should be fine.
        Objects should be represented as JSON. This was missed in documentation.
        Looks fine to me

        Show
        varun_saxena Varun Saxena added a comment - Updating documentation should be fine. Objects should be represented as JSON. This was missed in documentation. Looks fine to me
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 19s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 mvninstall 7m 7s trunk passed
        +1 mvnsite 0m 14s trunk passed
        +1 mvnsite 0m 11s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 asflicense 0m 16s The patch does not generate ASF License warnings.
        8m 23s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12825993/YARN-5577.patch
        JIRA Issue YARN-5577
        Optional Tests asflicense mvnsite
        uname Linux e94aae3c5765 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / c258171
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/12925/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 mvninstall 7m 7s trunk passed +1 mvnsite 0m 14s trunk passed +1 mvnsite 0m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 asflicense 0m 16s The patch does not generate ASF License warnings. 8m 23s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12825993/YARN-5577.patch JIRA Issue YARN-5577 Optional Tests asflicense mvnsite uname Linux e94aae3c5765 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / c258171 modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site Console output https://builds.apache.org/job/PreCommit-YARN-Build/12925/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
        Hide
        rohithsharma Rohith Sharma K S added a comment -

        Varun Saxena If patch is fine, do you mind committing it?

        Show
        rohithsharma Rohith Sharma K S added a comment - Varun Saxena If patch is fine, do you mind committing it?
        Hide
        varun_saxena Varun Saxena added a comment -

        Sorry I missed this one. Will commit it tomorrow.

        Show
        varun_saxena Varun Saxena added a comment - Sorry I missed this one. Will commit it tomorrow.
        Hide
        rohithsharma Rohith Sharma K S added a comment -

        ping Varun Saxena!!

        Show
        rohithsharma Rohith Sharma K S added a comment - ping Varun Saxena !!
        Hide
        varun_saxena Varun Saxena added a comment -

        Sorry Rohith Sharma K S, I was on leave so didn't commit it. Will do so shortly.
        I will commit it to trunk as its documentation related change. Will be brought in ATSv2 branch when we do trunk rebase.

        Show
        varun_saxena Varun Saxena added a comment - Sorry Rohith Sharma K S , I was on leave so didn't commit it. Will do so shortly. I will commit it to trunk as its documentation related change. Will be brought in ATSv2 branch when we do trunk rebase.
        Hide
        varun_saxena Varun Saxena added a comment -

        Committed to trunk.
        Thanks Rohith Sharma K S for your contribution

        Show
        varun_saxena Varun Saxena added a comment - Committed to trunk. Thanks Rohith Sharma K S for your contribution
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10459 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10459/)
        YARN-5577. [Atsv2] Document object passing in infofilters with an (varunsaxena: rev ea29e3bc27f15516f4346d1312eef703bcd3d032)

        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServiceV2.md
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10459 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10459/ ) YARN-5577 . [Atsv2] Document object passing in infofilters with an (varunsaxena: rev ea29e3bc27f15516f4346d1312eef703bcd3d032) (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServiceV2.md

          People

          • Assignee:
            rohithsharma Rohith Sharma K S
            Reporter:
            rohithsharma Rohith Sharma K S
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development