Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-5577

[Atsv2] Document object passing in infofilters with an example

        Details

        • Hadoop Flags:
          Reviewed

          Description

          In HierarchicalTimelineEntity, setparent/addChild allows to set parent/child entities at INFO level. The key is an string and value as an object.
          Like below, for YARN_CONTAINER entity parent entity set for application.

          "SYSTEM_INFO_PARENT_ENTITY": {
       "type": "YARN_APPLICATION",
       "id": "application_1471931266232_0024"
     }

          But to use infofilter on entity type YARN_CONTAINER for an specific applicationId, IIUC there is no way to pass object as value in infofilter.

          To make easier retrieval either

          1. publish parent/child entity id and type as string rather that object like below 
            "SYSTEM_INFO_PARENT_ENTITY_TYPE": "YARN_APPLICATION"
"SYSTEM_INFO_PARENT_ENTITY_ID":"application_1471931266232_0024"

            OR

          2. Add ability to provide object as filter with below format like infofilters=SYSTEM_INFO_PARENT_ENTITY eq ((type eq YARN_APPLICATION) AND (id eq application_1471931266232_0024))

          I believe 2nd approach will be well applicable for any entities. But I am not sure does HBase supports such a custom filters while scanning a table.

          1st approaches will be much easier to change.

          1. Text File
            YARN-5577.patch
            4 kB
            Rohith Sharma K S

            Activity

            Ascending order - Click to sort in descending order
            Hide
            Permalink
            Naganarasimha Naganarasimha G R added a comment -

            +1 for first approach,
            But Sangjin Lee , Vrushali C, Joep Rottinghuis, any idea why we choose to store it as object ?

            Show
            Naganarasimha Naganarasimha G R added a comment - +1 for first approach, But Sangjin Lee , Vrushali C , Joep Rottinghuis , any idea why we choose to store it as object ?
            Hide
            Permalink
            rohithsharma Rohith Sharma K S added a comment -

            I found the way to achieve filtering of an object in infofilter. Infofilters uses an JasonParser to parse the value. So this value can be object in the form of Json.
            For example : To list all the container entities belongs to app-attempt, query can be 

            GET /ws/v2/timeline/apps/application_1471931266232_0024/entities/YARN_CONTAINER?fields=INFO&infofilters=SYSTEM_INFO_PARENT_ENTITY eq {"type":"YARN_APPLICATION_ATTEMPT","id":"appattempt_1471931266232_0024_000002"}

            Value should be in the form of Json without any space i.e 

            {"type":"YARN_APPLICATION_ATTEMPT","id":"appattempt_1471931266232_0024_000002"}

            This has to be captured in the AtsV2 documentation along with an example which is missing now for infofilters.

            Show
            rohithsharma Rohith Sharma K S added a comment - I found the way to achieve filtering of an object in infofilter. Infofilters uses an JasonParser to parse the value. So this value can be object in the form of Json. For example : To list all the container entities belongs to app-attempt, query can be GET /ws/v2/timeline/apps/application_1471931266232_0024/entities/YARN_CONTAINER?fields=INFO&infofilters=SYSTEM_INFO_PARENT_ENTITY eq {"type":"YARN_APPLICATION_ATTEMPT","id":"appattempt_1471931266232_0024_000002"} Value should be in the form of Json without any space i.e { "type" : "YARN_APPLICATION_ATTEMPT" , "id" : "appattempt_1471931266232_0024_000002" } This has to be captured in the AtsV2 documentation along with an example which is missing now for infofilters.
            Hide
            Permalink
            rohithsharma Rohith Sharma K S added a comment -

            Updated the documentation patch for passing an object in infofilters along with an example.

            Show
            rohithsharma Rohith Sharma K S added a comment - Updated the documentation patch for passing an object in infofilters along with an example.
            Hide
            Permalink
            rohithsharma Rohith Sharma K S added a comment -

            Summary also updated as per the patch provided.

            Show
            rohithsharma Rohith Sharma K S added a comment - Summary also updated as per the patch provided.
            Hide
            Permalink
            varun_saxena Varun Saxena added a comment -

            Updating documentation should be fine.
            Objects should be represented as JSON. This was missed in documentation.
            Looks fine to me

            Show
            varun_saxena Varun Saxena added a comment - Updating documentation should be fine. Objects should be represented as JSON. This was missed in documentation. Looks fine to me
            Hide
            Permalink
            hadoopqa Hadoop QA added a comment -
            +1 overall



            Vote Subsystem Runtime Comment
            0 reexec 0m 19s Docker mode activated.
            +1 @author 0m 0s The patch does not contain any @author tags.
            +1 mvninstall 7m 7s trunk passed
            +1 mvnsite 0m 14s trunk passed
            +1 mvnsite 0m 11s the patch passed
            +1 whitespace 0m 0s The patch has no whitespace issues.
            +1 asflicense 0m 16s The patch does not generate ASF License warnings.
            8m 23s



            Subsystem Report/Notes
            Docker Image:yetus/hadoop:9560f25
            JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12825993/YARN-5577.patch
            JIRA Issue YARN-5577
            Optional Tests asflicense mvnsite
            uname Linux e94aae3c5765 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
            Build tool maven
            Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
            git revision trunk / c258171
            modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
            Console output https://builds.apache.org/job/PreCommit-YARN-Build/12925/console
            Powered by Apache Yetus 0.3.0 http://yetus.apache.org

            This message was automatically generated.

            Show
            hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 mvninstall 7m 7s trunk passed +1 mvnsite 0m 14s trunk passed +1 mvnsite 0m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 asflicense 0m 16s The patch does not generate ASF License warnings. 8m 23s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12825993/YARN-5577.patch JIRA Issue YARN-5577 Optional Tests asflicense mvnsite uname Linux e94aae3c5765 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / c258171 modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site Console output https://builds.apache.org/job/PreCommit-YARN-Build/12925/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
            Hide
            Permalink
            rohithsharma Rohith Sharma K S added a comment -

            Varun Saxena If patch is fine, do you mind committing it?

            Show
            rohithsharma Rohith Sharma K S added a comment - Varun Saxena If patch is fine, do you mind committing it?
            Hide
            Permalink
            varun_saxena Varun Saxena added a comment -

            Sorry I missed this one. Will commit it tomorrow.

            Show
            varun_saxena Varun Saxena added a comment - Sorry I missed this one. Will commit it tomorrow.
            Hide
            Permalink
            rohithsharma Rohith Sharma K S added a comment -

            ping Varun Saxena!!

            Show
            rohithsharma Rohith Sharma K S added a comment - ping Varun Saxena !!
            Hide
            Permalink
            varun_saxena Varun Saxena added a comment -

            Sorry Rohith Sharma K S, I was on leave so didn't commit it. Will do so shortly.
            I will commit it to trunk as its documentation related change. Will be brought in ATSv2 branch when we do trunk rebase.

            Show
            varun_saxena Varun Saxena added a comment - Sorry Rohith Sharma K S , I was on leave so didn't commit it. Will do so shortly. I will commit it to trunk as its documentation related change. Will be brought in ATSv2 branch when we do trunk rebase.
            Hide
            Permalink
            varun_saxena Varun Saxena added a comment -

            Committed to trunk.
            Thanks Rohith Sharma K S for your contribution

            Show
            varun_saxena Varun Saxena added a comment - Committed to trunk. Thanks Rohith Sharma K S for your contribution
            Hide
            Permalink
            hudson Hudson added a comment -

            SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10459 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10459/)
            YARN-5577. [Atsv2] Document object passing in infofilters with an (varunsaxena: rev ea29e3bc27f15516f4346d1312eef703bcd3d032)

            • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServiceV2.md
            Show
            hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10459 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10459/ ) YARN-5577 . [Atsv2] Document object passing in infofilters with an (varunsaxena: rev ea29e3bc27f15516f4346d1312eef703bcd3d032) (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServiceV2.md

              People

              • Assignee:
                rohithsharma Rohith Sharma K S
                Reporter:
                rohithsharma Rohith Sharma K S
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Development