Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-4596

SystemMetricPublisher should not swallow error messages from TimelineClient#putEntities

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: timelineserver
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      We should report error messages from the returned TimelineResponse when posting timeline entities through system metric publisher.

        Activity

        Hide
        varun_saxena Varun Saxena added a comment -

        Naganarasimha G R, Li Lu,

        Regarding below comment,

        but if some one uses rest directly then some entities will get posted and some will not, hence wanted to know whether its right to break in the middle ?

        Not really.
        None of the entities will get posted to backend store.

        BadRequestException is thrown during entity preprocessing phase. Not during store put. We first collect all the applicable entities which can be posted and then only call TimelineStore#put. We are not calling put from within the loop. Refer to TimelineDataManager#doPostEntities.

        It can although be argued that we can post entities which can be and send a put error(new error type) for the entity which has no entity id or entity type. But if we do not have entity id or type, how will client recognize which specific entity has failed from put error ? Maybe they can cache entities which they have sent and then re-look at entities they attempted to post and find out which ones had entity id and entity type missing. But will client be doing so ? Current clients would be looking at entity id and type to find out which entities failed and why.

        Entity ID and type are key pieces of information in ATSv1. If client is not sending them, it indicates some problem in the client side code. I think current code can be kept as it is because this will be consistent with TimelineClient side code too where none of the entities will be posted if entity id or type is missing in even one of them.
        Thoughts ?

        Show
        varun_saxena Varun Saxena added a comment - Naganarasimha G R , Li Lu , Regarding below comment, but if some one uses rest directly then some entities will get posted and some will not, hence wanted to know whether its right to break in the middle ? Not really. None of the entities will get posted to backend store. BadRequestException is thrown during entity preprocessing phase. Not during store put. We first collect all the applicable entities which can be posted and then only call TimelineStore#put . We are not calling put from within the loop. Refer to TimelineDataManager#doPostEntities . It can although be argued that we can post entities which can be and send a put error(new error type) for the entity which has no entity id or entity type. But if we do not have entity id or type, how will client recognize which specific entity has failed from put error ? Maybe they can cache entities which they have sent and then re-look at entities they attempted to post and find out which ones had entity id and entity type missing. But will client be doing so ? Current clients would be looking at entity id and type to find out which entities failed and why. Entity ID and type are key pieces of information in ATSv1. If client is not sending them, it indicates some problem in the client side code. I think current code can be kept as it is because this will be consistent with TimelineClient side code too where none of the entities will be posted if entity id or type is missing in even one of them. Thoughts ?
        Hide
        Naganarasimha Naganarasimha G R added a comment -

        Thanks Li Lu (inactive) for considering them !

        Show
        Naganarasimha Naganarasimha G R added a comment - Thanks Li Lu (inactive) for considering them !
        Hide
        gtCarrera9 Li Lu added a comment -

        I've created YARN-4604 and MAPREDUCE-6610 to trace the two problems raised in the discussion of this issue. Thanks Naganarasimha G R!

        Show
        gtCarrera9 Li Lu added a comment - I've created YARN-4604 and MAPREDUCE-6610 to trace the two problems raised in the discussion of this issue. Thanks Naganarasimha G R !
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #9133 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9133/)
        YARN-4596. SystemMetricPublisher should not swallow error messages from (jianhe: rev f385851141522633184ce394899c659af5ace92a)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/SystemMetricsPublisher.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java
        • hadoop-yarn-project/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9133 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9133/ ) YARN-4596 . SystemMetricPublisher should not swallow error messages from (jianhe: rev f385851141522633184ce394899c659af5ace92a) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/SystemMetricsPublisher.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java hadoop-yarn-project/CHANGES.txt
        Hide
        jianhe Jian He added a comment -

        Committed to trunk, branch-2, branch-2.8 , thanks Li !

        thanks Naganarasimha G R for reviewing the patch !

        Show
        jianhe Jian He added a comment - Committed to trunk, branch-2, branch-2.8 , thanks Li ! thanks Naganarasimha G R for reviewing the patch !
        Hide
        gtCarrera9 Li Lu added a comment -

        Ah I've got your point. Yes we can definitely improve the logic here. Please feel free to raise a JIRA about it.

        Show
        gtCarrera9 Li Lu added a comment - Ah I've got your point. Yes we can definitely improve the logic here. Please feel free to raise a JIRA about it.
        Hide
        Naganarasimha Naganarasimha G R added a comment -

        Thanks for the comments Li Lu (inactive), MR side fix is a small one, if possible to include in this jira then fine if not can raise and fix in another jira.
        for the second point actually in timeline client side its not a problem as there is similar check before posting but if some one uses rest directly then some entities will get posted and some will not, hence wanted to know whether its right to break in the middle ?

        Show
        Naganarasimha Naganarasimha G R added a comment - Thanks for the comments Li Lu (inactive) , MR side fix is a small one, if possible to include in this jira then fine if not can raise and fix in another jira. for the second point actually in timeline client side its not a problem as there is similar check before posting but if some one uses rest directly then some entities will get posted and some will not, hence wanted to know whether its right to break in the middle ?
        Hide
        gtCarrera9 Li Lu added a comment -

        BTW, the second point also appears to be orthogonal to the current problem. We can further decide (and hopefully streamline) the concrete post-effect of the putEntities() API. This will also assist us polish APIs for v2.

        Show
        gtCarrera9 Li Lu added a comment - BTW, the second point also appears to be orthogonal to the current problem. We can further decide (and hopefully streamline) the concrete post-effect of the putEntities() API. This will also assist us polish APIs for v2.
        Hide
        gtCarrera9 Li Lu added a comment -

        Naganarasimha G R Yes we can handle this in a separate JIRA (please feel free to work on this). TDM throws BadRequestException which represents a bad REST request. This is a connection level exception rather than a semantic level error (in using the timeline API). Therefore I prefer to leave it as the current behavior.

        Show
        gtCarrera9 Li Lu added a comment - Naganarasimha G R Yes we can handle this in a separate JIRA (please feel free to work on this). TDM throws BadRequestException which represents a bad REST request. This is a connection level exception rather than a semantic level error (in using the timeline API). Therefore I prefer to leave it as the current behavior.
        Hide
        Naganarasimha Naganarasimha G R added a comment -

        Hi Li Lu (inactive),
        Overall the patch looks fine but few issues (not exactly related to this patch but just felt lil related)

        • JobHistoryEventHandler.processEventForTimelineServer is also swallowing the error message, so may be we can handle in another jira?
        • In TimelineDataManager.doPostEntities(TimelineEntities, UserGroupInformation) throws BadRequestException when entity.getEntityId() == null || entity.getEntityType() == null should this not be captured in TimelinePutResponse.TimelinePutError and continue with next entity?
        Show
        Naganarasimha Naganarasimha G R added a comment - Hi Li Lu (inactive) , Overall the patch looks fine but few issues (not exactly related to this patch but just felt lil related) JobHistoryEventHandler.processEventForTimelineServer is also swallowing the error message, so may be we can handle in another jira? In TimelineDataManager.doPostEntities(TimelineEntities, UserGroupInformation) throws BadRequestException when entity.getEntityId() == null || entity.getEntityType() == null should this not be captured in TimelinePutResponse.TimelinePutError and continue with next entity?
        Hide
        Naganarasimha Naganarasimha G R added a comment -

        Li Lu (inactive), Oops i understood it wrongly, got what you are trying to address !

        Show
        Naganarasimha Naganarasimha G R added a comment - Li Lu (inactive) , Oops i understood it wrongly, got what you are trying to address !
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        0 mvndep 0m 17s Maven dependency ordering for branch
        +1 mvninstall 7m 58s trunk passed
        +1 compile 2m 7s trunk passed with JDK v1.8.0_66
        +1 compile 2m 13s trunk passed with JDK v1.7.0_91
        +1 checkstyle 0m 29s trunk passed
        +1 mvnsite 0m 59s trunk passed
        +1 mvneclipse 0m 29s trunk passed
        +1 findbugs 1m 49s trunk passed
        +1 javadoc 0m 35s trunk passed with JDK v1.8.0_66
        +1 javadoc 0m 45s trunk passed with JDK v1.7.0_91
        0 mvndep 0m 18s Maven dependency ordering for patch
        +1 mvninstall 0m 48s the patch passed
        +1 compile 1m 58s the patch passed with JDK v1.8.0_66
        +1 javac 1m 58s the patch passed
        +1 compile 2m 14s the patch passed with JDK v1.7.0_91
        +1 javac 2m 14s the patch passed
        +1 checkstyle 0m 30s the patch passed
        +1 mvnsite 0m 56s the patch passed
        +1 mvneclipse 0m 25s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 1m 54s the patch passed
        +1 javadoc 0m 32s the patch passed with JDK v1.8.0_66
        +1 javadoc 0m 40s the patch passed with JDK v1.7.0_91
        -1 unit 61m 32s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_66.
        +1 unit 6m 55s hadoop-yarn-applications-distributedshell in the patch passed with JDK v1.8.0_66.
        -1 unit 63m 29s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_91.
        +1 unit 7m 7s hadoop-yarn-applications-distributedshell in the patch passed with JDK v1.7.0_91.
        +1 asflicense 0m 20s Patch does not generate ASF License warnings.
        168m 45s



        Reason Tests
        JDK v1.8.0_66 Failed junit tests hadoop.yarn.server.resourcemanager.TestAMAuthorization
          hadoop.yarn.server.resourcemanager.TestClientRMTokens
        JDK v1.7.0_91 Failed junit tests hadoop.yarn.server.resourcemanager.TestAMAuthorization
          hadoop.yarn.server.resourcemanager.TestClientRMTokens



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12782615/YARN-4596-trunk.002.patch
        JIRA Issue YARN-4596
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 5a916532529c 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 2a30386
        Default Java 1.7.0_91
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-YARN-Build/10308/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/10308/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_91.txt
        unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10308/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-YARN-Build/10308/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_91.txt
        JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10308/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell U: hadoop-yarn-project/hadoop-yarn
        Max memory used 76MB
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/10308/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 0m 17s Maven dependency ordering for branch +1 mvninstall 7m 58s trunk passed +1 compile 2m 7s trunk passed with JDK v1.8.0_66 +1 compile 2m 13s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 29s trunk passed +1 mvnsite 0m 59s trunk passed +1 mvneclipse 0m 29s trunk passed +1 findbugs 1m 49s trunk passed +1 javadoc 0m 35s trunk passed with JDK v1.8.0_66 +1 javadoc 0m 45s trunk passed with JDK v1.7.0_91 0 mvndep 0m 18s Maven dependency ordering for patch +1 mvninstall 0m 48s the patch passed +1 compile 1m 58s the patch passed with JDK v1.8.0_66 +1 javac 1m 58s the patch passed +1 compile 2m 14s the patch passed with JDK v1.7.0_91 +1 javac 2m 14s the patch passed +1 checkstyle 0m 30s the patch passed +1 mvnsite 0m 56s the patch passed +1 mvneclipse 0m 25s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 54s the patch passed +1 javadoc 0m 32s the patch passed with JDK v1.8.0_66 +1 javadoc 0m 40s the patch passed with JDK v1.7.0_91 -1 unit 61m 32s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_66. +1 unit 6m 55s hadoop-yarn-applications-distributedshell in the patch passed with JDK v1.8.0_66. -1 unit 63m 29s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_91. +1 unit 7m 7s hadoop-yarn-applications-distributedshell in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 20s Patch does not generate ASF License warnings. 168m 45s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.TestClientRMTokens JDK v1.7.0_91 Failed junit tests hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.TestClientRMTokens Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12782615/YARN-4596-trunk.002.patch JIRA Issue YARN-4596 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 5a916532529c 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 2a30386 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-YARN-Build/10308/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10308/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10308/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-YARN-Build/10308/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10308/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell U: hadoop-yarn-project/hadoop-yarn Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/10308/console This message was automatically generated.
        Hide
        jianhe Jian He added a comment -

        latest patch looks good to me.

        Show
        jianhe Jian He added a comment - latest patch looks good to me.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 8m 4s trunk passed
        +1 compile 0m 29s trunk passed with JDK v1.8.0_66
        +1 compile 0m 32s trunk passed with JDK v1.7.0_91
        +1 checkstyle 0m 14s trunk passed
        +1 mvnsite 0m 38s trunk passed
        +1 mvneclipse 0m 16s trunk passed
        +1 findbugs 1m 17s trunk passed
        +1 javadoc 0m 23s trunk passed with JDK v1.8.0_66
        +1 javadoc 0m 29s trunk passed with JDK v1.7.0_91
        +1 mvninstall 0m 33s the patch passed
        +1 compile 0m 28s the patch passed with JDK v1.8.0_66
        +1 javac 0m 28s the patch passed
        +1 compile 0m 30s the patch passed with JDK v1.7.0_91
        +1 javac 0m 30s the patch passed
        +1 checkstyle 0m 15s the patch passed
        +1 mvnsite 0m 36s the patch passed
        +1 mvneclipse 0m 13s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 1m 22s the patch passed
        +1 javadoc 0m 21s the patch passed with JDK v1.8.0_66
        +1 javadoc 0m 26s the patch passed with JDK v1.7.0_91
        -1 unit 61m 30s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_66.
        -1 unit 62m 45s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_91.
        +1 asflicense 0m 20s Patch does not generate ASF License warnings.
        142m 46s



        Reason Tests
        JDK v1.8.0_66 Failed junit tests hadoop.yarn.server.resourcemanager.TestAMAuthorization
          hadoop.yarn.server.resourcemanager.TestClientRMTokens
        JDK v1.7.0_91 Failed junit tests hadoop.yarn.server.resourcemanager.TestAMAuthorization
          hadoop.yarn.server.resourcemanager.TestClientRMTokens



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12782590/YARN-4596-trunk.001.patch
        JIRA Issue YARN-4596
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux edbab8d89e6e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / c07f7fa
        Default Java 1.7.0_91
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-YARN-Build/10305/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/10305/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_91.txt
        unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10305/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-YARN-Build/10305/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_91.txt
        JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10305/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Max memory used 76MB
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/10305/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 8m 4s trunk passed +1 compile 0m 29s trunk passed with JDK v1.8.0_66 +1 compile 0m 32s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 14s trunk passed +1 mvnsite 0m 38s trunk passed +1 mvneclipse 0m 16s trunk passed +1 findbugs 1m 17s trunk passed +1 javadoc 0m 23s trunk passed with JDK v1.8.0_66 +1 javadoc 0m 29s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 33s the patch passed +1 compile 0m 28s the patch passed with JDK v1.8.0_66 +1 javac 0m 28s the patch passed +1 compile 0m 30s the patch passed with JDK v1.7.0_91 +1 javac 0m 30s the patch passed +1 checkstyle 0m 15s the patch passed +1 mvnsite 0m 36s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 22s the patch passed +1 javadoc 0m 21s the patch passed with JDK v1.8.0_66 +1 javadoc 0m 26s the patch passed with JDK v1.7.0_91 -1 unit 61m 30s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_66. -1 unit 62m 45s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_91. +1 asflicense 0m 20s Patch does not generate ASF License warnings. 142m 46s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.TestClientRMTokens JDK v1.7.0_91 Failed junit tests hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.TestClientRMTokens Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12782590/YARN-4596-trunk.001.patch JIRA Issue YARN-4596 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux edbab8d89e6e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / c07f7fa Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-YARN-Build/10305/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/10305/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/10305/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-YARN-Build/10305/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/10305/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/10305/console This message was automatically generated.
        Hide
        gtCarrera9 Li Lu added a comment -

        Just found distributed shell has the same problem. Update the patch to address them together.

        Show
        gtCarrera9 Li Lu added a comment - Just found distributed shell has the same problem. Update the patch to address them together.
        Hide
        gtCarrera9 Li Lu added a comment -

        A quick patch to log error messages in a TimelineResponse for SMP.

        Show
        gtCarrera9 Li Lu added a comment - A quick patch to log error messages in a TimelineResponse for SMP.
        Hide
        gtCarrera9 Li Lu added a comment -

        Hi Naganarasimha G R actually I was referring to the logic we have in the SMP itself, where we simply do

              client.putEntities(entity);
        

        when we actually post entities. However, timeline stores may put error messages in the response to indicate failures/errors when putting these entities. The SMP needs to at least log down those problems. Otherwise it's hard to reason about whether a putEntities call has actually landed onto the timeline storage.

        The code you pointed out is on the client side, and is taking care of the case when there is no (HTTP) response from the timeline server. If there are errors happened in timeline store, the HTTP response should be fine but the actual TimelinePutResponse will contain the error message.

        Show
        gtCarrera9 Li Lu added a comment - Hi Naganarasimha G R actually I was referring to the logic we have in the SMP itself, where we simply do client.putEntities(entity); when we actually post entities. However, timeline stores may put error messages in the response to indicate failures/errors when putting these entities. The SMP needs to at least log down those problems. Otherwise it's hard to reason about whether a putEntities call has actually landed onto the timeline storage. The code you pointed out is on the client side, and is taking care of the case when there is no (HTTP) response from the timeline server. If there are errors happened in timeline store, the HTTP response should be fine but the actual TimelinePutResponse will contain the error message.
        Hide
        Naganarasimha Naganarasimha G R added a comment -

        Hi Li Lu (inactive) IIUC you are referring to org.apache.hadoop.yarn.client.api.impl.TimelineWriter.doPosting(Object, String)

         if (resp == null ||
                resp.getClientResponseStatus() != ClientResponse.Status.OK){
        ...
        }
        

        we need to check if resp Obj is available (not null) and if so then we need to try to get resp.getEntity(TimelinePutResponse.class) and may be have a proper toString overrides to log the errors rights?

        Show
        Naganarasimha Naganarasimha G R added a comment - Hi Li Lu (inactive) IIUC you are referring to org.apache.hadoop.yarn.client.api.impl.TimelineWriter.doPosting(Object, String) if (resp == null || resp.getClientResponseStatus() != ClientResponse.Status.OK){ ... } we need to check if resp Obj is available (not null) and if so then we need to try to get resp.getEntity(TimelinePutResponse.class) and may be have a proper toString overrides to log the errors rights?

          People

          • Assignee:
            gtCarrera9 Li Lu
            Reporter:
            gtCarrera9 Li Lu
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development