Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-8567

Fetching yarn logs fails for long running application if it is not present in timeline store

Details

    • Reviewed

    Description

      Using yarn logs command for a long running application which has been running longer than the configured timeline service ttl yarn.timeline-service.ttl-ms  fails with the following exception.

      Exception in thread "main" org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException: The entity for application application_152347939332_00001 doesn't exist in the timeline store
      at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerOnTimelineStore.getApplication(ApplicationHistoryManagerOnTimelineStore.java:670)
      at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerOnTimelineStore.getContainers(ApplicationHistoryManagerOnTimelineStore.java:219)
      at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryClientService.getContainers(ApplicationHistoryClientService.java:211)
      at org.apache.hadoop.yarn.api.impl.pb.service.ApplicationHistoryProtocolPBServiceImpl.getContainers(ApplicationHistoryProtocolPBServiceImpl.java:172)
      at org.apache.hadoop.yarn.proto.ApplicationHistoryProtocol$ApplicationHistoryProtocolService$2.callBlockingMethod(ApplicationHistoryProtocol.java:201)
      at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640)
      at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
      at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2313)
      at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2309)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:422)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
      at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2309)
      
      at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
      at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
      at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
      at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
      at org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
      at org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:101)
      at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationHistoryProtocolPBClientImpl.getContainers(ApplicationHistoryProtocolPBClientImpl.java:183)
      at org.apache.hadoop.yarn.client.api.impl.AHSClientImpl.getContainers(AHSClientImpl.java:151)
      at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getContainers(YarnClientImpl.java:720)
      at org.apache.hadoop.yarn.client.cli.LogsCLI.getContainerReportsFromRunningApplication(LogsCLI.java:1089)
      at org.apache.hadoop.yarn.client.cli.LogsCLI.getContainersLogRequestForRunningApplication(LogsCLI.java:1064)
      at org.apache.hadoop.yarn.client.cli.LogsCLI.fetchApplicationLogs(LogsCLI.java:976)
      at org.apache.hadoop.yarn.client.cli.LogsCLI.runCommand(LogsCLI.java:300)
      at org.apache.hadoop.yarn.client.cli.LogsCLI.run(LogsCLI.java:107)
      at org.apache.hadoop.yarn.client.cli.LogsCLI.main(LogsCLI.java:327)
      

      Attachments

        1. YARN-8567.002.patch
          4 kB
          Tarun Parimi
        2. YARN-8567.001.patch
          1.0 kB
          Tarun Parimi

        Activity

          tarunparimi Tarun Parimi added a comment -

          AHSClientImpl#getContainers failed because the application entity got deleted as it exceeded yarn.timeline-service.ttl-ms . 

          I checked in the debug logs that ClientRMService#getContainers is successful since the application is still running and is present in the ResourceManager. 

          We seem to be only catching IOException here. Ideally we should catch YarnException also in this case so that the response from RM is at least returned if the application is found in RM. Attaching a patch for the same.

          tarunparimi Tarun Parimi added a comment - AHSClientImpl#getContainers failed because the application entity got deleted as it exceeded  yarn.timeline-service.ttl-ms .   I checked in the debug logs that ClientRMService#getContainers is successful since the application is still running and is present in the ResourceManager.  We seem to be only catching IOException here. Ideally we should catch YarnException also in this case so that the response from RM is at least returned if the application is found in RM. Attaching a patch for the same.
          genericqa genericqa added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 26s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
                trunk Compile Tests
          +1 mvninstall 27m 40s trunk passed
          +1 compile 0m 31s trunk passed
          +1 checkstyle 0m 13s trunk passed
          +1 mvnsite 0m 32s trunk passed
          +1 shadedclient 11m 50s branch has no errors when building and testing our client artifacts.
          +1 findbugs 0m 37s trunk passed
          +1 javadoc 0m 20s trunk passed
                Patch Compile Tests
          +1 mvninstall 0m 29s the patch passed
          +1 compile 0m 24s the patch passed
          +1 javac 0m 24s the patch passed
          +1 checkstyle 0m 11s the patch passed
          +1 mvnsite 0m 26s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 shadedclient 12m 27s patch has no errors when building and testing our client artifacts.
          +1 findbugs 0m 43s the patch passed
          +1 javadoc 0m 18s the patch passed
                Other Tests
          +1 unit 24m 1s hadoop-yarn-client in the patch passed.
          +1 asflicense 0m 26s The patch does not generate ASF License warnings.
          81m 54s



          Subsystem Report/Notes
          Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08
          JIRA Issue YARN-8567
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12932723/YARN-8567.001.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
          uname Linux 60c67b3d24de 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14 08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/patchprocess/precommit/personality/provided.sh
          git revision trunk / bbe2f62
          maven version: Apache Maven 3.3.9
          Default Java 1.8.0_171
          findbugs v3.1.0-RC1
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/21343/testReport/
          Max. process+thread count 706 (vs. ulimit of 10000)
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/21343/console
          Powered by Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          genericqa genericqa added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 26s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.       trunk Compile Tests +1 mvninstall 27m 40s trunk passed +1 compile 0m 31s trunk passed +1 checkstyle 0m 13s trunk passed +1 mvnsite 0m 32s trunk passed +1 shadedclient 11m 50s branch has no errors when building and testing our client artifacts. +1 findbugs 0m 37s trunk passed +1 javadoc 0m 20s trunk passed       Patch Compile Tests +1 mvninstall 0m 29s the patch passed +1 compile 0m 24s the patch passed +1 javac 0m 24s the patch passed +1 checkstyle 0m 11s the patch passed +1 mvnsite 0m 26s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 shadedclient 12m 27s patch has no errors when building and testing our client artifacts. +1 findbugs 0m 43s the patch passed +1 javadoc 0m 18s the patch passed       Other Tests +1 unit 24m 1s hadoop-yarn-client in the patch passed. +1 asflicense 0m 26s The patch does not generate ASF License warnings. 81m 54s Subsystem Report/Notes Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 JIRA Issue YARN-8567 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12932723/YARN-8567.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle uname Linux 60c67b3d24de 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14 08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/patchprocess/precommit/personality/provided.sh git revision trunk / bbe2f62 maven version: Apache Maven 3.3.9 Default Java 1.8.0_171 findbugs v3.1.0-RC1 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/21343/testReport/ Max. process+thread count 706 (vs. ulimit of 10000) modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client Console output https://builds.apache.org/job/PreCommit-YARN-Build/21343/console Powered by Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          rohithsharma Rohith Sharma K S added a comment - - edited

          +1, patch doesn't seem apply.. would you rebase it and add a tests?

          rohithsharma Rohith Sharma K S added a comment - - edited +1, patch doesn't seem apply.. would you rebase it and add a tests?
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          -1 patch 0m 5s YARN-8567 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



          This message was automatically generated.

          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 5s YARN-8567 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Issue YARN-8567 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12932723/YARN-8567.001.patch Console output https://builds.apache.org/job/PreCommit-YARN-Build/22975/console Powered by Apache Yetus 0.8.0 http://yetus.apache.org This message was automatically generated.
          tarunparimi Tarun Parimi added a comment -

          Thanks for the review rohithsharma . Attached a new patch after rebasing and added a unit test.

          tarunparimi Tarun Parimi added a comment - Thanks for the review rohithsharma . Attached a new patch after rebasing and added a unit test.
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 13s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
                trunk Compile Tests
          +1 mvninstall 19m 28s trunk passed
          +1 compile 0m 31s trunk passed
          +1 checkstyle 0m 17s trunk passed
          +1 mvnsite 0m 29s trunk passed
          +1 shadedclient 11m 11s branch has no errors when building and testing our client artifacts.
          +1 findbugs 0m 34s trunk passed
          +1 javadoc 0m 16s trunk passed
                Patch Compile Tests
          +1 mvninstall 0m 27s the patch passed
          +1 compile 0m 23s the patch passed
          +1 javac 0m 23s the patch passed
          +1 checkstyle 0m 13s the patch passed
          +1 mvnsite 0m 24s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 shadedclient 11m 53s patch has no errors when building and testing our client artifacts.
          +1 findbugs 0m 43s the patch passed
          +1 javadoc 0m 16s the patch passed
                Other Tests
          -1 unit 19m 0s hadoop-yarn-client in the patch failed.
          +1 asflicense 0m 25s The patch does not generate ASF License warnings.
          66m 54s



          Reason Tests
          Failed junit tests hadoop.yarn.client.cli.TestRMAdminCLI



          Subsystem Report/Notes
          Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f
          JIRA Issue YARN-8567
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12953614/YARN-8567.002.patch
          Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
          uname Linux 206f8a2fe44e 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/patchprocess/precommit/personality/provided.sh
          git revision trunk / cb26f15
          maven version: Apache Maven 3.3.9
          Default Java 1.8.0_191
          findbugs v3.1.0-RC1
          unit https://builds.apache.org/job/PreCommit-YARN-Build/22976/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/22976/testReport/
          Max. process+thread count 590 (vs. ulimit of 10000)
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/22976/console
          Powered by Apache Yetus 0.8.0 http://yetus.apache.org

          This message was automatically generated.

          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       trunk Compile Tests +1 mvninstall 19m 28s trunk passed +1 compile 0m 31s trunk passed +1 checkstyle 0m 17s trunk passed +1 mvnsite 0m 29s trunk passed +1 shadedclient 11m 11s branch has no errors when building and testing our client artifacts. +1 findbugs 0m 34s trunk passed +1 javadoc 0m 16s trunk passed       Patch Compile Tests +1 mvninstall 0m 27s the patch passed +1 compile 0m 23s the patch passed +1 javac 0m 23s the patch passed +1 checkstyle 0m 13s the patch passed +1 mvnsite 0m 24s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 shadedclient 11m 53s patch has no errors when building and testing our client artifacts. +1 findbugs 0m 43s the patch passed +1 javadoc 0m 16s the patch passed       Other Tests -1 unit 19m 0s hadoop-yarn-client in the patch failed. +1 asflicense 0m 25s The patch does not generate ASF License warnings. 66m 54s Reason Tests Failed junit tests hadoop.yarn.client.cli.TestRMAdminCLI Subsystem Report/Notes Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f JIRA Issue YARN-8567 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12953614/YARN-8567.002.patch Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle uname Linux 206f8a2fe44e 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/patchprocess/precommit/personality/provided.sh git revision trunk / cb26f15 maven version: Apache Maven 3.3.9 Default Java 1.8.0_191 findbugs v3.1.0-RC1 unit https://builds.apache.org/job/PreCommit-YARN-Build/22976/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/22976/testReport/ Max. process+thread count 590 (vs. ulimit of 10000) modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client Console output https://builds.apache.org/job/PreCommit-YARN-Build/22976/console Powered by Apache Yetus 0.8.0 http://yetus.apache.org This message was automatically generated.
          tarunparimi Tarun Parimi added a comment -

          Test failure is unrelated to the patch and YARN-9162 has been already raised for it.

          tarunparimi Tarun Parimi added a comment - Test failure is unrelated to the patch and YARN-9162 has been already raised for it.
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #15702 (See https://builds.apache.org/job/Hadoop-trunk-Commit/15702/)
          YARN-8567. Fetching yarn logs fails for long running application if it (rohithsharmaks: rev 573b1587918c4c0efdb7e9fff6f5be12bf31b619)

          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestYarnClient.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/YarnClientImpl.java
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #15702 (See https://builds.apache.org/job/Hadoop-trunk-Commit/15702/ ) YARN-8567 . Fetching yarn logs fails for long running application if it (rohithsharmaks: rev 573b1587918c4c0efdb7e9fff6f5be12bf31b619) (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestYarnClient.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/YarnClientImpl.java

          committed to trunk! thanks tarunparimi for the patch!

          rohithsharma Rohith Sharma K S added a comment - committed to trunk! thanks tarunparimi for the patch!

          back ported to branch-3.2. updated the fix version as 3.2.1

          rohithsharma Rohith Sharma K S added a comment - back ported to branch-3.2. updated the fix version as 3.2.1

          People

            tarunparimi Tarun Parimi
            tarunparimi Tarun Parimi
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: