Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-alpha2
    • Component/s: scheduler
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      This JIRA intend to provide a lifetime monitor service.
      The service will monitor the applications where the life time is configured. If the application is running beyond the lifetime, it will be killed.
      The lifetime will be considered from the submit time.

      The thread monitoring interval is configurable.

      1. 0001-YARN-4205.patch
        34 kB
        Rohith Sharma K S
      2. 0002-YARN-4205.patch
        35 kB
        Rohith Sharma K S
      3. 0003-YARN-4205.patch
        49 kB
        Rohith Sharma K S
      4. 0004-YARN-4205.patch
        49 kB
        Rohith Sharma K S
      5. 0005-YARN-4205.patch
        49 kB
        Rohith Sharma K S
      6. 0006-YARN-4205.patch
        49 kB
        Rohith Sharma K S
      7. 0007-YARN-4205.1.patch
        54 kB
        Rohith Sharma K S
      8. 0007-YARN-4205.2.patch
        57 kB
        Rohith Sharma K S
      9. 0007-YARN-4205.patch
        46 kB
        Rohith Sharma K S
      10. YARN-4205_01.patch
        34 kB
        nijel
      11. YARN-4205_02.patch
        34 kB
        nijel
      12. YARN-4205_03.patch
        34 kB
        nijel
      13. YARN-4205-addendum.001.patch
        1 kB
        Li Lu

        Issue Links

          Activity

          Hide
          nijel nijel added a comment -

          Uploading initial version.

          Show
          nijel nijel added a comment - Uploading initial version.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 19m 58s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 3 new or modified test files.
          +1 javac 7m 54s There were no new javac warning messages.
          +1 javadoc 10m 15s There were no new javadoc warning messages.
          +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 54s The applied patch generated 1 new checkstyle issues (total was 238, now 238).
          -1 whitespace 0m 2s The patch has 4 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 32s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 4m 41s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          -1 yarn tests 0m 21s Tests failed in hadoop-yarn-api.
          +1 yarn tests 2m 2s Tests passed in hadoop-yarn-common.
          -1 yarn tests 139m 40s Tests failed in hadoop-yarn-server-resourcemanager.
              189m 51s  



          Reason Tests
          Failed unit tests hadoop.yarn.conf.TestYarnConfigurationFields
            hadoop.yarn.server.resourcemanager.TestApplicationCleanup
            hadoop.yarn.server.resourcemanager.TestApplicationMasterService
            hadoop.yarn.server.resourcemanager.rmcontainer.TestRMContainerImpl
            hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRMRPCNodeUpdates
            hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRMRPCResponseId
            hadoop.yarn.server.resourcemanager.TestKillApplicationWithRMHA
            hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySched
            hadoop.yarn.server.resourcemanager.TestApplicationACLs
            hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart
            hadoop.yarn.server.resourcemanager.security.TestClientToAMTokens
            hadoop.yarn.server.resourcemanager.TestClientRMService
            hadoop.yarn.server.resourcemanager.ahs.TestRMApplicationHistoryWriter
            hadoop.yarn.server.resourcemanager.TestRMHA
            hadoop.yarn.server.resourcemanager.rmapp.TestNodesListManager
            hadoop.yarn.server.resourcemanager.logaggregationstatus.TestRMAppLogAggregationStatus
            hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer
            hadoop.yarn.server.resourcemanager.security.TestAMRMTokens
          Timed out tests org.apache.hadoop.yarn.server.resourcemanager.TestRM
            org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart
            org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesAppsModification
            org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesApps



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12762177/YARN-4205_01.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 71a81b6
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9256/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
          whitespace https://builds.apache.org/job/PreCommit-YARN-Build/9256/artifact/patchprocess/whitespace.txt
          hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/9256/artifact/patchprocess/testrun_hadoop-yarn-api.txt
          hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/9256/artifact/patchprocess/testrun_hadoop-yarn-common.txt
          hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9256/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9256/testReport/
          Java 1.7.0_55
          uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/9256/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 19m 58s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 3 new or modified test files. +1 javac 7m 54s There were no new javac warning messages. +1 javadoc 10m 15s There were no new javadoc warning messages. +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 54s The applied patch generated 1 new checkstyle issues (total was 238, now 238). -1 whitespace 0m 2s The patch has 4 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 32s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 4m 41s The patch does not introduce any new Findbugs (version 3.0.0) warnings. -1 yarn tests 0m 21s Tests failed in hadoop-yarn-api. +1 yarn tests 2m 2s Tests passed in hadoop-yarn-common. -1 yarn tests 139m 40s Tests failed in hadoop-yarn-server-resourcemanager.     189m 51s   Reason Tests Failed unit tests hadoop.yarn.conf.TestYarnConfigurationFields   hadoop.yarn.server.resourcemanager.TestApplicationCleanup   hadoop.yarn.server.resourcemanager.TestApplicationMasterService   hadoop.yarn.server.resourcemanager.rmcontainer.TestRMContainerImpl   hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRMRPCNodeUpdates   hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRMRPCResponseId   hadoop.yarn.server.resourcemanager.TestKillApplicationWithRMHA   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySched   hadoop.yarn.server.resourcemanager.TestApplicationACLs   hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart   hadoop.yarn.server.resourcemanager.security.TestClientToAMTokens   hadoop.yarn.server.resourcemanager.TestClientRMService   hadoop.yarn.server.resourcemanager.ahs.TestRMApplicationHistoryWriter   hadoop.yarn.server.resourcemanager.TestRMHA   hadoop.yarn.server.resourcemanager.rmapp.TestNodesListManager   hadoop.yarn.server.resourcemanager.logaggregationstatus.TestRMAppLogAggregationStatus   hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer   hadoop.yarn.server.resourcemanager.security.TestAMRMTokens Timed out tests org.apache.hadoop.yarn.server.resourcemanager.TestRM   org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart   org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesAppsModification   org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesApps Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12762177/YARN-4205_01.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 71a81b6 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9256/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/9256/artifact/patchprocess/whitespace.txt hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/9256/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/9256/artifact/patchprocess/testrun_hadoop-yarn-common.txt hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9256/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9256/testReport/ Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/9256/console This message was automatically generated.
          Hide
          leftnoteasy Wangda Tan added a comment -

          Comments:

          • Could you add a short desc of this JIRA?
          • LOG.debug("Starting RMAppLifeTimeMonitorThread."); need add isDebugEnabled check
          • RMAppLifeTimeMonitorService.rmApps -> applicationIdToLifetime? (or shorter name if you prefer), it's not rmApps actually
          • life time or lifetime? I prefer "life time", "LifeTime", "lifetime" in your patch, I prefer use "lifetime" for consistency.
          • I can found some "timeout" in your patch, use "lifetime" instead?
          • public synchronized void unregister, synchronized could be removed?
          Show
          leftnoteasy Wangda Tan added a comment - Comments: Could you add a short desc of this JIRA? LOG.debug("Starting RMAppLifeTimeMonitorThread."); need add isDebugEnabled check RMAppLifeTimeMonitorService.rmApps -> applicationIdToLifetime? (or shorter name if you prefer), it's not rmApps actually life time or lifetime? I prefer "life time", "LifeTime", "lifetime" in your patch, I prefer use "lifetime" for consistency. I can found some "timeout" in your patch, use "lifetime" instead? public synchronized void unregister , synchronized could be removed?
          Hide
          nijel nijel added a comment -

          Thanks Wangda Tan for the comments
          Sorry for the small mistakes

          Updated the patch

          RMAppLifeTimeMonitorService.rmApps -> applicationIdToLifetime? (or shorter name if you prefer), it's not rmApps actually

          changed to monitoredapps

          public synchronized void unregister, synchronized could be removed?

          Done

          Other comments are fixed.

          Show
          nijel nijel added a comment - Thanks Wangda Tan for the comments Sorry for the small mistakes Updated the patch RMAppLifeTimeMonitorService.rmApps -> applicationIdToLifetime? (or shorter name if you prefer), it's not rmApps actually changed to monitoredapps public synchronized void unregister, synchronized could be removed? Done Other comments are fixed.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 19m 10s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 2 new or modified test files.
          +1 javac 7m 59s There were no new javac warning messages.
          +1 javadoc 10m 14s There were no new javadoc warning messages.
          +1 release audit 0m 21s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 53s The applied patch generated 1 new checkstyle issues (total was 238, now 238).
          -1 whitespace 0m 3s The patch has 7 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 32s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 4m 40s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 yarn tests 0m 24s Tests passed in hadoop-yarn-api.
          +1 yarn tests 2m 6s Tests passed in hadoop-yarn-common.
          -1 yarn tests 185m 18s Tests failed in hadoop-yarn-server-resourcemanager.
              234m 45s  



          Reason Tests
          Failed unit tests hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRMRPCNodeUpdates
            hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer
            hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA
            hadoop.yarn.server.resourcemanager.TestAMAuthorization
            hadoop.yarn.server.resourcemanager.TestApplicationACLs
            hadoop.yarn.server.resourcemanager.TestApplicationMasterService
            hadoop.yarn.server.resourcemanager.scheduler.fifo.TestFifoScheduler
            hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart
            hadoop.yarn.server.resourcemanager.security.TestAMRMTokens
            hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler
            hadoop.yarn.server.resourcemanager.ahs.TestRMApplicationHistoryWriter
            hadoop.yarn.server.resourcemanager.scheduler.TestAbstractYarnScheduler
            hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySched
            hadoop.yarn.server.resourcemanager.TestClientRMService
            hadoop.yarn.server.resourcemanager.rmapp.TestRMAppTransitions
            hadoop.yarn.server.resourcemanager.TestApplicationCleanup
            hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart
            hadoop.yarn.server.resourcemanager.scheduler.TestSchedulerUtils
            hadoop.yarn.server.resourcemanager.scheduler.fair.TestAppRunnability
            hadoop.yarn.server.resourcemanager.TestApplicationMasterLauncher
            hadoop.yarn.server.resourcemanager.TestMoveApplication
            hadoop.yarn.server.resourcemanager.TestKillApplicationWithRMHA
            hadoop.yarn.server.resourcemanager.rmapp.TestNodesListManager
            hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerQueueACLs
            hadoop.yarn.server.resourcemanager.logaggregationstatus.TestRMAppLogAggregationStatus
            hadoop.yarn.server.resourcemanager.TestResourceManager
            hadoop.yarn.server.resourcemanager.TestContainerResourceUsage
            hadoop.yarn.server.resourcemanager.scheduler.TestSchedulerHealth
            hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRMRPCResponseId
            hadoop.yarn.server.resourcemanager.security.TestClientToAMTokens
            hadoop.yarn.server.resourcemanager.rmapp.TestApplicationLifeTimeMonitorService
            hadoop.yarn.server.resourcemanager.TestResourceTrackerService
            hadoop.yarn.server.resourcemanager.scheduler.capacity.TestApplicationPriority
            hadoop.yarn.server.resourcemanager.rmcontainer.TestRMContainerImpl
          Timed out tests org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation
            org.apache.hadoop.yarn.server.resourcemanager.TestRM
            org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesAppsModification
            org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesApps



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12762396/YARN-4205_02.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 83e65c5
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9265/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
          whitespace https://builds.apache.org/job/PreCommit-YARN-Build/9265/artifact/patchprocess/whitespace.txt
          hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/9265/artifact/patchprocess/testrun_hadoop-yarn-api.txt
          hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/9265/artifact/patchprocess/testrun_hadoop-yarn-common.txt
          hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9265/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9265/testReport/
          Java 1.7.0_55
          uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/9265/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 19m 10s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 2 new or modified test files. +1 javac 7m 59s There were no new javac warning messages. +1 javadoc 10m 14s There were no new javadoc warning messages. +1 release audit 0m 21s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 53s The applied patch generated 1 new checkstyle issues (total was 238, now 238). -1 whitespace 0m 3s The patch has 7 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 32s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 4m 40s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 0m 24s Tests passed in hadoop-yarn-api. +1 yarn tests 2m 6s Tests passed in hadoop-yarn-common. -1 yarn tests 185m 18s Tests failed in hadoop-yarn-server-resourcemanager.     234m 45s   Reason Tests Failed unit tests hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRMRPCNodeUpdates   hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer   hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA   hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.TestApplicationACLs   hadoop.yarn.server.resourcemanager.TestApplicationMasterService   hadoop.yarn.server.resourcemanager.scheduler.fifo.TestFifoScheduler   hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart   hadoop.yarn.server.resourcemanager.security.TestAMRMTokens   hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler   hadoop.yarn.server.resourcemanager.ahs.TestRMApplicationHistoryWriter   hadoop.yarn.server.resourcemanager.scheduler.TestAbstractYarnScheduler   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySched   hadoop.yarn.server.resourcemanager.TestClientRMService   hadoop.yarn.server.resourcemanager.rmapp.TestRMAppTransitions   hadoop.yarn.server.resourcemanager.TestApplicationCleanup   hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart   hadoop.yarn.server.resourcemanager.scheduler.TestSchedulerUtils   hadoop.yarn.server.resourcemanager.scheduler.fair.TestAppRunnability   hadoop.yarn.server.resourcemanager.TestApplicationMasterLauncher   hadoop.yarn.server.resourcemanager.TestMoveApplication   hadoop.yarn.server.resourcemanager.TestKillApplicationWithRMHA   hadoop.yarn.server.resourcemanager.rmapp.TestNodesListManager   hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerQueueACLs   hadoop.yarn.server.resourcemanager.logaggregationstatus.TestRMAppLogAggregationStatus   hadoop.yarn.server.resourcemanager.TestResourceManager   hadoop.yarn.server.resourcemanager.TestContainerResourceUsage   hadoop.yarn.server.resourcemanager.scheduler.TestSchedulerHealth   hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRMRPCResponseId   hadoop.yarn.server.resourcemanager.security.TestClientToAMTokens   hadoop.yarn.server.resourcemanager.rmapp.TestApplicationLifeTimeMonitorService   hadoop.yarn.server.resourcemanager.TestResourceTrackerService   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestApplicationPriority   hadoop.yarn.server.resourcemanager.rmcontainer.TestRMContainerImpl Timed out tests org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation   org.apache.hadoop.yarn.server.resourcemanager.TestRM   org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesAppsModification   org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesApps Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12762396/YARN-4205_02.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 83e65c5 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9265/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/9265/artifact/patchprocess/whitespace.txt hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/9265/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/9265/artifact/patchprocess/testrun_hadoop-yarn-common.txt hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9265/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9265/testReport/ Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/9265/console This message was automatically generated.
          Hide
          nijel nijel added a comment -

          Test cases failing as "method not found" for the method added in api project.
          These tests passing locally !

          I am not getting the reason for this fail. Any issue with build can cause this ?

          Show
          nijel nijel added a comment - Test cases failing as "method not found" for the method added in api project. These tests passing locally ! I am not getting the reason for this fail. Any issue with build can cause this ?
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          It seems build machine has issue. I will kick off the jenkins again. I applied the patch locally and run the tests. All tests cases are running.
          Some comments

          1. In RMAppLifeTimeMonitorService : Along with if (!application.isAppFinalStateStored()) {, also should handle states such as FINAL_SAVING, KILLING. Probably these states can be predefined in static variables.
          2. In test, some of the commented lines can be removed if not used.
          Show
          rohithsharma Rohith Sharma K S added a comment - It seems build machine has issue. I will kick off the jenkins again. I applied the patch locally and run the tests. All tests cases are running. Some comments In RMAppLifeTimeMonitorService : Along with if (!application.isAppFinalStateStored()) { , also should handle states such as FINAL_SAVING, KILLING. Probably these states can be predefined in static variables. In test, some of the commented lines can be removed if not used.
          Hide
          nijel nijel added a comment -

          Thanks Rohith Sharma K S for the comments
          Updated the patch.

          Show
          nijel nijel added a comment - Thanks Rohith Sharma K S for the comments Updated the patch.
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          Hi nijel the patch appears to be required to rebase to apply cleanly. Would you rebase the patch please?

          Show
          rohithsharma Rohith Sharma K S added a comment - Hi nijel the patch appears to be required to rebase to apply cleanly. Would you rebase the patch please?
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          -1 patch 0m 4s YARN-4205 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



          Subsystem Report/Notes
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12764043/YARN-4205_03.patch
          JIRA Issue YARN-4205
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/10561/console
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 4s YARN-4205 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12764043/YARN-4205_03.patch JIRA Issue YARN-4205 Console output https://builds.apache.org/job/PreCommit-YARN-Build/10561/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          Its been long time rebasing the patch. I will rebase the patch.

          Show
          rohithsharma Rohith Sharma K S added a comment - Its been long time rebasing the patch. I will rebase the patch.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          -1 patch 0m 7s YARN-4205 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



          Subsystem Report/Notes
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12764043/YARN-4205_03.patch
          JIRA Issue YARN-4205
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/12942/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 7s YARN-4205 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12764043/YARN-4205_03.patch JIRA Issue YARN-4205 Console output https://builds.apache.org/job/PreCommit-YARN-Build/12942/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          Rebased the patch against trunk. And I have done few changes than earlier patches.

          1. Kept consistently lifetime across all the usage.
          2. Configuration name I modified with prefix yarn.resourcemanager since it is rm specific configuration.
          3. Other minor changes is done here and there.
          Show
          rohithsharma Rohith Sharma K S added a comment - Rebased the patch against trunk. And I have done few changes than earlier patches. Kept consistently lifetime across all the usage. Configuration name I modified with prefix yarn.resourcemanager since it is rm specific configuration. Other minor changes is done here and there.
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          Sunil G Jian He Wangda Tan requesting your review on rebased patch. kindly review

          Show
          rohithsharma Rohith Sharma K S added a comment - Sunil G Jian He Wangda Tan requesting your review on rebased patch. kindly review
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 23s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 0m 10s Maven dependency ordering for branch
          +1 mvninstall 7m 26s trunk passed
          +1 compile 2m 18s trunk passed
          +1 checkstyle 0m 43s trunk passed
          +1 mvnsite 1m 38s trunk passed
          +1 mvneclipse 0m 42s trunk passed
          +1 findbugs 2m 58s trunk passed
          +1 javadoc 1m 5s trunk passed
          0 mvndep 0m 10s Maven dependency ordering for patch
          +1 mvninstall 1m 19s the patch passed
          +1 compile 2m 33s the patch passed
          +1 cc 2m 33s the patch passed
          +1 javac 2m 33s the patch passed
          -1 checkstyle 0m 51s hadoop-yarn-project/hadoop-yarn: The patch generated 14 new + 472 unchanged - 2 fixed = 486 total (was 474)
          +1 mvnsite 1m 42s the patch passed
          +1 mvneclipse 0m 39s the patch passed
          -1 whitespace 0m 0s The patch has 6 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          -1 findbugs 1m 10s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
          -1 javadoc 0m 16s hadoop-yarn-api in the patch failed.
          +1 unit 0m 26s hadoop-yarn-api in the patch passed.
          +1 unit 2m 21s hadoop-yarn-common in the patch passed.
          +1 unit 35m 7s hadoop-yarn-server-resourcemanager in the patch passed.
          +1 asflicense 0m 17s The patch does not generate ASF License warnings.
          68m 25s



          Reason Tests
          FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
            Possible null pointer dereference of application in org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppLifetimeMonitor$RMAppLifeTimeMonitorThread.run() on exception path Dereferenced at RMAppLifetimeMonitor.java:application in org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppLifetimeMonitor$RMAppLifeTimeMonitorThread.run() on exception path Dereferenced at RMAppLifetimeMonitor.java:[line 134]



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12826654/0001-YARN-4205.patch
          JIRA Issue YARN-4205
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc xml
          uname Linux 5ead7b8b8966 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 08f55cc
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12990/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          whitespace https://builds.apache.org/job/PreCommit-YARN-Build/12990/artifact/patchprocess/whitespace-eol.txt
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/12990/artifact/patchprocess/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.html
          javadoc https://builds.apache.org/job/PreCommit-YARN-Build/12990/artifact/patchprocess/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12990/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/12990/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 23s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 0m 10s Maven dependency ordering for branch +1 mvninstall 7m 26s trunk passed +1 compile 2m 18s trunk passed +1 checkstyle 0m 43s trunk passed +1 mvnsite 1m 38s trunk passed +1 mvneclipse 0m 42s trunk passed +1 findbugs 2m 58s trunk passed +1 javadoc 1m 5s trunk passed 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 1m 19s the patch passed +1 compile 2m 33s the patch passed +1 cc 2m 33s the patch passed +1 javac 2m 33s the patch passed -1 checkstyle 0m 51s hadoop-yarn-project/hadoop-yarn: The patch generated 14 new + 472 unchanged - 2 fixed = 486 total (was 474) +1 mvnsite 1m 42s the patch passed +1 mvneclipse 0m 39s the patch passed -1 whitespace 0m 0s The patch has 6 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 xml 0m 1s The patch has no ill-formed XML file. -1 findbugs 1m 10s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) -1 javadoc 0m 16s hadoop-yarn-api in the patch failed. +1 unit 0m 26s hadoop-yarn-api in the patch passed. +1 unit 2m 21s hadoop-yarn-common in the patch passed. +1 unit 35m 7s hadoop-yarn-server-resourcemanager in the patch passed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 68m 25s Reason Tests FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager   Possible null pointer dereference of application in org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppLifetimeMonitor$RMAppLifeTimeMonitorThread.run() on exception path Dereferenced at RMAppLifetimeMonitor.java:application in org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppLifetimeMonitor$RMAppLifeTimeMonitorThread.run() on exception path Dereferenced at RMAppLifetimeMonitor.java: [line 134] Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12826654/0001-YARN-4205.patch JIRA Issue YARN-4205 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc xml uname Linux 5ead7b8b8966 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 08f55cc Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12990/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/12990/artifact/patchprocess/whitespace-eol.txt findbugs https://builds.apache.org/job/PreCommit-YARN-Build/12990/artifact/patchprocess/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.html javadoc https://builds.apache.org/job/PreCommit-YARN-Build/12990/artifact/patchprocess/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12990/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/12990/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          This looks good in general.

          As discussed on the parent JIRA YARN-3813

          • Sunil G said

            1. we are starting the monitor thread always regardless whether application demands for applicationtimeout or not. I feel we can have a configuration to enable this feature in RM level. Thoughts?

          • And nijel said

            this service will consider only apps which are configured with a timeout. So leaving as a default service.

          I agree with this above sentiment - we don't need another configuration application.lifetimeout-monitor.interval-ms.

          The other important question is about when the clock start ticking for the life-time. Clearly if we start it when the app gets submitted, we'd automatically also handle YARN-2266. But some apps will need to distinguish queue_timeout from execution_lifetime. Can we distinguish those? Obviously one of the followup challenges is to clearly define where queuing starts/ends & execution starts/ends.

          Clearly queue_timeout should definitely not account time that YARN spends in things like persisting to state-store etc.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - This looks good in general. As discussed on the parent JIRA YARN-3813 Sunil G said 1. we are starting the monitor thread always regardless whether application demands for applicationtimeout or not. I feel we can have a configuration to enable this feature in RM level. Thoughts? And nijel said this service will consider only apps which are configured with a timeout. So leaving as a default service. I agree with this above sentiment - we don't need another configuration application.lifetimeout-monitor.interval-ms. The other important question is about when the clock start ticking for the life-time. Clearly if we start it when the app gets submitted, we'd automatically also handle YARN-2266 . But some apps will need to distinguish queue_timeout from execution_lifetime . Can we distinguish those? Obviously one of the followup challenges is to clearly define where queuing starts/ends & execution starts/ends. Clearly queue_timeout should definitely not account time that YARN spends in things like persisting to state-store etc.
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          As discussed with Jian He, thought of using AbstractMonitoring service to monitor applications. The difference from previous patch is

          1. AbstractMonitorService : added register API to pass an time.
          2. Extend the abstract monitoring service and modified such way that monitor individual object with its own expiry interval. For this, added getExpiryInterval method with default implementation to return one expiry interval. If an object has its own expiry interval then it can be extended and return it.
            #Test : modified test with right assertions.
          Show
          rohithsharma Rohith Sharma K S added a comment - As discussed with Jian He , thought of using AbstractMonitoring service to monitor applications. The difference from previous patch is AbstractMonitorService : added register API to pass an time. Extend the abstract monitoring service and modified such way that monitor individual object with its own expiry interval. For this, added getExpiryInterval method with default implementation to return one expiry interval. If an object has its own expiry interval then it can be extended and return it. #Test : modified test with right assertions.
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          I agree with this above sentiment - we don't need another configuration application.lifetimeout-monitor.interval-ms.

          IIUC, the conversation between Sunil and Nijel had for starting another thread i.e RMAppLifetimeMonitorThread. And I believe monitoring interval should be configurable at RM level.

          The other important question is about when the clock start ticking for the life-time. Clearly if we start it when the app gets submitted, we'd automatically also handle YARN-2266

          The new patch attached will be registering in NEW_SAVING transition. Basically the time for storing RMStateStore also part of application life timeout. RMstatStore storing should also part of app life time. Lifetime is considered from NEW_SAVING to completing states i.e after RUNNING any states.

          Show
          rohithsharma Rohith Sharma K S added a comment - I agree with this above sentiment - we don't need another configuration application.lifetimeout-monitor.interval-ms. IIUC, the conversation between Sunil and Nijel had for starting another thread i.e RMAppLifetimeMonitorThread. And I believe monitoring interval should be configurable at RM level. The other important question is about when the clock start ticking for the life-time. Clearly if we start it when the app gets submitted, we'd automatically also handle YARN-2266 The new patch attached will be registering in NEW_SAVING transition. Basically the time for storing RMStateStore also part of application life timeout. RMstatStore storing should also part of app life time. Lifetime is considered from NEW_SAVING to completing states i.e after RUNNING any states.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 22s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 0m 25s Maven dependency ordering for branch
          +1 mvninstall 6m 46s trunk passed
          +1 compile 2m 17s trunk passed
          +1 checkstyle 0m 44s trunk passed
          +1 mvnsite 1m 36s trunk passed
          +1 mvneclipse 0m 43s trunk passed
          +1 findbugs 2m 52s trunk passed
          +1 javadoc 1m 5s trunk passed
          0 mvndep 0m 9s Maven dependency ordering for patch
          +1 mvninstall 1m 18s the patch passed
          +1 compile 2m 16s the patch passed
          +1 cc 2m 16s the patch passed
          +1 javac 2m 16s the patch passed
          -1 checkstyle 0m 43s hadoop-yarn-project/hadoop-yarn: The patch generated 14 new + 475 unchanged - 3 fixed = 489 total (was 478)
          +1 mvnsite 1m 27s the patch passed
          +1 mvneclipse 0m 36s the patch passed
          -1 whitespace 0m 0s The patch has 7 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 xml 0m 2s The patch has no ill-formed XML file.
          +1 findbugs 3m 9s the patch passed
          -1 javadoc 0m 15s hadoop-yarn-api in the patch failed.
          +1 unit 0m 23s hadoop-yarn-api in the patch passed.
          +1 unit 2m 16s hadoop-yarn-common in the patch passed.
          +1 unit 38m 50s hadoop-yarn-server-resourcemanager in the patch passed.
          +1 asflicense 0m 17s The patch does not generate ASF License warnings.
          70m 18s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12827021/0002-YARN-4205.patch
          JIRA Issue YARN-4205
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc xml
          uname Linux fd2df95033ee 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 07650bc
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13013/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          whitespace https://builds.apache.org/job/PreCommit-YARN-Build/13013/artifact/patchprocess/whitespace-eol.txt
          javadoc https://builds.apache.org/job/PreCommit-YARN-Build/13013/artifact/patchprocess/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13013/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/13013/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 22s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 0m 25s Maven dependency ordering for branch +1 mvninstall 6m 46s trunk passed +1 compile 2m 17s trunk passed +1 checkstyle 0m 44s trunk passed +1 mvnsite 1m 36s trunk passed +1 mvneclipse 0m 43s trunk passed +1 findbugs 2m 52s trunk passed +1 javadoc 1m 5s trunk passed 0 mvndep 0m 9s Maven dependency ordering for patch +1 mvninstall 1m 18s the patch passed +1 compile 2m 16s the patch passed +1 cc 2m 16s the patch passed +1 javac 2m 16s the patch passed -1 checkstyle 0m 43s hadoop-yarn-project/hadoop-yarn: The patch generated 14 new + 475 unchanged - 3 fixed = 489 total (was 478) +1 mvnsite 1m 27s the patch passed +1 mvneclipse 0m 36s the patch passed -1 whitespace 0m 0s The patch has 7 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 xml 0m 2s The patch has no ill-formed XML file. +1 findbugs 3m 9s the patch passed -1 javadoc 0m 15s hadoop-yarn-api in the patch failed. +1 unit 0m 23s hadoop-yarn-api in the patch passed. +1 unit 2m 16s hadoop-yarn-common in the patch passed. +1 unit 38m 50s hadoop-yarn-server-resourcemanager in the patch passed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 70m 18s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12827021/0002-YARN-4205.patch JIRA Issue YARN-4205 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc xml uname Linux fd2df95033ee 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 07650bc Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13013/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/13013/artifact/patchprocess/whitespace-eol.txt javadoc https://builds.apache.org/job/PreCommit-YARN-Build/13013/artifact/patchprocess/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13013/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/13013/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          jianhe Jian He added a comment -

          Thanks Rohith, looks good overall, few comments:

          • can you clarify the definition of lifetime in the API, and also mention the unit of the time in the getter API
          • RMAppRecoveredTransition: this will cause a lot of loggings for active apps on RM recovery, remove it as it is already logged in the normal run path ? or move to debug level ?
            LOG.info("Application " + app.applicationId
    + " is registered with Application lifetime monitor after recovery. "
    + "The lifetime configured is " + applicationLifetime + " seconds");
            
          • use getLong ?
                int monitorInterval = conf.getInt(
                    YarnConfiguration.RM_APPLICATION_LIFETIME_MONITOR_INTERVAL_MS,
                    YarnConfiguration.DEFAULT_RM_APPLICATION_LIFETIME_MONITOR_INTERVAL_MS);
            
          Show
          jianhe Jian He added a comment - Thanks Rohith, looks good overall, few comments: can you clarify the definition of lifetime in the API, and also mention the unit of the time in the getter API RMAppRecoveredTransition: this will cause a lot of loggings for active apps on RM recovery, remove it as it is already logged in the normal run path ? or move to debug level ? LOG.info( "Application " + app.applicationId
 + " is registered with Application lifetime monitor after recovery. " 
 + "The lifetime configured is " + applicationLifetime + " seconds" ); use getLong ? int monitorInterval = conf.getInt( YarnConfiguration.RM_APPLICATION_LIFETIME_MONITOR_INTERVAL_MS, YarnConfiguration.DEFAULT_RM_APPLICATION_LIFETIME_MONITOR_INTERVAL_MS);
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          can you clarify the definition of lifetime in the API

          Lifetime of an application is nothing but overall time at which application has run. It includes sum of RMStateStore(connection time + retry time + ... )+ Scheduler allocation delay + container run time. In other words it is execution_lifetime as Vinod mentioned.

          But some apps will need to distinguish queue_timeout from execution_lifetime. Can we distinguish those?

          In general case, RMStateStore time will be very less may be less than seconds. Most of the time spent on either allocation-delay or container-run-time. I think this need not to be handle separately.

          Obviously one of the followup challenges is to clearly define where queuing starts/ends & execution starts/ends.

          IAC, to go ahead with queue_timeout, then time can be from Attempt SUBMITTED/SCHEDULED to Attempt_removed_from_scheduler. Thoughts?

          Show
          rohithsharma Rohith Sharma K S added a comment - can you clarify the definition of lifetime in the API Lifetime of an application is nothing but overall time at which application has run. It includes sum of RMStateStore(connection time + retry time + ... )+ Scheduler allocation delay + container run time . In other words it is execution_lifetime as Vinod mentioned. But some apps will need to distinguish queue_timeout from execution_lifetime . Can we distinguish those? In general case, RMStateStore time will be very less may be less than seconds. Most of the time spent on either allocation-delay or container-run-time. I think this need not to be handle separately. Obviously one of the followup challenges is to clearly define where queuing starts/ends & execution starts/ends. IAC, to go ahead with queue_timeout, then time can be from Attempt SUBMITTED/SCHEDULED to Attempt_removed_from_scheduler. Thoughts?
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          And I believe monitoring interval should be configurable at RM level.

          Okay, that makes sense. As long as we have a reasonable default that no-one touches in practice (like the AM-liveliness configs), we are good.

          Also yarn.app.lifetime.monitor.interval-sec -> yarn.resourcemanager.app-timeouts-monitor.interval-sec ?

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - And I believe monitoring interval should be configurable at RM level. Okay, that makes sense. As long as we have a reasonable default that no-one touches in practice (like the AM-liveliness configs), we are good. Also yarn.app.lifetime.monitor.interval-sec -> yarn.resourcemanager.app-timeouts-monitor.interval-sec ?
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          Update the patch fixing review comments. This patch has following changes from previous.

          1. Added ApplicationTimeouts class that contains lifetime values. This class can be used in future to support for any other timeouts such as queu_timeout or statestore_timeout.
          Show
          rohithsharma Rohith Sharma K S added a comment - Update the patch fixing review comments. This patch has following changes from previous. Added ApplicationTimeouts class that contains lifetime values. This class can be used in future to support for any other timeouts such as queu_timeout or statestore_timeout.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 20s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          0 mvndep 0m 9s Maven dependency ordering for branch
          +1 mvninstall 6m 44s trunk passed
          +1 compile 2m 18s trunk passed
          +1 checkstyle 0m 44s trunk passed
          +1 mvnsite 1m 34s trunk passed
          +1 mvneclipse 0m 42s trunk passed
          +1 findbugs 2m 51s trunk passed
          +1 javadoc 1m 6s trunk passed
          0 mvndep 0m 10s Maven dependency ordering for patch
          +1 mvninstall 1m 17s the patch passed
          +1 compile 2m 17s the patch passed
          +1 cc 2m 17s the patch passed
          +1 javac 2m 17s the patch passed
          -1 checkstyle 0m 42s hadoop-yarn-project/hadoop-yarn: The patch generated 25 new + 503 unchanged - 3 fixed = 528 total (was 506)
          +1 mvnsite 1m 29s the patch passed
          +1 mvneclipse 0m 36s the patch passed
          -1 whitespace 0m 0s The patch has 7 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          -1 findbugs 1m 9s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
          -1 javadoc 0m 18s hadoop-yarn-api in the patch failed.
          +1 unit 0m 24s hadoop-yarn-api in the patch passed.
          +1 unit 2m 36s hadoop-yarn-common in the patch passed.
          +1 unit 45m 34s hadoop-yarn-server-resourcemanager in the patch passed.
          +1 asflicense 0m 21s The patch does not generate ASF License warnings.
          77m 17s



          Reason Tests
          FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
            Result of integer multiplication cast to long in org.apache.hadoop.yarn.api.records.ApplicationTimeouts.hashCode() At ApplicationTimeouts.java:to long in org.apache.hadoop.yarn.api.records.ApplicationTimeouts.hashCode() At ApplicationTimeouts.java:[line 77]



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12828034/0003-YARN-4205.patch
          JIRA Issue YARN-4205
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc xml
          uname Linux f6e754d86bd8 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / cc01ed70
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13083/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          whitespace https://builds.apache.org/job/PreCommit-YARN-Build/13083/artifact/patchprocess/whitespace-eol.txt
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/13083/artifact/patchprocess/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api.html
          javadoc https://builds.apache.org/job/PreCommit-YARN-Build/13083/artifact/patchprocess/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13083/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/13083/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 20s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 0m 9s Maven dependency ordering for branch +1 mvninstall 6m 44s trunk passed +1 compile 2m 18s trunk passed +1 checkstyle 0m 44s trunk passed +1 mvnsite 1m 34s trunk passed +1 mvneclipse 0m 42s trunk passed +1 findbugs 2m 51s trunk passed +1 javadoc 1m 6s trunk passed 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 1m 17s the patch passed +1 compile 2m 17s the patch passed +1 cc 2m 17s the patch passed +1 javac 2m 17s the patch passed -1 checkstyle 0m 42s hadoop-yarn-project/hadoop-yarn: The patch generated 25 new + 503 unchanged - 3 fixed = 528 total (was 506) +1 mvnsite 1m 29s the patch passed +1 mvneclipse 0m 36s the patch passed -1 whitespace 0m 0s The patch has 7 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 xml 0m 1s The patch has no ill-formed XML file. -1 findbugs 1m 9s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) -1 javadoc 0m 18s hadoop-yarn-api in the patch failed. +1 unit 0m 24s hadoop-yarn-api in the patch passed. +1 unit 2m 36s hadoop-yarn-common in the patch passed. +1 unit 45m 34s hadoop-yarn-server-resourcemanager in the patch passed. +1 asflicense 0m 21s The patch does not generate ASF License warnings. 77m 17s Reason Tests FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api   Result of integer multiplication cast to long in org.apache.hadoop.yarn.api.records.ApplicationTimeouts.hashCode() At ApplicationTimeouts.java:to long in org.apache.hadoop.yarn.api.records.ApplicationTimeouts.hashCode() At ApplicationTimeouts.java: [line 77] Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12828034/0003-YARN-4205.patch JIRA Issue YARN-4205 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc xml uname Linux f6e754d86bd8 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / cc01ed70 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13083/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/13083/artifact/patchprocess/whitespace-eol.txt findbugs https://builds.apache.org/job/PreCommit-YARN-Build/13083/artifact/patchprocess/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api.html javadoc https://builds.apache.org/job/PreCommit-YARN-Build/13083/artifact/patchprocess/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13083/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/13083/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Added ApplicationTimeouts class that contains lifetime values. This class can be used in future to support for any other timeouts such as queu_timeout or statestore_timeout.

          Makes sense.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Added ApplicationTimeouts class that contains lifetime values. This class can be used in future to support for any other timeouts such as queu_timeout or statestore_timeout. Makes sense.
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          Updated patch by fixing QA reported issues findbug and others.

          Show
          rohithsharma Rohith Sharma K S added a comment - Updated patch by fixing QA reported issues findbug and others.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 20s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          0 mvndep 0m 48s Maven dependency ordering for branch
          +1 mvninstall 6m 48s trunk passed
          +1 compile 2m 19s trunk passed
          +1 checkstyle 0m 44s trunk passed
          +1 mvnsite 1m 34s trunk passed
          +1 mvneclipse 0m 43s trunk passed
          +1 findbugs 2m 48s trunk passed
          +1 javadoc 1m 5s trunk passed
          0 mvndep 0m 9s Maven dependency ordering for patch
          +1 mvninstall 1m 19s the patch passed
          +1 compile 2m 29s the patch passed
          +1 cc 2m 29s the patch passed
          +1 javac 2m 29s the patch passed
          -1 checkstyle 0m 46s hadoop-yarn-project/hadoop-yarn: The patch generated 14 new + 503 unchanged - 3 fixed = 517 total (was 506)
          +1 mvnsite 1m 46s the patch passed
          +1 mvneclipse 0m 43s the patch passed
          -1 whitespace 0m 0s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 3m 48s the patch passed
          -1 javadoc 0m 19s hadoop-yarn-api in the patch failed.
          +1 unit 0m 29s hadoop-yarn-api in the patch passed.
          +1 unit 2m 31s hadoop-yarn-common in the patch passed.
          +1 unit 39m 6s hadoop-yarn-server-resourcemanager in the patch passed.
          +1 asflicense 0m 17s The patch does not generate ASF License warnings.
          72m 43s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12828218/0004-YARN-4205.patch
          JIRA Issue YARN-4205
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc xml
          uname Linux 9b59987ad5b3 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 729de3e
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13093/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          whitespace https://builds.apache.org/job/PreCommit-YARN-Build/13093/artifact/patchprocess/whitespace-eol.txt
          javadoc https://builds.apache.org/job/PreCommit-YARN-Build/13093/artifact/patchprocess/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13093/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/13093/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 20s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 0m 48s Maven dependency ordering for branch +1 mvninstall 6m 48s trunk passed +1 compile 2m 19s trunk passed +1 checkstyle 0m 44s trunk passed +1 mvnsite 1m 34s trunk passed +1 mvneclipse 0m 43s trunk passed +1 findbugs 2m 48s trunk passed +1 javadoc 1m 5s trunk passed 0 mvndep 0m 9s Maven dependency ordering for patch +1 mvninstall 1m 19s the patch passed +1 compile 2m 29s the patch passed +1 cc 2m 29s the patch passed +1 javac 2m 29s the patch passed -1 checkstyle 0m 46s hadoop-yarn-project/hadoop-yarn: The patch generated 14 new + 503 unchanged - 3 fixed = 517 total (was 506) +1 mvnsite 1m 46s the patch passed +1 mvneclipse 0m 43s the patch passed -1 whitespace 0m 0s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 3m 48s the patch passed -1 javadoc 0m 19s hadoop-yarn-api in the patch failed. +1 unit 0m 29s hadoop-yarn-api in the patch passed. +1 unit 2m 31s hadoop-yarn-common in the patch passed. +1 unit 39m 6s hadoop-yarn-server-resourcemanager in the patch passed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 72m 43s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12828218/0004-YARN-4205.patch JIRA Issue YARN-4205 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc xml uname Linux 9b59987ad5b3 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 729de3e Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13093/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/13093/artifact/patchprocess/whitespace-eol.txt javadoc https://builds.apache.org/job/PreCommit-YARN-Build/13093/artifact/patchprocess/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13093/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/13093/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          jianhe Jian He added a comment -

          latest patch looks good to me.

          Show
          jianhe Jian He added a comment - latest patch looks good to me.
          Hide
          gsaha Gour Saha added a comment -

          Few comments:

          [ApplicationTimeouts.java]

          This class already has timeouts in its name. Should we name the fields as lifetime (and the future ones like queueTime and stateStoreTime)? lifetime is pretty clear by itself, lifeTimeout sounds like double superlative. Thoughts?

          [YarnConfiguration.java]
            // Configurations for applicaiton life time monitor feature
            public static final String RM_APPLICATION_LIFETIME_MONITOR_INTERVAL_MS =
                RM_PREFIX + "application.lifetimeout-monitor.interval-ms";
          

          Similarly, I think lifetimeout-monitor is a mouthful. In similar lines as above, I suggest application-timeouts.lifetime-monitor.interval-ms (and the future ones like application-timeouts.queuetime-monitor.interval-ms and application-timeouts.statestoretime-monitor.interval-ms)

          [ApplicationSubmissionContext.java]
            /**
             * Get <code>ApplicationTimeouts</code> of the application.
             *
             * @param applicationTimeouts for the application.
             */
            @Public
            @Unstable
            public abstract void setApplicationTimeouts(
                ApplicationTimeouts applicationTimeouts);
          

          Please change
          Get <code>ApplicationTimeouts</code> of the application.
          to
          Set <code>ApplicationTimeouts</code> for the application.

          [yarn_protos.proto]
            optional int64 life_timeout = 1 [default = -1];
          

          life_timeout -> lifetime

          [AbstractLivelinessMonitor.java]
            public synchronized void register(O ob, long timeout) {
              running.put(ob, timeout);
          

          For the second argument, do we mean timeout or submitTime?

          [RMAppLifetimeMonitor.java]
              // Don't trigger an KILL event if application is in completed states
          

          Change to -
          // Don't trigger a KILL event if application is in any of the completed states

          [MockRM.java]

          Add a new line before the below method -

            public RMApp submitApp(int masterMemory, Priority priority,
          
          [TestApplicationLifetimeMonitor.java]
                MockNM nm1 = rm.registerNode("127.0.0.1:1234", 16 * 1024);
          and
                  new MockNM("127.0.0.1:1234", 8192, rm1.getResourceTrackerService());
          

          Hardcoded ports cause unit test parallelization challenges. Is it possible to request a free port from the OS?

          Show
          gsaha Gour Saha added a comment - Few comments: [ApplicationTimeouts.java] This class already has timeouts in its name. Should we name the fields as lifetime (and the future ones like queueTime and stateStoreTime )? lifetime is pretty clear by itself, lifeTimeout sounds like double superlative. Thoughts? [YarnConfiguration.java] // Configurations for applicaiton life time monitor feature public static final String RM_APPLICATION_LIFETIME_MONITOR_INTERVAL_MS = RM_PREFIX + "application.lifetimeout-monitor.interval-ms" ; Similarly, I think lifetimeout-monitor is a mouthful. In similar lines as above, I suggest application-timeouts.lifetime-monitor.interval-ms (and the future ones like application-timeouts.queuetime-monitor.interval-ms and application-timeouts.statestoretime-monitor.interval-ms ) [ApplicationSubmissionContext.java] /** * Get <code>ApplicationTimeouts</code> of the application. * * @param applicationTimeouts for the application. */ @Public @Unstable public abstract void setApplicationTimeouts( ApplicationTimeouts applicationTimeouts); Please change Get <code>ApplicationTimeouts</code> of the application. to Set <code>ApplicationTimeouts</code> for the application. [yarn_protos.proto] optional int64 life_timeout = 1 [ default = -1]; life_timeout -> lifetime [AbstractLivelinessMonitor.java] public synchronized void register(O ob, long timeout) { running.put(ob, timeout); For the second argument, do we mean timeout or submitTime? [RMAppLifetimeMonitor.java] // Don't trigger an KILL event if application is in completed states Change to - // Don't trigger a KILL event if application is in any of the completed states [MockRM.java] Add a new line before the below method - public RMApp submitApp( int masterMemory, Priority priority, [TestApplicationLifetimeMonitor.java] MockNM nm1 = rm.registerNode( "127.0.0.1:1234" , 16 * 1024); and new MockNM( "127.0.0.1:1234" , 8192, rm1.getResourceTrackerService()); Hardcoded ports cause unit test parallelization challenges. Is it possible to request a free port from the OS?
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          Thanks Gour Saha for review.

          For the second argument, do we mean timeout or submitTime?

          Its neither timeout nor submitTime. It is basically startTime for monitoring. May be it can be change to monitorStartTime?

          Hardcoded ports cause unit test parallelization challenges. Is it possible to request a free port from the OS?

          This is fine, MockNM framework does not bind to any port. So, random port issue does not occur.

          I will update patch fixing rest of the comments.

          Show
          rohithsharma Rohith Sharma K S added a comment - Thanks Gour Saha for review. For the second argument, do we mean timeout or submitTime? Its neither timeout nor submitTime. It is basically startTime for monitoring. May be it can be change to monitorStartTime? Hardcoded ports cause unit test parallelization challenges. Is it possible to request a free port from the OS? This is fine, MockNM framework does not bind to any port. So, random port issue does not occur. I will update patch fixing rest of the comments.
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          Updated patch fixing review comments.

          Show
          rohithsharma Rohith Sharma K S added a comment - Updated patch fixing review comments.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 19s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          0 mvndep 1m 3s Maven dependency ordering for branch
          +1 mvninstall 7m 20s trunk passed
          +1 compile 2m 25s trunk passed
          +1 checkstyle 0m 45s trunk passed
          +1 mvnsite 1m 38s trunk passed
          +1 mvneclipse 0m 42s trunk passed
          +1 findbugs 2m 59s trunk passed
          -1 javadoc 0m 22s hadoop-yarn-server-resourcemanager in trunk failed.
          0 mvndep 0m 10s Maven dependency ordering for patch
          +1 mvninstall 1m 21s the patch passed
          +1 compile 2m 26s the patch passed
          +1 cc 2m 26s the patch passed
          +1 javac 2m 26s the patch passed
          -1 checkstyle 0m 44s hadoop-yarn-project/hadoop-yarn: The patch generated 13 new + 502 unchanged - 3 fixed = 515 total (was 505)
          +1 mvnsite 1m 31s the patch passed
          +1 mvneclipse 0m 35s the patch passed
          -1 whitespace 0m 0s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 xml 0m 2s The patch has no ill-formed XML file.
          +1 findbugs 3m 12s the patch passed
          -1 javadoc 0m 15s hadoop-yarn-api in the patch failed.
          -1 javadoc 0m 18s hadoop-yarn-server-resourcemanager in the patch failed.
          +1 unit 0m 22s hadoop-yarn-api in the patch passed.
          +1 unit 2m 15s hadoop-yarn-common in the patch passed.
          +1 unit 35m 5s hadoop-yarn-server-resourcemanager in the patch passed.
          +1 asflicense 0m 18s The patch does not generate ASF License warnings.
          68m 13s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12829326/0005-YARN-4205.patch
          JIRA Issue YARN-4205
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc xml
          uname Linux 9b565b799e60 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / e52d6e7
          Default Java 1.8.0_101
          findbugs v3.0.0
          javadoc https://builds.apache.org/job/PreCommit-YARN-Build/13159/artifact/patchprocess/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13159/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          whitespace https://builds.apache.org/job/PreCommit-YARN-Build/13159/artifact/patchprocess/whitespace-eol.txt
          javadoc https://builds.apache.org/job/PreCommit-YARN-Build/13159/artifact/patchprocess/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api.txt
          javadoc https://builds.apache.org/job/PreCommit-YARN-Build/13159/artifact/patchprocess/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13159/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/13159/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 1m 3s Maven dependency ordering for branch +1 mvninstall 7m 20s trunk passed +1 compile 2m 25s trunk passed +1 checkstyle 0m 45s trunk passed +1 mvnsite 1m 38s trunk passed +1 mvneclipse 0m 42s trunk passed +1 findbugs 2m 59s trunk passed -1 javadoc 0m 22s hadoop-yarn-server-resourcemanager in trunk failed. 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 1m 21s the patch passed +1 compile 2m 26s the patch passed +1 cc 2m 26s the patch passed +1 javac 2m 26s the patch passed -1 checkstyle 0m 44s hadoop-yarn-project/hadoop-yarn: The patch generated 13 new + 502 unchanged - 3 fixed = 515 total (was 505) +1 mvnsite 1m 31s the patch passed +1 mvneclipse 0m 35s the patch passed -1 whitespace 0m 0s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 xml 0m 2s The patch has no ill-formed XML file. +1 findbugs 3m 12s the patch passed -1 javadoc 0m 15s hadoop-yarn-api in the patch failed. -1 javadoc 0m 18s hadoop-yarn-server-resourcemanager in the patch failed. +1 unit 0m 22s hadoop-yarn-api in the patch passed. +1 unit 2m 15s hadoop-yarn-common in the patch passed. +1 unit 35m 5s hadoop-yarn-server-resourcemanager in the patch passed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 68m 13s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12829326/0005-YARN-4205.patch JIRA Issue YARN-4205 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc xml uname Linux 9b565b799e60 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / e52d6e7 Default Java 1.8.0_101 findbugs v3.0.0 javadoc https://builds.apache.org/job/PreCommit-YARN-Build/13159/artifact/patchprocess/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13159/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/13159/artifact/patchprocess/whitespace-eol.txt javadoc https://builds.apache.org/job/PreCommit-YARN-Build/13159/artifact/patchprocess/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api.txt javadoc https://builds.apache.org/job/PreCommit-YARN-Build/13159/artifact/patchprocess/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13159/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/13159/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          Many checkstyle and javadoc errors are reported at project level.

          Show
          rohithsharma Rohith Sharma K S added a comment - Many checkstyle and javadoc errors are reported at project level.
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          It looks 3 java doc warnings are related, I will upload new patch. Before thatI will wait for Gour to review last patch.

          Show
          rohithsharma Rohith Sharma K S added a comment - It looks 3 java doc warnings are related, I will upload new patch. Before thatI will wait for Gour to review last patch.
          Hide
          gsaha Gour Saha added a comment -

          Thanks Rohith Sharma K S. A few minor cosmetic follow-up comments, and 2 additional fundamental questions.

          [ApplicationTimeouts.java]
             * Get life timeout of an application. The application will be killed
          

          Change life timeout to lifetime.

             * @param lifeTimeout of an application in seconds.
          

          Change lifeTimeout to lifetime.

            public abstract void setLifetime(long lifeTime);
          

          Change lifeTime to lifetime (lowercase t)

          [yarn-default.xml]
              The RMAppLifeTimeMonitor Service uses this value as monitor interval.
          

          Change to "The RMAppLifetimeMonitor Service uses this value as lifetime monitor interval." (note, lower-cased t in RMAppLifetimeMonitor and added lifetime after "value as")

          [TestApplicationLifetimeMonitor.java]
                Assert.assertTrue("Applicaiton killed before life timeout value",
          

          Change "life timeout" to "lifetime" (note, this change is needed in 2 lines)

            public void testApplicationLifeTimeMonitor() throws Exception {
          

          testApplicationLifeTimeMonitor -> testApplicationLifetimeMonitor (lowercase t)

            public void testApplicationLifeTimeOnRMRestart() throws Exception {
          

          testApplicationLifeTimeOnRMRestart -> testApplicationLifetimeOnRMRestart (lowercase t)

          [RMContextImpl.java]
                RMAppLifetimeMonitor rmAppLifeTimeMonitor) {
          

          rmAppLifeTimeMonitor -> rmAppLifetimeMonitor (lowercase t)

          [MockRM.java]
                long applicationLifeTime) throws Exception {
          

          applicationLifeTime -> applicationLifetime (lowercase t)

          There are 2 fundamental questions that come to my mind and I wanted to run across with you -

          1. Should AMRMClientAsync.onShutdownRequest callback be raised to give AM to do some last minute work/cleanup/graceful-shutdown-opportunity? I don't think we need to, but still wanted to call it out and know your thoughts on this.

          2. Seems like the lifetime is counted from the start of the application submission. Shouldn't it be counted from the time YARN allocates resource for the AM and launches it? What if YARN takes more time than the lifetime to allocate resource for the app? Seems like the KILL event will be raised immediately after the app reaches the RUNNING state in this case. Am I correct?

          Show
          gsaha Gour Saha added a comment - Thanks Rohith Sharma K S . A few minor cosmetic follow-up comments, and 2 additional fundamental questions. [ApplicationTimeouts.java] * Get life timeout of an application. The application will be killed Change life timeout to lifetime. * @param lifeTimeout of an application in seconds. Change lifeTimeout to lifetime. public abstract void setLifetime( long lifeTime); Change lifeTime to lifetime (lowercase t) [yarn-default.xml] The RMAppLifeTimeMonitor Service uses this value as monitor interval. Change to "The RMAppLifetimeMonitor Service uses this value as lifetime monitor interval." (note, lower-cased t in RMAppLifetimeMonitor and added lifetime after "value as") [TestApplicationLifetimeMonitor.java] Assert.assertTrue( "Applicaiton killed before life timeout value" , Change "life timeout" to "lifetime" (note, this change is needed in 2 lines) public void testApplicationLifeTimeMonitor() throws Exception { testApplicationLifeTimeMonitor -> testApplicationLifetimeMonitor (lowercase t) public void testApplicationLifeTimeOnRMRestart() throws Exception { testApplicationLifeTimeOnRMRestart -> testApplicationLifetimeOnRMRestart (lowercase t) [RMContextImpl.java] RMAppLifetimeMonitor rmAppLifeTimeMonitor) { rmAppLifeTimeMonitor -> rmAppLifetimeMonitor (lowercase t) [MockRM.java] long applicationLifeTime) throws Exception { applicationLifeTime -> applicationLifetime (lowercase t) There are 2 fundamental questions that come to my mind and I wanted to run across with you - 1. Should AMRMClientAsync.onShutdownRequest callback be raised to give AM to do some last minute work/cleanup/graceful-shutdown-opportunity? I don't think we need to, but still wanted to call it out and know your thoughts on this. 2. Seems like the lifetime is counted from the start of the application submission. Shouldn't it be counted from the time YARN allocates resource for the AM and launches it? What if YARN takes more time than the lifetime to allocate resource for the app? Seems like the KILL event will be raised immediately after the app reaches the RUNNING state in this case. Am I correct?
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          Shouldn't it be counted from the time YARN allocates resource for the AM and launches it? What if YARN takes more time than the lifetime to allocate resource for the app? Seems like the KILL event will be raised immediately after the app reaches the RUNNING state in this case. Am I correct?

          Basically this point got discussed in earlier comments also, this JIRA is to track lifetime of an application. Defining lifetime is nothing but overall execution_time of application from submission time. Application get killed at any point of time. Basically user is imposing timeout for execution time. Use case is every 5 minutes he submits a application and user need output with in 5 minutes. Use do not worry for time-consumed on statestore or allocation or any other which is not user facing also.
          Considering each stages of timeout, in future any other timeouts can be added to ApplicationTimeouts class. ApplicationTimeouts java doc also gives definition of lifetime.

          Should AMRMClientAsync.onShutdownRequest callback be raised to give AM to do some last minute work/cleanup/graceful-shutdown-opportunity?

          Good point and need to discuss more in general. IIUC, killing an application does not allow AM containers to do some clean up. I think the JIRA YARN-2261 is intended to do such post clean up.

          Show
          rohithsharma Rohith Sharma K S added a comment - Shouldn't it be counted from the time YARN allocates resource for the AM and launches it? What if YARN takes more time than the lifetime to allocate resource for the app? Seems like the KILL event will be raised immediately after the app reaches the RUNNING state in this case. Am I correct? Basically this point got discussed in earlier comments also, this JIRA is to track lifetime of an application. Defining lifetime is nothing but overall execution_time of application from submission time. Application get killed at any point of time. Basically user is imposing timeout for execution time. Use case is every 5 minutes he submits a application and user need output with in 5 minutes. Use do not worry for time-consumed on statestore or allocation or any other which is not user facing also. Considering each stages of timeout, in future any other timeouts can be added to ApplicationTimeouts class. ApplicationTimeouts java doc also gives definition of lifetime. Should AMRMClientAsync.onShutdownRequest callback be raised to give AM to do some last minute work/cleanup/graceful-shutdown-opportunity? Good point and need to discuss more in general. IIUC, killing an application does not allow AM containers to do some clean up. I think the JIRA YARN-2261 is intended to do such post clean up.
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          Updated patch fixing review comments

          Show
          rohithsharma Rohith Sharma K S added a comment - Updated patch fixing review comments
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 14s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          0 mvndep 0m 58s Maven dependency ordering for branch
          +1 mvninstall 6m 51s trunk passed
          +1 compile 2m 17s trunk passed
          +1 checkstyle 0m 45s trunk passed
          +1 mvnsite 1m 34s trunk passed
          +1 mvneclipse 0m 41s trunk passed
          +1 findbugs 2m 51s trunk passed
          +1 javadoc 1m 8s trunk passed
          0 mvndep 0m 9s Maven dependency ordering for patch
          +1 mvninstall 1m 26s the patch passed
          +1 compile 2m 22s the patch passed
          +1 cc 2m 22s the patch passed
          +1 javac 2m 22s the patch passed
          -1 checkstyle 0m 43s hadoop-yarn-project/hadoop-yarn: The patch generated 13 new + 502 unchanged - 3 fixed = 515 total (was 505)
          +1 mvnsite 1m 27s the patch passed
          +1 mvneclipse 0m 36s the patch passed
          -1 whitespace 0m 0s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 3m 12s the patch passed
          +1 javadoc 0m 59s the patch passed
          +1 unit 0m 23s hadoop-yarn-api in the patch passed.
          +1 unit 2m 16s hadoop-yarn-common in the patch passed.
          +1 unit 35m 6s hadoop-yarn-server-resourcemanager in the patch passed.
          +1 asflicense 0m 16s The patch does not generate ASF License warnings.
          67m 10s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12829492/0006-YARN-4205.patch
          JIRA Issue YARN-4205
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc xml
          uname Linux 2991a91b8ebf 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 964e546
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13173/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          whitespace https://builds.apache.org/job/PreCommit-YARN-Build/13173/artifact/patchprocess/whitespace-eol.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13173/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/13173/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 0m 58s Maven dependency ordering for branch +1 mvninstall 6m 51s trunk passed +1 compile 2m 17s trunk passed +1 checkstyle 0m 45s trunk passed +1 mvnsite 1m 34s trunk passed +1 mvneclipse 0m 41s trunk passed +1 findbugs 2m 51s trunk passed +1 javadoc 1m 8s trunk passed 0 mvndep 0m 9s Maven dependency ordering for patch +1 mvninstall 1m 26s the patch passed +1 compile 2m 22s the patch passed +1 cc 2m 22s the patch passed +1 javac 2m 22s the patch passed -1 checkstyle 0m 43s hadoop-yarn-project/hadoop-yarn: The patch generated 13 new + 502 unchanged - 3 fixed = 515 total (was 505) +1 mvnsite 1m 27s the patch passed +1 mvneclipse 0m 36s the patch passed -1 whitespace 0m 0s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 3m 12s the patch passed +1 javadoc 0m 59s the patch passed +1 unit 0m 23s hadoop-yarn-api in the patch passed. +1 unit 2m 16s hadoop-yarn-common in the patch passed. +1 unit 35m 6s hadoop-yarn-server-resourcemanager in the patch passed. +1 asflicense 0m 16s The patch does not generate ASF License warnings. 67m 10s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12829492/0006-YARN-4205.patch JIRA Issue YARN-4205 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc xml uname Linux 2991a91b8ebf 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 964e546 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13173/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/13173/artifact/patchprocess/whitespace-eol.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13173/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/13173/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          gsaha Gour Saha added a comment -

          Basically this point got discussed in earlier comments also, this JIRA is to track lifetime of an application. Defining lifetime is nothing but overall execution_time of application from submission time. Application get killed at any point of time. Basically user is imposing timeout for execution time. Use case is every 5 minutes he submits a application and user need output with in 5 minutes. Use do not worry for time-consumed on statestore or allocation or any other which is not user facing also.
          Considering each stages of timeout, in future any other timeouts can be added to ApplicationTimeouts class. ApplicationTimeouts java doc also gives definition of lifetime.

          So if the app is sitting in ACCEPTED state, as per current implementation the timeout monitor has started, right? If yes, then it does not fully satisfy the lifetime feature that YARN-4692 is looking for. In that case, we need to introduce monitor-start-time as a configurable parameter in ApplicationTimeouts. App submitters should be able to specify it along with the timeout value. The start time could be a set of relevant app state transition watermarks, like AppTimeoutsMonitorStartTime.SUBMISSION_TIME, AppTimeoutsMonitorStartTime.LAUNCH_TIME, etc. Thoughts?

          Show
          gsaha Gour Saha added a comment - Basically this point got discussed in earlier comments also, this JIRA is to track lifetime of an application. Defining lifetime is nothing but overall execution_time of application from submission time. Application get killed at any point of time. Basically user is imposing timeout for execution time. Use case is every 5 minutes he submits a application and user need output with in 5 minutes. Use do not worry for time-consumed on statestore or allocation or any other which is not user facing also. Considering each stages of timeout, in future any other timeouts can be added to ApplicationTimeouts class. ApplicationTimeouts java doc also gives definition of lifetime. So if the app is sitting in ACCEPTED state, as per current implementation the timeout monitor has started, right? If yes, then it does not fully satisfy the lifetime feature that YARN-4692 is looking for. In that case, we need to introduce monitor-start-time as a configurable parameter in ApplicationTimeouts. App submitters should be able to specify it along with the timeout value. The start time could be a set of relevant app state transition watermarks, like AppTimeoutsMonitorStartTime.SUBMISSION_TIME, AppTimeoutsMonitorStartTime.LAUNCH_TIME, etc. Thoughts?
          Hide
          sunilg Sunil G added a comment -

          Hi Gour Saha

          Generally this comment makes sense for me.
          If we are passing monitor-start-time as config, are we looking in case where client can specify the time from which monitoring has to start? Or could we pass some types like SUBMISSION/LAUNCH and RM will mark start time once app crosses that state. Latter may help to have accuracy for start-time.

          Show
          sunilg Sunil G added a comment - Hi Gour Saha Generally this comment makes sense for me. If we are passing monitor-start-time as config, are we looking in case where client can specify the time from which monitoring has to start? Or could we pass some types like SUBMISSION/LAUNCH and RM will mark start time once app crosses that state. Latter may help to have accuracy for start-time.
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          Thanks Gour Saha for the suggestion. Some clarifications on this JIRA. Few points might be re-iterated from my earlier comments also.

          1. This JIRA mainly aims at providing overall execution timeout of application. And monitor start from app-submission-time to completing states.
          2. And the patch tries to maintain compatibility wrt client APIs. Any support for new timeouts(allocation 0r queue_timeout etc) in future, implementer need to add new set of API's in ApplicationTimeout object and corresponding implementation in server side. To the user, there will no change in the ApplicationSubmissionContext APIs.
          3. The timeout can be started from any of the states like SUBMISSION/LAUNCH/RUNNING for an application. But some of the constraints from YARN are
            1. Say, if there is any multiple attempt failure, then there will be multiple launch time. Does timeout for each attempt? monitor start time would be keep changing where each time attempt launch need to register and unregister when failure happens. Similar issue is for RUNNIG state also.
            2. Some timeouts can not be supported such as state_store because these operations are/can-be too frequent. In this cases, any issue with statestore, RM itself will go down. Concrete use case for supporting this timeout is not yet known.

          And would you elaborate use case for having timeout from launch-time/running states?

          cc :/Vinod Kumar Vavilapalli Jian He

          Show
          rohithsharma Rohith Sharma K S added a comment - Thanks Gour Saha for the suggestion. Some clarifications on this JIRA. Few points might be re-iterated from my earlier comments also. This JIRA mainly aims at providing overall execution timeout of application. And monitor start from app-submission-time to completing states. And the patch tries to maintain compatibility wrt client APIs. Any support for new timeouts(allocation 0r queue_timeout etc) in future, implementer need to add new set of API's in ApplicationTimeout object and corresponding implementation in server side. To the user, there will no change in the ApplicationSubmissionContext APIs. The timeout can be started from any of the states like SUBMISSION/LAUNCH/RUNNING for an application. But some of the constraints from YARN are Say, if there is any multiple attempt failure, then there will be multiple launch time. Does timeout for each attempt? monitor start time would be keep changing where each time attempt launch need to register and unregister when failure happens. Similar issue is for RUNNIG state also. Some timeouts can not be supported such as state_store because these operations are/can-be too frequent. In this cases, any issue with statestore, RM itself will go down. Concrete use case for supporting this timeout is not yet known. And would you elaborate use case for having timeout from launch-time/running states? cc :/ Vinod Kumar Vavilapalli Jian He
          Hide
          gsaha Gour Saha added a comment -

          YARN-4692 also needs total execution timeout of application. It does not need queue or state-store timeouts. The only additional thing it needs, is the monitor start time to provide options like LAUNCH_FIRST and LAUNCH_EVERYTIME on top of SUBMISSION.

          YARN-4692 is for long running or semi long-running services. Long running services are meant to run forever. Semi long-running services are meant to run for several hours or few days or weeks. For the semi long-running usecases, we have applications like a CI (continuous-integration) app or a System Test app, which typically needs several hours to finish its job. CI app or System Test app owners are not going to be happy that their app did not run at all, even though it was given a timeout of 10 hours because YARN got a chance to allocate resource to it at the 9:59 hour mark since submission. That is why we need LAUNCH_FIRST.

          LAUNCH_EVERYTIME will cover scenarios like say a load test app needs to run end-to-end for at least 2 days straight to certify a product. If such an app is pre-empted at say the 47 hour mark, it needs a fresh 48 hour lifetime the next time it is re-launched.

          Show
          gsaha Gour Saha added a comment - YARN-4692 also needs total execution timeout of application. It does not need queue or state-store timeouts. The only additional thing it needs, is the monitor start time to provide options like LAUNCH_FIRST and LAUNCH_EVERYTIME on top of SUBMISSION. YARN-4692 is for long running or semi long-running services. Long running services are meant to run forever. Semi long-running services are meant to run for several hours or few days or weeks. For the semi long-running usecases, we have applications like a CI (continuous-integration) app or a System Test app, which typically needs several hours to finish its job. CI app or System Test app owners are not going to be happy that their app did not run at all, even though it was given a timeout of 10 hours because YARN got a chance to allocate resource to it at the 9:59 hour mark since submission. That is why we need LAUNCH_FIRST. LAUNCH_EVERYTIME will cover scenarios like say a load test app needs to run end-to-end for at least 2 days straight to certify a product. If such an app is pre-empted at say the 47 hour mark, it needs a fresh 48 hour lifetime the next time it is re-launched.
          Hide
          jianhe Jian He added a comment -

          Gour Saha,
          Current ApplicationTimeout has getLifetime which indicates the total lifeTime since application is submitted.
          IIUC, you need a time duration since the app is actually launched, am I right ?
          If we add a new API in ApplicationTimeout (like getRunningDuration) which means how long since the application is launched and the monitor would start monitoring since the app is actually launched instead of submission, would that satisfy your requirement ?

          Show
          jianhe Jian He added a comment - Gour Saha , Current ApplicationTimeout has getLifetime which indicates the total lifeTime since application is submitted. IIUC, you need a time duration since the app is actually launched, am I right ? If we add a new API in ApplicationTimeout (like getRunningDuration) which means how long since the application is launched and the monitor would start monitoring since the app is actually launched instead of submission, would that satisfy your requirement ?
          Hide
          jianhe Jian He added a comment -

          Or, we could also change the ApplicationTimeouts class to a map data structure where the key could be LIFE_DURATION, RUNTIME_DURATION, and the value is the time duration.

          Show
          jianhe Jian He added a comment - Or, we could also change the ApplicationTimeouts class to a map data structure where the key could be LIFE_DURATION, RUNTIME_DURATION, and the value is the time duration.
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          We ( Jian He Gour Saha Vinod Kumar Vavilapalli and me ) had offline discussion. Summary is as follows

          1. This JIRA mainly focus on providing framework to user to specify ApplicationTimeouts.
          2. Interestingly, new point that come across is that user should also provide multiple timeouts for application. Say, user wants his application should not run more than 10 hours AND do not want to queued up in scheduler more than 5 hours.
          3. Gour Saha use case is very much valid and it is require more discussion on handling from YARN. So, better to handle in separate JIRA.

          If I have missed any points, please feel free to add more.

          Show
          rohithsharma Rohith Sharma K S added a comment - We ( Jian He Gour Saha Vinod Kumar Vavilapalli and me ) had offline discussion. Summary is as follows This JIRA mainly focus on providing framework to user to specify ApplicationTimeouts. Interestingly, new point that come across is that user should also provide multiple timeouts for application. Say, user wants his application should not run more than 10 hours AND do not want to queued up in scheduler more than 5 hours. Gour Saha use case is very much valid and it is require more discussion on handling from YARN. So, better to handle in separate JIRA. If I have missed any points, please feel free to add more.
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          After I revisited the patch with discussion points, below are 2 points to be modified as a framework in current patch.

          1. The application-monitor-thread always register with ApplicationId. As a framework to support multiple timeouts for same applications, it is require to modify key as String rather than ApplicationId.
          2. ApplicationTimeouts has provided APIs to specify timeout. Instead API's, this class can be modified to take multiple timeouts as Map<ApplicationTimeoutType, long>. This allows user to provide multiple timeouts in flexible manner rather than setting each specific API like setLifetime, setQueueTime, setRuntime.
          Show
          rohithsharma Rohith Sharma K S added a comment - After I revisited the patch with discussion points, below are 2 points to be modified as a framework in current patch. The application-monitor-thread always register with ApplicationId. As a framework to support multiple timeouts for same applications, it is require to modify key as String rather than ApplicationId. ApplicationTimeouts has provided APIs to specify timeout. Instead API's, this class can be modified to take multiple timeouts as Map<ApplicationTimeoutType, long>. This allows user to provide multiple timeouts in flexible manner rather than setting each specific API like setLifetime, setQueueTime, setRuntime.
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          Updated the patch with following major changes from previous one with more general way.

          1. Moved RMAppLifetimeMonitor to separate package i.e org.apache.hadoop.yarn.server.resourcemanager.rmapp.monitor
          2. Created RMAppToMonitor custom class to hold monitoring key. This consists of ApplicationId+ApplicationTimeoutType. This is essential because same application can have multiple timeouts.
          3. Removed ApplicationTimeouts class.
          4. In ApplicationSubmissionContext#setApplicationTimeouts is changed to Map<ApplicationTimeoutType, Long>. In future, if we want to support more timeouts like in Gour use case, we just need to add enum and provide corresponding implementation in RMAppImpl.
          Show
          rohithsharma Rohith Sharma K S added a comment - Updated the patch with following major changes from previous one with more general way. Moved RMAppLifetimeMonitor to separate package i.e org.apache.hadoop.yarn.server.resourcemanager.rmapp.monitor Created RMAppToMonitor custom class to hold monitoring key. This consists of ApplicationId+ApplicationTimeoutType. This is essential because same application can have multiple timeouts. Removed ApplicationTimeouts class. In ApplicationSubmissionContext#setApplicationTimeouts is changed to Map<ApplicationTimeoutType, Long>. In future, if we want to support more timeouts like in Gour use case, we just need to add enum and provide corresponding implementation in RMAppImpl.
          Hide
          jianhe Jian He added a comment -

          latest patch looks good to me

          Show
          jianhe Jian He added a comment - latest patch looks good to me
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          I forgot to include test file. Updating new patch with Test file included.

          Show
          rohithsharma Rohith Sharma K S added a comment - I forgot to include test file. Updating new patch with Test file included.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 20s Maven dependency ordering for branch
          +1 mvninstall 9m 35s trunk passed
          +1 compile 2m 57s trunk passed
          +1 checkstyle 0m 48s trunk passed
          +1 mvnsite 1m 52s trunk passed
          +1 mvneclipse 0m 46s trunk passed
          +1 findbugs 3m 4s trunk passed
          +1 javadoc 1m 6s trunk passed
          0 mvndep 0m 9s Maven dependency ordering for patch
          +1 mvninstall 1m 19s the patch passed
          +1 compile 2m 17s the patch passed
          +1 cc 2m 17s the patch passed
          +1 javac 2m 17s the patch passed
          -1 checkstyle 0m 42s hadoop-yarn-project/hadoop-yarn: The patch generated 18 new + 540 unchanged - 3 fixed = 558 total (was 543)
          +1 mvnsite 1m 30s the patch passed
          +1 mvneclipse 0m 36s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 3m 18s the patch passed
          +1 javadoc 1m 8s the patch passed
          +1 unit 0m 29s hadoop-yarn-api in the patch passed.
          -1 unit 2m 36s hadoop-yarn-common in the patch failed.
          -1 unit 40m 37s hadoop-yarn-server-resourcemanager in the patch failed.
          +1 asflicense 0m 20s The patch does not generate ASF License warnings.
          76m 44s



          Reason Tests
          Failed junit tests hadoop.yarn.api.TestPBImplRecords
            hadoop.yarn.server.resourcemanager.TestRMRestart
            hadoop.yarn.server.resourcemanager.TestRMAdminService
            hadoop.yarn.server.resourcemanager.TestNodeBlacklistingOnAMFailures



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12830678/0007-YARN-4205.patch
          JIRA Issue YARN-4205
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc xml
          uname Linux f39ec40bf432 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 9b0fd01
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13235/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/13235/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/13235/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/13235/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt https://builds.apache.org/job/PreCommit-YARN-Build/13235/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13235/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/13235/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 20s Maven dependency ordering for branch +1 mvninstall 9m 35s trunk passed +1 compile 2m 57s trunk passed +1 checkstyle 0m 48s trunk passed +1 mvnsite 1m 52s trunk passed +1 mvneclipse 0m 46s trunk passed +1 findbugs 3m 4s trunk passed +1 javadoc 1m 6s trunk passed 0 mvndep 0m 9s Maven dependency ordering for patch +1 mvninstall 1m 19s the patch passed +1 compile 2m 17s the patch passed +1 cc 2m 17s the patch passed +1 javac 2m 17s the patch passed -1 checkstyle 0m 42s hadoop-yarn-project/hadoop-yarn: The patch generated 18 new + 540 unchanged - 3 fixed = 558 total (was 543) +1 mvnsite 1m 30s the patch passed +1 mvneclipse 0m 36s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 3m 18s the patch passed +1 javadoc 1m 8s the patch passed +1 unit 0m 29s hadoop-yarn-api in the patch passed. -1 unit 2m 36s hadoop-yarn-common in the patch failed. -1 unit 40m 37s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 76m 44s Reason Tests Failed junit tests hadoop.yarn.api.TestPBImplRecords   hadoop.yarn.server.resourcemanager.TestRMRestart   hadoop.yarn.server.resourcemanager.TestRMAdminService   hadoop.yarn.server.resourcemanager.TestNodeBlacklistingOnAMFailures Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12830678/0007-YARN-4205.patch JIRA Issue YARN-4205 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc xml uname Linux f39ec40bf432 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 9b0fd01 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13235/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/13235/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/13235/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/13235/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt https://builds.apache.org/job/PreCommit-YARN-Build/13235/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13235/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/13235/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 0m 13s Maven dependency ordering for branch
          +1 mvninstall 7m 58s trunk passed
          +1 compile 2m 31s trunk passed
          +1 checkstyle 0m 46s trunk passed
          +1 mvnsite 1m 38s trunk passed
          +1 mvneclipse 0m 44s trunk passed
          +1 findbugs 3m 12s trunk passed
          +1 javadoc 1m 7s trunk passed
          0 mvndep 0m 10s Maven dependency ordering for patch
          +1 mvninstall 1m 43s the patch passed
          +1 compile 3m 7s the patch passed
          +1 cc 3m 7s the patch passed
          +1 javac 3m 7s the patch passed
          -1 checkstyle 0m 51s hadoop-yarn-project/hadoop-yarn: The patch generated 20 new + 539 unchanged - 3 fixed = 559 total (was 542)
          +1 mvnsite 1m 51s the patch passed
          +1 mvneclipse 0m 43s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 2s The patch has no ill-formed XML file.
          +1 findbugs 4m 10s the patch passed
          +1 javadoc 1m 13s the patch passed
          +1 unit 0m 29s hadoop-yarn-api in the patch passed.
          -1 unit 2m 35s hadoop-yarn-common in the patch failed.
          -1 unit 37m 48s hadoop-yarn-server-resourcemanager in the patch failed.
          +1 asflicense 0m 25s The patch does not generate ASF License warnings.
          74m 36s



          Reason Tests
          Failed junit tests hadoop.yarn.api.TestPBImplRecords
            hadoop.yarn.server.resourcemanager.scheduler.capacity.TestIncreaseAllocationExpirer
            hadoop.yarn.server.resourcemanager.TestRMAdminService



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12830685/0007-YARN-4205.1.patch
          JIRA Issue YARN-4205
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc xml
          uname Linux 9550dff67904 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 9b0fd01
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13236/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/13236/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/13236/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/13236/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt https://builds.apache.org/job/PreCommit-YARN-Build/13236/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13236/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/13236/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 0m 13s Maven dependency ordering for branch +1 mvninstall 7m 58s trunk passed +1 compile 2m 31s trunk passed +1 checkstyle 0m 46s trunk passed +1 mvnsite 1m 38s trunk passed +1 mvneclipse 0m 44s trunk passed +1 findbugs 3m 12s trunk passed +1 javadoc 1m 7s trunk passed 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 1m 43s the patch passed +1 compile 3m 7s the patch passed +1 cc 3m 7s the patch passed +1 javac 3m 7s the patch passed -1 checkstyle 0m 51s hadoop-yarn-project/hadoop-yarn: The patch generated 20 new + 539 unchanged - 3 fixed = 559 total (was 542) +1 mvnsite 1m 51s the patch passed +1 mvneclipse 0m 43s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. +1 findbugs 4m 10s the patch passed +1 javadoc 1m 13s the patch passed +1 unit 0m 29s hadoop-yarn-api in the patch passed. -1 unit 2m 35s hadoop-yarn-common in the patch failed. -1 unit 37m 48s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 25s The patch does not generate ASF License warnings. 74m 36s Reason Tests Failed junit tests hadoop.yarn.api.TestPBImplRecords   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestIncreaseAllocationExpirer   hadoop.yarn.server.resourcemanager.TestRMAdminService Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12830685/0007-YARN-4205.1.patch JIRA Issue YARN-4205 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc xml uname Linux 9550dff67904 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 9b0fd01 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13236/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/13236/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/13236/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/13236/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt https://builds.apache.org/job/PreCommit-YARN-Build/13236/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13236/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/13236/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          Fixing TestPBImplRecords failure and some of the checkstyles

          Show
          rohithsharma Rohith Sharma K S added a comment - Fixing TestPBImplRecords failure and some of the checkstyles
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          0 mvndep 0m 10s Maven dependency ordering for branch
          +1 mvninstall 7m 22s trunk passed
          +1 compile 2m 18s trunk passed
          +1 checkstyle 0m 44s trunk passed
          +1 mvnsite 1m 33s trunk passed
          +1 mvneclipse 0m 40s trunk passed
          +1 findbugs 2m 49s trunk passed
          +1 javadoc 1m 3s trunk passed
          0 mvndep 0m 10s Maven dependency ordering for patch
          +1 mvninstall 1m 15s the patch passed
          +1 compile 2m 14s the patch passed
          +1 cc 2m 14s the patch passed
          +1 javac 2m 14s the patch passed
          -1 checkstyle 0m 43s hadoop-yarn-project/hadoop-yarn: The patch generated 3 new + 569 unchanged - 3 fixed = 572 total (was 572)
          +1 mvnsite 1m 27s the patch passed
          +1 mvneclipse 0m 35s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 3m 7s the patch passed
          +1 javadoc 0m 57s the patch passed
          +1 unit 0m 22s hadoop-yarn-api in the patch passed.
          +1 unit 2m 15s hadoop-yarn-common in the patch passed.
          +1 unit 34m 52s hadoop-yarn-server-resourcemanager in the patch passed.
          +1 asflicense 0m 18s The patch does not generate ASF License warnings.
          66m 6s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12830718/0007-YARN-4205.2.patch
          JIRA Issue YARN-4205
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc xml
          uname Linux 403272cbfc8b 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / e19b37e
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13237/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13237/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/13237/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 0m 10s Maven dependency ordering for branch +1 mvninstall 7m 22s trunk passed +1 compile 2m 18s trunk passed +1 checkstyle 0m 44s trunk passed +1 mvnsite 1m 33s trunk passed +1 mvneclipse 0m 40s trunk passed +1 findbugs 2m 49s trunk passed +1 javadoc 1m 3s trunk passed 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 1m 15s the patch passed +1 compile 2m 14s the patch passed +1 cc 2m 14s the patch passed +1 javac 2m 14s the patch passed -1 checkstyle 0m 43s hadoop-yarn-project/hadoop-yarn: The patch generated 3 new + 569 unchanged - 3 fixed = 572 total (was 572) +1 mvnsite 1m 27s the patch passed +1 mvneclipse 0m 35s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 3m 7s the patch passed +1 javadoc 0m 57s the patch passed +1 unit 0m 22s hadoop-yarn-api in the patch passed. +1 unit 2m 15s hadoop-yarn-common in the patch passed. +1 unit 34m 52s hadoop-yarn-server-resourcemanager in the patch passed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 66m 6s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12830718/0007-YARN-4205.2.patch JIRA Issue YARN-4205 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc xml uname Linux 403272cbfc8b 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / e19b37e Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13237/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13237/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/13237/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          jianhe Jian He added a comment -

          Committed to trunk, branch-2, thanks Rohith !
          Thanks Gour, Vinod, nijel for the review !

          Show
          jianhe Jian He added a comment - Committed to trunk, branch-2, thanks Rohith ! Thanks Gour, Vinod, nijel for the review !
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10512 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10512/)
          YARN-4205. Add a service for monitoring application life time out. (jianhe: rev 2ae5a3a5bf5ea355370469a53eeccff0b5220081)

          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/AbstractLivelinessMonitor.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMActiveServiceContext.java
          • (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/monitor/package-info.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java
          • (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/TestApplicationLifetimeMonitor.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationSubmissionContext.java
          • (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationTimeoutType.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContextImpl.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/TestPBImplRecords.java
          • (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/monitor/RMAppToMonitor.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ProtoUtils.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContext.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMServerUtils.java
          • (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/monitor/RMAppLifetimeMonitor.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ApplicationSubmissionContextPBImpl.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10512 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10512/ ) YARN-4205 . Add a service for monitoring application life time out. (jianhe: rev 2ae5a3a5bf5ea355370469a53eeccff0b5220081) (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/AbstractLivelinessMonitor.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMActiveServiceContext.java (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/monitor/package-info.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/TestApplicationLifetimeMonitor.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationSubmissionContext.java (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationTimeoutType.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContextImpl.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/TestPBImplRecords.java (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/monitor/RMAppToMonitor.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ProtoUtils.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContext.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMServerUtils.java (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/monitor/RMAppLifetimeMonitor.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ApplicationSubmissionContextPBImpl.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          thank Jian He for review and committing patch.. thanks Sunil, Vinod and Gour for reviewing patch.. special thanks to nijel for providing POC patch!!

          Show
          rohithsharma Rohith Sharma K S added a comment - thank Jian He for review and committing patch.. thanks Sunil, Vinod and Gour for reviewing patch.. special thanks to nijel for providing POC patch!!
          Hide
          cnauroth Chris Nauroth added a comment -

          I think this patch broke compilation on branch-2.

            private Map<RMAppToMonitor, Long> monitoredApps =
                new HashMap<RMAppToMonitor, Long>();
          
              monitoredApps.putIfAbsent(appToMonitor, timeout);
          

          Map#putIfAbsent was added in JDK 1.8, but we want to be able to compile branch-2 for JDK 1.7.

          Can someone please take a look?

          Show
          cnauroth Chris Nauroth added a comment - I think this patch broke compilation on branch-2. private Map<RMAppToMonitor, Long > monitoredApps = new HashMap<RMAppToMonitor, Long >(); monitoredApps.putIfAbsent(appToMonitor, timeout); Map#putIfAbsent was added in JDK 1.8, but we want to be able to compile branch-2 for JDK 1.7. Can someone please take a look?
          Hide
          gtCarrera9 Li Lu added a comment -

          Upload an addendum patch to expand the (non-synchronized) putIfAbsent method to unblock branch-2 builds.

          Show
          gtCarrera9 Li Lu added a comment - Upload an addendum patch to expand the (non-synchronized) putIfAbsent method to unblock branch-2 builds.
          Hide
          cnauroth Chris Nauroth added a comment -

          +1 for the addendum patch. Li Lu, thank you.

          Show
          cnauroth Chris Nauroth added a comment - +1 for the addendum patch. Li Lu , thank you.
          Hide
          gtCarrera9 Li Lu added a comment -

          Chris Nauroth shall we put this patch to branch-2 only, or to trunk as well?

          Show
          gtCarrera9 Li Lu added a comment - Chris Nauroth shall we put this patch to branch-2 only, or to trunk as well?
          Hide
          cnauroth Chris Nauroth added a comment -

          Li Lu, we are all clear to use JDK 8 features in trunk, so I think committing only to branch-2 is fine.

          Show
          cnauroth Chris Nauroth added a comment - Li Lu , we are all clear to use JDK 8 features in trunk, so I think committing only to branch-2 is fine.
          Hide
          gtCarrera9 Li Lu added a comment -

          OK, I'll commit it shortly then.

          Show
          gtCarrera9 Li Lu added a comment - OK, I'll commit it shortly then.
          Hide
          gtCarrera9 Li Lu added a comment -

          I committed the addendum patch to branch-2. Thanks Chris Nauroth for the review. Jian He Rohith Sharma K S if there's any concerns with this fix, please feel free to post a new one. Thanks!

          Show
          gtCarrera9 Li Lu added a comment - I committed the addendum patch to branch-2. Thanks Chris Nauroth for the review. Jian He Rohith Sharma K S if there's any concerns with this fix, please feel free to post a new one. Thanks!
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          +1 for addendum patch.. Thanks Li Lu for quick response.

          Show
          rohithsharma Rohith Sharma K S added a comment - +1 for addendum patch.. Thanks Li Lu for quick response.
          Hide
          jianhe Jian He added a comment -

          Chris Nauroth, Li Lu, thanks for helping on this ! my bad.

          Show
          jianhe Jian He added a comment - Chris Nauroth , Li Lu , thanks for helping on this ! my bad.
          Hide
          imstefanlee stefanlee added a comment -

          I have a doubt that if it is suitable fo long time job ,e.g spark streaming job?

          Show
          imstefanlee stefanlee added a comment - I have a doubt that if it is suitable fo long time job ,e.g spark streaming job?
          Hide
          rohithsharma Rohith Sharma K S added a comment -

          You can set timeout for any job.

          Show
          rohithsharma Rohith Sharma K S added a comment - You can set timeout for any job.

            People

            • Assignee:
              rohithsharma Rohith Sharma K S
              Reporter:
              nijel nijel
            • Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development