Details

    • Type: Sub-task
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.2.0
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      This is to make sure user is notified job finished after job is really done. This does increase client latency but can reduce some races during unregister like YARN-540

      1. MAPREDUCE-5505.1.patch
        7 kB
        Zhijie Shen
      2. MAPREDUCE-5505.1.patch
        7 kB
        Zhijie Shen
      3. MAPREDUCE-5505.3.patch
        14 kB
        Zhijie Shen
      4. MAPREDUCE-5505.4.patch
        27 kB
        Zhijie Shen
      5. MAPREDUCE-5505.5.patch
        32 kB
        Zhijie Shen
      6. MAPREDUCE-5505.6.patch
        17 kB
        Zhijie Shen
      7. MAPREDUCE-5505.7.patch
        19 kB
        Zhijie Shen

        Issue Links

          Activity

          Hide
          zjshen Zhijie Shen added a comment -

          Take it over. Thanks!

          Show
          zjshen Zhijie Shen added a comment - Take it over. Thanks!
          Hide
          zjshen Zhijie Shen added a comment -

          IMHO, when getReport() from a job, we can check whether the AM has been unregistered or not (probably by checking whether FinalApplicationStatus is undefined or not). If the AM is unregistered, we return the final state. Otherwise, we return RUNNING instead. Thoughts?

          Show
          zjshen Zhijie Shen added a comment - IMHO, when getReport() from a job, we can check whether the AM has been unregistered or not (probably by checking whether FinalApplicationStatus is undefined or not). If the AM is unregistered, we return the final state. Otherwise, we return RUNNING instead. Thoughts?
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Ideally we should have the state transition itself to go to SUCCEEDED after unregister. Let's see if that is very difficult to do. Clearly we'll need more signalling.

          OTOH, I think we should hard-code everywhere to say AM that runs only one job. Part of the problem is that AM at the top level thinks it can run multiple jobs, though in practice we only run on job. This could simplify some of the things in the MR AM.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Ideally we should have the state transition itself to go to SUCCEEDED after unregister. Let's see if that is very difficult to do. Clearly we'll need more signalling. OTOH, I think we should hard-code everywhere to say AM that runs only one job. Part of the problem is that AM at the top level thinks it can run multiple jobs, though in practice we only run on job. This could simplify some of the things in the MR AM.
          Hide
          bikassaha Bikas Saha added a comment -

          Can we make either
          1) JobImpl go to succeeded/terminal state only after app master serviceStop() has completed?
          2) Change the MRClientProtocol to return job.state as the previous job state when job.state == succeeded. Then when MRClientProtocol.stop() is called it starts returning job.state == succeeded for 5 secs. Then it stops and the app master stops.

          Show
          bikassaha Bikas Saha added a comment - Can we make either 1) JobImpl go to succeeded/terminal state only after app master serviceStop() has completed? 2) Change the MRClientProtocol to return job.state as the previous job state when job.state == succeeded. Then when MRClientProtocol.stop() is called it starts returning job.state == succeeded for 5 secs. Then it stops and the app master stops.
          Hide
          zjshen Zhijie Shen added a comment -

          Ideally we should have the state transition itself to go to SUCCEEDED after unregister.

          unregister needs to happens when a job is after COMMITTING. If we want unregister to be before SUCCEEDED, we have to insert another state in middle, where unregister can happen. Then, we still need to to map the state in middle to a external state. If we map it to SUCCEEDED, the problem is still there; if we map it to RUNNING, it is equivalent to returning RUNNING when AM is unregistered (in the prior comment).

          In addition, the problem is not limited to SUCCEEDED only, is it? FAILED, KILLED and ERROR shouldn't be returned before unregister. In particular, FAILED and KILLED map to two internal states respectively. In this case, I'm afraid it's even more difficult to find state in between to execute unregister.

          1) JobImpl go to succeeded/terminal state only after app master serviceStop() has completed?

          In addition to the reason above, if serviceStop() is completed, the client is already unable to get the latest state from AM.

          2) Change the MRClientProtocol to return job.state as the previous job state when job.state == succeeded. Then when MRClientProtocol.stop() is called it starts returning job.state == succeeded for 5 secs. Then it stops and the app master stops.

          +1 for this method. It sounds work with limited touch. Again, we should consider FAILED, KILLED and ERROR as well, right?

          Show
          zjshen Zhijie Shen added a comment - Ideally we should have the state transition itself to go to SUCCEEDED after unregister. unregister needs to happens when a job is after COMMITTING. If we want unregister to be before SUCCEEDED, we have to insert another state in middle, where unregister can happen. Then, we still need to to map the state in middle to a external state. If we map it to SUCCEEDED, the problem is still there; if we map it to RUNNING, it is equivalent to returning RUNNING when AM is unregistered (in the prior comment). In addition, the problem is not limited to SUCCEEDED only, is it? FAILED, KILLED and ERROR shouldn't be returned before unregister. In particular, FAILED and KILLED map to two internal states respectively. In this case, I'm afraid it's even more difficult to find state in between to execute unregister. 1) JobImpl go to succeeded/terminal state only after app master serviceStop() has completed? In addition to the reason above, if serviceStop() is completed, the client is already unable to get the latest state from AM. 2) Change the MRClientProtocol to return job.state as the previous job state when job.state == succeeded. Then when MRClientProtocol.stop() is called it starts returning job.state == succeeded for 5 secs. Then it stops and the app master stops. +1 for this method. It sounds work with limited touch. Again, we should consider FAILED, KILLED and ERROR as well, right?
          Hide
          bikassaha Bikas Saha added a comment -

          Thats right. MRClientProtocolService should be the last service to stop and should return the state previous to the final termination state of the job. It will return the final termination state when it begins to stop. Do it for 5 secs and then stop itself. The termination state could be failed/killed/error/succeeded. ie. the user continues to see the previous state until its safe to see the final termination state.

          Show
          bikassaha Bikas Saha added a comment - Thats right. MRClientProtocolService should be the last service to stop and should return the state previous to the final termination state of the job. It will return the final termination state when it begins to stop. Do it for 5 secs and then stop itself. The termination state could be failed/killed/error/succeeded. ie. the user continues to see the previous state until its safe to see the final termination state.
          Hide
          zjshen Zhijie Shen added a comment -

          Created the patch. A bit different from what Bikas was proposed:

          1. Not wait until MRClientProtocolSerivice stops, but wait until unregister is called, which is in RMCommunicator#serviceStop. The delay is enough already.

          2. Returning the prior state when the job is not unregistered in JobImpl#getState(), such that both client and webUI can get consistent state.

          Show
          zjshen Zhijie Shen added a comment - Created the patch. A bit different from what Bikas was proposed: 1. Not wait until MRClientProtocolSerivice stops, but wait until unregister is called, which is in RMCommunicator#serviceStop. The delay is enough already. 2. Returning the prior state when the job is not unregistered in JobImpl#getState(), such that both client and webUI can get consistent state.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12603873/MAPREDUCE-5505.1.patch
          against trunk revision .

          -1 patch. Trunk compilation may be broken.

          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4008//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12603873/MAPREDUCE-5505.1.patch against trunk revision . -1 patch . Trunk compilation may be broken. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4008//console This message is automatically generated.
          Hide
          zjshen Zhijie Shen added a comment -

          The compilation failure seems to be unrelated. Reupload the patch to kick out Hadoop QA again.

          Show
          zjshen Zhijie Shen added a comment - The compilation failure seems to be unrelated. Reupload the patch to kick out Hadoop QA again.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12603894/MAPREDUCE-5505.1.patch
          against trunk revision .

          -1 patch. Trunk compilation may be broken.

          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4009//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12603894/MAPREDUCE-5505.1.patch against trunk revision . -1 patch . Trunk compilation may be broken. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4009//console This message is automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12603905/MAPREDUCE-5505.1.patch
          against trunk revision .

          -1 patch. Trunk compilation may be broken.

          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4010//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12603905/MAPREDUCE-5505.1.patch against trunk revision . -1 patch . Trunk compilation may be broken. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4010//console This message is automatically generated.
          Hide
          jianhe Jian He added a comment -

          One more issue is that we may need better exception handling in RMCommunicator.unregister. Now, it's simply ignoring any exceptions coming out of finishApplicationMaster. It can be the case that finishApplicationMaster throws exception before unregister event is sent out. Since MR is ignoring exceptions, JobClient will see SUCCEEDED, but on RM side, since unregister event is not received,RMApp stays on running and eventually failed.

          Show
          jianhe Jian He added a comment - One more issue is that we may need better exception handling in RMCommunicator.unregister. Now, it's simply ignoring any exceptions coming out of finishApplicationMaster. It can be the case that finishApplicationMaster throws exception before unregister event is sent out. Since MR is ignoring exceptions, JobClient will see SUCCEEDED, but on RM side, since unregister event is not received,RMApp stays on running and eventually failed.
          Hide
          bikassaha Bikas Saha added a comment -

          I dont think making jobimpl aware of all this is a good idea. MRClientService is already the last thing to stop. So its a good place to contain this race condition handling logic. We will probably need to move the getReport() method from JobImpl -> MRClientService so that the translation happens over there.
          However, if this needs to be urgently fixed and other folks think this is a good idea then I am ok with this approach.

          Show
          bikassaha Bikas Saha added a comment - I dont think making jobimpl aware of all this is a good idea. MRClientService is already the last thing to stop. So its a good place to contain this race condition handling logic. We will probably need to move the getReport() method from JobImpl -> MRClientService so that the translation happens over there. However, if this needs to be urgently fixed and other folks think this is a good idea then I am ok with this approach.
          Hide
          zjshen Zhijie Shen added a comment -

          JobClient will see SUCCEEDED, but on RM side, since unregister event is not received,RMApp stays on running and eventually failed

          JobClient won't see SUCCEEDED with the patch. Given exception is thrown by finishApplicationMaster, markUnregistered will not be called, and JobClient will still see RUNNING.

          Anyway, +1 for better handling the exception from finishApplicationMaster. Differentiate the handling of, for example, authorization and non-existence.

          Show
          zjshen Zhijie Shen added a comment - JobClient will see SUCCEEDED, but on RM side, since unregister event is not received,RMApp stays on running and eventually failed JobClient won't see SUCCEEDED with the patch. Given exception is thrown by finishApplicationMaster, markUnregistered will not be called, and JobClient will still see RUNNING. Anyway, +1 for better handling the exception from finishApplicationMaster. Differentiate the handling of, for example, authorization and non-existence.
          Hide
          zjshen Zhijie Shen added a comment -

          I dont think making jobimpl aware of all this is a good idea.

          The reason to make JobImpl to be aware of unregister is that JobImpl#getState() needs the info to decide whether it returns the final state (SUCCEEDED, FAILED, KILLED, ERROR) or the prior state (RUNNING). JobImpl#getState() is called by not only getReport() but also JobInfo. As I mentioned above, I did this in getState() to ensure client protocol and web UI can see the consistent state.

          Therefore, in this case, we can move the getReport() JobImpl -> MRClientService, but it is not necessary.

          MRClientService is already the last thing to stop. So its a good place to contain this race condition handling logic.

          +1. Agree it's a good place. When MRAppMaster#serviceStop is called, RMCommunicator#serviceStop is called before MRClientService#serviceStop, where unregister has already been executed. Therefore after this execution, it's already safe to notify the client/webUI of the final state of the job, because one AM has exact one job.

          Show
          zjshen Zhijie Shen added a comment - I dont think making jobimpl aware of all this is a good idea. The reason to make JobImpl to be aware of unregister is that JobImpl#getState() needs the info to decide whether it returns the final state (SUCCEEDED, FAILED, KILLED, ERROR) or the prior state (RUNNING). JobImpl#getState() is called by not only getReport() but also JobInfo. As I mentioned above, I did this in getState() to ensure client protocol and web UI can see the consistent state. Therefore, in this case, we can move the getReport() JobImpl -> MRClientService, but it is not necessary. MRClientService is already the last thing to stop. So its a good place to contain this race condition handling logic. +1. Agree it's a good place. When MRAppMaster#serviceStop is called, RMCommunicator#serviceStop is called before MRClientService#serviceStop, where unregister has already been executed. Therefore after this execution, it's already safe to notify the client/webUI of the final state of the job, because one AM has exact one job.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12603905/MAPREDUCE-5505.1.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The following test timeouts occurred in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app:

          org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4013//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4013//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12603905/MAPREDUCE-5505.1.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The following test timeouts occurred in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app: org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4013//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4013//console This message is automatically generated.
          Hide
          zjshen Zhijie Shen added a comment -

          The test failure should be unrelated. See MAPREDUCE-5514.

          Show
          zjshen Zhijie Shen added a comment - The test failure should be unrelated. See MAPREDUCE-5514 .
          Hide
          jianhe Jian He added a comment -

          markUnregistered will not be called, and JobClient will still see RUNNING.

          Correct, JobClient will see RUNNING until AM exits, in which case JobClient will keep waiting until next AM comes up(MAPREDUCE-5488 made this change).
          Here we make a decision that if unregister call fails, the MR job is deemed as fail and will be restarted by RM.

          isUnregistered use atomic boolean ?

          test case: also assert job state is running before markUnregistered is called.

          Show
          jianhe Jian He added a comment - markUnregistered will not be called, and JobClient will still see RUNNING. Correct, JobClient will see RUNNING until AM exits, in which case JobClient will keep waiting until next AM comes up( MAPREDUCE-5488 made this change). Here we make a decision that if unregister call fails, the MR job is deemed as fail and will be restarted by RM. isUnregistered use atomic boolean ? test case: also assert job state is running before markUnregistered is called.
          Hide
          jianhe Jian He added a comment -
          case REBOOT:
                  if (isUnregistered && appContext.isLastAMRetry()) {
                    return JobState.ERROR;
          

          We probably don't need "appContext.isLastAMRetry()" this check here, since if this is the last retry, the app will fail on RM side. After this AM exits, JobClient is able to query RM for final status, in which case JobClient will be told FAILED. This is good for all transitions to follow the same logic.

          If that's the case we can create a common function to handle the logic that for every final state, if registered return final state, otherwise return the previous state

          Show
          jianhe Jian He added a comment - case REBOOT: if (isUnregistered && appContext.isLastAMRetry()) { return JobState.ERROR; We probably don't need "appContext.isLastAMRetry()" this check here, since if this is the last retry, the app will fail on RM side. After this AM exits, JobClient is able to query RM for final status, in which case JobClient will be told FAILED. This is good for all transitions to follow the same logic. If that's the case we can create a common function to handle the logic that for every final state, if registered return final state, otherwise return the previous state
          Hide
          bikassaha Bikas Saha added a comment -

          Are we sure that previous state is always RUNNING before FAILED?

          +      case FAILED:
          +        if (isUnregistered) {
          +          return JobState.FAILED;
          +        } else {
          +          return JobState.RUNNING;
          

          Instead of isUnregistered, let us create an AtomicBoolean called safeToReportTerminationToUser. Instead of JobImpl, this boolean can be made visible via the AppContext object so that everyone has access to it. When to set the boolean to true? We could do it in RMCommunicator after unregister succeeds (like in this patch). Or we can do it in MRClientService.serviceStop(). Since MRClientService is the last service to stop() we can be sure that everything finished nicely. MRClientService.serviceStop() can set the boolean. Then we can move the sleep(5sec) from MRAppMaster to MRClientService.serviceStop() after setting the boolean.
          We should leave a comment explaining this in MRAppMaster.shutdown() before the call to clientService.stop() so that its easy for someone else to track this logic.

          Please do run single node tests to verify the behavior for real along with RM restart.

          Show
          bikassaha Bikas Saha added a comment - Are we sure that previous state is always RUNNING before FAILED? + case FAILED: + if (isUnregistered) { + return JobState.FAILED; + } else { + return JobState.RUNNING; Instead of isUnregistered, let us create an AtomicBoolean called safeToReportTerminationToUser. Instead of JobImpl, this boolean can be made visible via the AppContext object so that everyone has access to it. When to set the boolean to true? We could do it in RMCommunicator after unregister succeeds (like in this patch). Or we can do it in MRClientService.serviceStop(). Since MRClientService is the last service to stop() we can be sure that everything finished nicely. MRClientService.serviceStop() can set the boolean. Then we can move the sleep(5sec) from MRAppMaster to MRClientService.serviceStop() after setting the boolean. We should leave a comment explaining this in MRAppMaster.shutdown() before the call to clientService.stop() so that its easy for someone else to track this logic. Please do run single node tests to verify the behavior for real along with RM restart.
          Hide
          jianhe Jian He added a comment -

          We probably don't need "appContext.isLastAMRetry()" this check here,

          In fact, in case of REBOOT,we can always return RUNNING. Either the AM is restarted and the jobClient continues to run, or the AM failed because of unregister fail or this is lastRetry, in which case RM can tell JobClient that the app FAILED.

          Show
          jianhe Jian He added a comment - We probably don't need "appContext.isLastAMRetry()" this check here, In fact, in case of REBOOT,we can always return RUNNING. Either the AM is restarted and the jobClient continues to run, or the AM failed because of unregister fail or this is lastRetry, in which case RM can tell JobClient that the app FAILED.
          Hide
          bikassaha Bikas Saha added a comment -

          We could do that but lets leave it out of the scope of the current jira. For this one, lets keep doing what we used to do.

          Show
          bikassaha Bikas Saha added a comment - We could do that but lets leave it out of the scope of the current jira. For this one, lets keep doing what we used to do.
          Hide
          zjshen Zhijie Shen added a comment -

          Thanks Jian and Bikas for the comments. I've uploaded a new patch, which uses atomic boolean instead. It is placed at MRAppMaster, and exposed through AppContext, as other variables are. As the flag is put in MRAppMaster, instead of setting it in ClientService#serviceStop(), MRAppMaster#shutDownJob(), where other services have been stopped already, and ClientService is to be stopped. Therefore, it should be equivalent to what Bikas proposed, but simplify the code.

          Show
          zjshen Zhijie Shen added a comment - Thanks Jian and Bikas for the comments. I've uploaded a new patch, which uses atomic boolean instead. It is placed at MRAppMaster, and exposed through AppContext, as other variables are. As the flag is put in MRAppMaster, instead of setting it in ClientService#serviceStop(), MRAppMaster#shutDownJob(), where other services have been stopped already, and ClientService is to be stopped. Therefore, it should be equivalent to what Bikas proposed, but simplify the code.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12604165/MAPREDUCE-5505.3.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The following test timeouts occurred in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs:

          org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4018//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4018//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12604165/MAPREDUCE-5505.3.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 4 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The following test timeouts occurred in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs: org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4018//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4018//console This message is automatically generated.
          Hide
          zjshen Zhijie Shen added a comment -

          The test failure is unrelated. MAPREUDE-5514 is handling it separately.

          Show
          zjshen Zhijie Shen added a comment - The test failure is unrelated. MAPREUDE-5514 is handling it separately.
          Hide
          bikassaha Bikas Saha added a comment -

          Typo - already ...

          // Except ClientService, other services are already, it is safe to let
          

          Lets use a true value to be back-compatible in case it gets used.

          +  public boolean safeToReportTerminationToUser() {
          +    // bogus - Not Required
          +    return false;
          

          Typo - mock

          +    AppContext mocContext = mock(AppContext.class);
          

          Please put a comment for this non-obvious code. We need it because noone is calling shutdown job right. Alternatively, can MRApp.createJob() be changed to call MRAppMaster.shutdown() or set the boolean value to true. This would be closer to ideal that the current approach.

          +    boolean safe = this.safeToReportTerminationToUser.get();
          +    if (!safe) {
          +      this.safeToReportTerminationToUser.set(true);
          +    }
          

          Can we have a test that verifies the main straightline case. Job succeeds and returns running until the boolean is set.?

          How can we be sure that the previous state == RUNNING

          +      case KILL_ABORT:
          +      case KILLED:
          +        if (appContext.safeToReportTerminationToUser()) {
          +          return JobState.KILLED;
          +        } else {
          +          return JobState.RUNNING;
          

          Has this been tested on a single node cluster with a real job?

          Show
          bikassaha Bikas Saha added a comment - Typo - already ... // Except ClientService, other services are already, it is safe to let Lets use a true value to be back-compatible in case it gets used. + public boolean safeToReportTerminationToUser() { + // bogus - Not Required + return false ; Typo - mock + AppContext mocContext = mock(AppContext.class); Please put a comment for this non-obvious code. We need it because noone is calling shutdown job right. Alternatively, can MRApp.createJob() be changed to call MRAppMaster.shutdown() or set the boolean value to true. This would be closer to ideal that the current approach. + boolean safe = this .safeToReportTerminationToUser.get(); + if (!safe) { + this .safeToReportTerminationToUser.set( true ); + } Can we have a test that verifies the main straightline case. Job succeeds and returns running until the boolean is set.? How can we be sure that the previous state == RUNNING + case KILL_ABORT: + case KILLED: + if (appContext.safeToReportTerminationToUser()) { + return JobState.KILLED; + } else { + return JobState.RUNNING; Has this been tested on a single node cluster with a real job?
          Hide
          zjshen Zhijie Shen added a comment -

          Thanks Bikas Saha for reviewing the patch. I've updated it accordingly.

          Typo - already ...

          Fixed

          Lets use a true value to be back-compatible in case it gets used.

          Fixed

          Typo - mock

          Fixed

          Please put a comment for this non-obvious code. We need it because noone is calling shutdown job right. Alternatively, can MRApp.createJob() be changed to call MRAppMaster.shutdown() or set the boolean value to true. This would be closer to ideal that the current approach.

          Agree the code is non-obvious. Instead of moving setting safeToReportTerminationToUser to MRApp.createJob(), I moved it to the constructor of MRApp, because safeToReportTerminationToUser is the per MRAppMaster variable.

          Can we have a test that verifies the main straightline case. Job succeeds and returns running until the boolean is set.?

          Added TestMRApp#testJobSuccess

          How can we be sure that the previous state == RUNNING

          Its a general issue. To solve it in all the transitions of JobImpl, I added the code to remember the last non-final state. Then, whenever safeToReportTerminationToUser is false, JobImpl returns the stored previous state instead of the final state, i.e., SUCCEEDED, FAILED, KILLED and ERROR.

          Has this been tested on a single node cluster with a real job?

          Tested locally. The job client saw RUNNING until AM got unregistered.

          Show
          zjshen Zhijie Shen added a comment - Thanks Bikas Saha for reviewing the patch. I've updated it accordingly. Typo - already ... Fixed Lets use a true value to be back-compatible in case it gets used. Fixed Typo - mock Fixed Please put a comment for this non-obvious code. We need it because noone is calling shutdown job right. Alternatively, can MRApp.createJob() be changed to call MRAppMaster.shutdown() or set the boolean value to true. This would be closer to ideal that the current approach. Agree the code is non-obvious. Instead of moving setting safeToReportTerminationToUser to MRApp.createJob(), I moved it to the constructor of MRApp, because safeToReportTerminationToUser is the per MRAppMaster variable. Can we have a test that verifies the main straightline case. Job succeeds and returns running until the boolean is set.? Added TestMRApp#testJobSuccess How can we be sure that the previous state == RUNNING Its a general issue. To solve it in all the transitions of JobImpl, I added the code to remember the last non-final state. Then, whenever safeToReportTerminationToUser is false, JobImpl returns the stored previous state instead of the final state, i.e., SUCCEEDED, FAILED, KILLED and ERROR. Has this been tested on a single node cluster with a real job? Tested locally. The job client saw RUNNING until AM got unregistered.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12604732/MAPREDUCE-5505.4.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 5 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs:

          org.apache.hadoop.mapreduce.v2.app.job.impl.TestJobImpl

          The following test timeouts occurred in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs:

          org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4033//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4033//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12604732/MAPREDUCE-5505.4.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 5 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs: org.apache.hadoop.mapreduce.v2.app.job.impl.TestJobImpl The following test timeouts occurred in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs: org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4033//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4033//console This message is automatically generated.
          Hide
          zjshen Zhijie Shen added a comment -

          Updated the new patch to fix test failure.

          Show
          zjshen Zhijie Shen added a comment - Updated the new patch to fix test failure.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12604741/MAPREDUCE-5505.5.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 5 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The following test timeouts occurred in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs:

          org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4034//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4034//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12604741/MAPREDUCE-5505.5.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 5 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The following test timeouts occurred in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs: org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4034//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4034//console This message is automatically generated.
          Hide
          zjshen Zhijie Shen added a comment -

          Test failure on TestRMContainerAllocator is not related

          Show
          zjshen Zhijie Shen added a comment - Test failure on TestRMContainerAllocator is not related
          Hide
          bikassaha Bikas Saha added a comment -

          Instead of calling rememberLastState in every transition (which is also not future proof) can we call it in JobImpl.handle(). This is where the transition method gets called.

          Similarly, createRunningStubbedJob() can implicitly create a mockContext that returns true. Only in the relevant tests, we can pass in/override the mockContext to return false. This will reduce the code change.

          Rest looks good to me.

          Show
          bikassaha Bikas Saha added a comment - Instead of calling rememberLastState in every transition (which is also not future proof) can we call it in JobImpl.handle(). This is where the transition method gets called. Similarly, createRunningStubbedJob() can implicitly create a mockContext that returns true. Only in the relevant tests, we can pass in/override the mockContext to return false. This will reduce the code change. Rest looks good to me.
          Hide
          zjshen Zhijie Shen added a comment -

          The new patch addressed the last Bikas' comments. Bikas Saha, would you please have a look again?

          Show
          zjshen Zhijie Shen added a comment - The new patch addressed the last Bikas' comments. Bikas Saha , would you please have a look again?
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12604909/MAPREDUCE-5505.6.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 5 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The following test timeouts occurred in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs:

          org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4037//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4037//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12604909/MAPREDUCE-5505.6.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 5 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The following test timeouts occurred in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs: org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4037//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4037//console This message is automatically generated.
          Hide
          zjshen Zhijie Shen added a comment -

          The new patch gives the default value to lastNonFinalState, and recovers some tests that are missed in the last patch

          Show
          zjshen Zhijie Shen added a comment - The new patch gives the default value to lastNonFinalState, and recovers some tests that are missed in the last patch
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12604920/MAPREDUCE-5505.7.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 5 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The following test timeouts occurred in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs:

          org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4038//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4038//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12604920/MAPREDUCE-5505.7.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 5 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The following test timeouts occurred in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs: org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4038//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4038//console This message is automatically generated.
          Hide
          zjshen Zhijie Shen added a comment -

          Again, the test failure is unrelated.

          Show
          zjshen Zhijie Shen added a comment - Again, the test failure is unrelated.
          Hide
          bikassaha Bikas Saha added a comment -

          +1. Committed to trunk, branch-2 and branch-2.1-beta

          Show
          bikassaha Bikas Saha added a comment - +1. Committed to trunk, branch-2 and branch-2.1-beta
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #4464 (See https://builds.apache.org/job/Hadoop-trunk-Commit/4464/)
          MAPREDUCE-5505. Clients should be notified job finished only after job successfully unregistered (Zhijie Shen via bikas) (bikas: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1526071)

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/AppContext.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MockAppContext.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRApp.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRuntimeEstimators.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestJobImpl.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistory.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #4464 (See https://builds.apache.org/job/Hadoop-trunk-Commit/4464/ ) MAPREDUCE-5505 . Clients should be notified job finished only after job successfully unregistered (Zhijie Shen via bikas) (bikas: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1526071 ) /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/AppContext.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MockAppContext.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRApp.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRuntimeEstimators.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestJobImpl.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistory.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk #343 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/343/)
          MAPREDUCE-5505. Clients should be notified job finished only after job successfully unregistered (Zhijie Shen via bikas) (bikas: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1526071)

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/AppContext.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MockAppContext.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRApp.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRuntimeEstimators.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestJobImpl.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistory.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #343 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/343/ ) MAPREDUCE-5505 . Clients should be notified job finished only after job successfully unregistered (Zhijie Shen via bikas) (bikas: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1526071 ) /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/AppContext.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MockAppContext.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRApp.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRuntimeEstimators.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestJobImpl.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistory.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #1559 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1559/)
          MAPREDUCE-5505. Clients should be notified job finished only after job successfully unregistered (Zhijie Shen via bikas) (bikas: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1526071)

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/AppContext.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MockAppContext.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRApp.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRuntimeEstimators.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestJobImpl.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistory.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1559 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1559/ ) MAPREDUCE-5505 . Clients should be notified job finished only after job successfully unregistered (Zhijie Shen via bikas) (bikas: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1526071 ) /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/AppContext.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MockAppContext.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRApp.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRuntimeEstimators.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestJobImpl.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistory.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Hdfs-trunk #1533 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1533/)
          MAPREDUCE-5505. Clients should be notified job finished only after job successfully unregistered (Zhijie Shen via bikas) (bikas: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1526071)

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/AppContext.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MockAppContext.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRApp.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRuntimeEstimators.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestJobImpl.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistory.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk #1533 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1533/ ) MAPREDUCE-5505 . Clients should be notified job finished only after job successfully unregistered (Zhijie Shen via bikas) (bikas: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1526071 ) /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/AppContext.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MockAppContext.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRApp.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRuntimeEstimators.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestJobImpl.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistory.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #347 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/347/)
          MAPREDUCE-5538. Fixed MR AppMaster to send job-notification URL only after the job is really done - a bug caused by MAPREDUCE-5505. Contributed by Zhijie Shen. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1527219)

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestJobEndNotifier.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #347 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/347/ ) MAPREDUCE-5538 . Fixed MR AppMaster to send job-notification URL only after the job is really done - a bug caused by MAPREDUCE-5505 . Contributed by Zhijie Shen. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1527219 ) /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestJobEndNotifier.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Hdfs-trunk #1537 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1537/)
          MAPREDUCE-5538. Fixed MR AppMaster to send job-notification URL only after the job is really done - a bug caused by MAPREDUCE-5505. Contributed by Zhijie Shen. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1527219)

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestJobEndNotifier.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk #1537 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1537/ ) MAPREDUCE-5538 . Fixed MR AppMaster to send job-notification URL only after the job is really done - a bug caused by MAPREDUCE-5505 . Contributed by Zhijie Shen. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1527219 ) /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestJobEndNotifier.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #1563 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1563/)
          MAPREDUCE-5538. Fixed MR AppMaster to send job-notification URL only after the job is really done - a bug caused by MAPREDUCE-5505. Contributed by Zhijie Shen. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1527219)

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestJobEndNotifier.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1563 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1563/ ) MAPREDUCE-5538 . Fixed MR AppMaster to send job-notification URL only after the job is really done - a bug caused by MAPREDUCE-5505 . Contributed by Zhijie Shen. (vinodkv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1527219 ) /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestJobEndNotifier.java
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Closing old tickets that are already shipped in a release.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Closing old tickets that are already shipped in a release.

            People

            • Assignee:
              zjshen Zhijie Shen
              Reporter:
              jianhe Jian He
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development