Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2925

job -status <JOB_ID> is giving continuously info message for completed jobs on the console

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.0, 0.24.0
    • Component/s: mrv2
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      This below message is coming continuously on the console.

      11/09/02 16:00:00 INFO mapred.ClientServiceDelegate: Failed to contact AM for job job_1314955256658_0009  Will retry..
      11/09/02 16:00:00 INFO mapred.ClientServiceDelegate: Application state is completed. Redirecting to job history server null
      11/09/02 16:00:00 INFO mapred.ClientServiceDelegate: Failed to contact AM for job job_1314955256658_0009  Will retry..
      11/09/02 16:00:00 INFO mapred.ClientServiceDelegate: Application state is completed. Redirecting to job history server null
      
      1. MAPREDUCE-2925.patch
        10 kB
        Devaraj K
      2. MAPREDUCE-2925-1.patch
        12 kB
        Devaraj K
      3. MAPREDUCE-2925-2.patch
        12 kB
        Devaraj K
      4. MAPREDUCE-2925-3.patch
        13 kB
        Devaraj K
      5. MAPREDUCE-2925-4.patch
        14 kB
        Vinod Kumar Vavilapalli

        Issue Links

          Activity

          Hide
          Arun C Murthy added a comment -

          Looks like a bug where the client can't deal with absence of job-history server - we should quit after a while...

          Show
          Arun C Murthy added a comment - Looks like a bug where the client can't deal with absence of job-history server - we should quit after a while...
          Hide
          Devaraj K added a comment -

          1. If the application report is null from RM, redirecting to history server because the history server still may aware of the job.

          2. If the AM/History server throws InvocationTargetException then retrying finite number of times. These retries can be configurable with the property "yarn.ipc.client.am.rm.retries" with default value as 7.

          Show
          Devaraj K added a comment - 1. If the application report is null from RM, redirecting to history server because the history server still may aware of the job. 2. If the AM/History server throws InvocationTargetException then retrying finite number of times. These retries can be configurable with the property "yarn.ipc.client.am.rm.retries" with default value as 7.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12494392/MAPREDUCE-2925.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/733//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/733//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12494392/MAPREDUCE-2925.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/733//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/733//console This message is automatically generated.
          Hide
          Arun C Murthy added a comment -

          Devaraj, sorry to come in late.

          I'm concerned about adding application level retries... a better solution is to use the 'rpcTimeout' in getProxy to limit #retries. Can you please use that? Thanks.

          Show
          Arun C Murthy added a comment - Devaraj, sorry to come in late. I'm concerned about adding application level retries... a better solution is to use the 'rpcTimeout' in getProxy to limit #retries. Can you please use that? Thanks.
          Hide
          Devaraj K added a comment -

          Thanks Arun for reviewing and suggestion.

          There are few problems around this.

          1. If the RM doesn't give application report it is giving NullPointerException.
          This can be handled by redirecting to history server as it still may aware of the application.

          2. After redirecting to History Server, if the history server doesn't have information about it(or it failed to give because of some other reason), it is going to infinite loop and keep on printing the message.

          I have faced the similar problem. RM is giving the application report with status as success and then it is redirecting to History server. History server is not able to find the application info, it throwing the exception. That is converting to InvocationTargetException and it is retrying infinitely.

          3. If it throws other than 'YarnRemoteException' and 'InvocationTargetException' also it goes to infinite times. This needs to break at some point.

          Here we need to differentiate remote end exceptions and connection failures to RM/AM/HS, if it is remote end exception then it can be reported directly. If it is connection failure then retry can happen in the RPC and after retries it can be reported.

          Please provide your suggestions.

          Show
          Devaraj K added a comment - Thanks Arun for reviewing and suggestion. There are few problems around this. 1. If the RM doesn't give application report it is giving NullPointerException. This can be handled by redirecting to history server as it still may aware of the application. 2. After redirecting to History Server, if the history server doesn't have information about it(or it failed to give because of some other reason), it is going to infinite loop and keep on printing the message. I have faced the similar problem. RM is giving the application report with status as success and then it is redirecting to History server. History server is not able to find the application info, it throwing the exception. That is converting to InvocationTargetException and it is retrying infinitely. 3. If it throws other than 'YarnRemoteException' and 'InvocationTargetException' also it goes to infinite times. This needs to break at some point. Here we need to differentiate remote end exceptions and connection failures to RM/AM/HS, if it is remote end exception then it can be reported directly. If it is connection failure then retry can happen in the RPC and after retries it can be reported. Please provide your suggestions.
          Hide
          Arun C Murthy added a comment -

          Seems like not having any retries if JobHistory throws an exception is ok?

          Show
          Arun C Murthy added a comment - Seems like not having any retries if JobHistory throws an exception is ok?
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Misssed this one. Here's what I think we should do:

          • Client should check if JobHistoryServer(JHS) is configured or not. To facilitate this, we should not have a default address for JHS on the client. If it is not configured, client should not attempt to reach JHS.
          • In ClientServiceDelegate, we need to unwrap the InvocationTargetException to get the exception thrown by the method. If the wrapped exception is of type YarnRemoteException, we should return from the call. (BTW, this is the real bug you are running into and needs fixing). Otherwise it is a connection error and we should just keep retrying.
          Show
          Vinod Kumar Vavilapalli added a comment - Misssed this one. Here's what I think we should do: Client should check if JobHistoryServer ( JHS ) is configured or not. To facilitate this, we should not have a default address for JHS on the client. If it is not configured, client should not attempt to reach JHS . In ClientServiceDelegate , we need to unwrap the InvocationTargetException to get the exception thrown by the method. If the wrapped exception is of type YarnRemoteException , we should return from the call. (BTW, this is the real bug you are running into and needs fixing). Otherwise it is a connection error and we should just keep retrying.
          Hide
          Sharad Agarwal added a comment -

          Just to add that retry logic is used to reconnect to newly started AM if the previous one crashes. So retrying fixed no of time will affect the AM restart. Instead we should depend on the exception type as mentioned by Vinod in previous comment.

          Show
          Sharad Agarwal added a comment - Just to add that retry logic is used to reconnect to newly started AM if the previous one crashes. So retrying fixed no of time will affect the AM restart. Instead we should depend on the exception type as mentioned by Vinod in previous comment.
          Hide
          Devaraj K added a comment -

          Thanks Vinod and Sharad for clarifying.

          Client should check if JobHistoryServer(JHS) is configured or not. To facilitate this, we should not have a default address for JHS on the client. If it is not configured, client should not attempt to reach JHS.

          It is good idea. We can implement this.

          In ClientServiceDelegate, we need to unwrap the InvocationTargetException to get the exception thrown by the method. If the wrapped exception is of type YarnRemoteException, we should return from the call. (BTW, this is the real bug you are running into and needs fixing). Otherwise it is a connection error and we should just keep retrying.

          I am also thinking exactly same for this case.

          I will update the patch with these changes.

          Show
          Devaraj K added a comment - Thanks Vinod and Sharad for clarifying. Client should check if JobHistoryServer(JHS) is configured or not. To facilitate this, we should not have a default address for JHS on the client. If it is not configured, client should not attempt to reach JHS. It is good idea. We can implement this. In ClientServiceDelegate, we need to unwrap the InvocationTargetException to get the exception thrown by the method. If the wrapped exception is of type YarnRemoteException, we should return from the call. (BTW, this is the real bug you are running into and needs fixing). Otherwise it is a connection error and we should just keep retrying. I am also thinking exactly same for this case. I will update the patch with these changes.
          Hide
          Devaraj K added a comment -

          Updated the patch as per the above comments. Please review this.

          Show
          Devaraj K added a comment - Updated the patch as per the above comments. Please review this.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12494804/MAPREDUCE-2925-1.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.mapred.TestClientRedirect

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/764//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/764//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12494804/MAPREDUCE-2925-1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.mapred.TestClientRedirect +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/764//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/764//console This message is automatically generated.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12494821/MAPREDUCE-2925-2.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/765//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/765//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12494821/MAPREDUCE-2925-2.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/765//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/765//console This message is automatically generated.
          Hide
          Arun C Murthy added a comment -

          Devaraj, this is close!

          Couple of nits:

          1. Please add back the comment explaining why getApplicationReport is null.
          2. There is a case you still need to handle for null in the 'catch (IOException)' block of getProxy.
          3. Please use LOG.warn(msg, exception) rather than 2 LOG.warn (go ahead and fix the other instance there too).
          Show
          Arun C Murthy added a comment - Devaraj, this is close! Couple of nits: Please add back the comment explaining why getApplicationReport is null. There is a case you still need to handle for null in the 'catch (IOException)' block of getProxy. Please use LOG.warn(msg, exception) rather than 2 LOG.warn (go ahead and fix the other instance there too).
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Few more comments:

          ClientServiceDelegate

          • Please move the application null check into the while loop, it can happen any time. (1)
          • I think it is a bit harsh to throw an exception when HS isn't configured. May be just log the warn the message and return a NotRunningJob. What do you think?
          • We don't use YarnRemoteExceptionPBImpl directly, instead we use RPCUtil.getRemoteException() (Hiding the underlying impl and all). Once you do (1) above, your patch actually won't need this change in the main code, but you will need it in the test. Also there are other similar usages in ClientServiceDelegate though unrelated to your patch. Mind changing them as well?

          TestClientServiceDelegate: Overall looks good, appreciate the meaningfulness of the tests, good job!

          • Again, replace the PB Impl objects creation using BuilderUtils methods or Records.newRecord()
          • I suggest the test names to be testUnknownAppInRM, testHistoryServerNotConfigured (with the two tests related to non-configured JHS combined into one), testRemoteExceptionFromHistoryServer, testRetriesOnConnectionFailure respectively.
          Show
          Vinod Kumar Vavilapalli added a comment - Few more comments: ClientServiceDelegate Please move the application null check into the while loop, it can happen any time. (1) I think it is a bit harsh to throw an exception when HS isn't configured. May be just log the warn the message and return a NotRunningJob . What do you think? We don't use YarnRemoteExceptionPBImpl directly, instead we use RPCUtil.getRemoteException() (Hiding the underlying impl and all). Once you do (1) above, your patch actually won't need this change in the main code, but you will need it in the test. Also there are other similar usages in ClientServiceDelegate though unrelated to your patch. Mind changing them as well? TestClientServiceDelegate: Overall looks good, appreciate the meaningfulness of the tests, good job! Again, replace the PB Impl objects creation using BuilderUtils methods or Records.newRecord() I suggest the test names to be testUnknownAppInRM , testHistoryServerNotConfigured (with the two tests related to non-configured JHS combined into one), testRemoteExceptionFromHistoryServer , testRetriesOnConnectionFailure respectively.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Devaraj (K), any update on this? Tx.

          Show
          Vinod Kumar Vavilapalli added a comment - Devaraj (K), any update on this? Tx.
          Hide
          Devaraj K added a comment -

          Vinod, Patch is ready with the above changes, I will upload it within 30 mins.

          Show
          Devaraj K added a comment - Vinod, Patch is ready with the above changes, I will upload it within 30 mins.
          Hide
          Devaraj K added a comment -

          Updated the patch by fixing the above comments.

          Show
          Devaraj K added a comment - Updated the patch by fixing the above comments.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12495031/MAPREDUCE-2925-3.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/778//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/778//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12495031/MAPREDUCE-2925-3.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/778//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/778//console This message is automatically generated.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Your latest patch looks good. +1.

          Minor update to remove a reference to PB record in the test.

          Show
          Vinod Kumar Vavilapalli added a comment - Your latest patch looks good. +1. Minor update to remove a reference to PB record in the test.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12495044/MAPREDUCE-2925-4.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/779//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/779//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12495044/MAPREDUCE-2925-4.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/779//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/779//console This message is automatically generated.
          Hide
          Devaraj K added a comment -

          Thanks Vinod for reviewing and making change in the testcase.

          Show
          Devaraj K added a comment - Thanks Vinod for reviewing and making change in the testcase.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Thanks Vinod for reviewing and making change in the testcase.

          Sure, not a problem. Please try to catch such issues yourselves the next time on.

          Show
          Vinod Kumar Vavilapalli added a comment - Thanks Vinod for reviewing and making change in the testcase. Sure, not a problem. Please try to catch such issues yourselves the next time on.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          I just committed this to trunk and branch-0.23. Thanks Devaraj (K)!

          Show
          Vinod Kumar Vavilapalli added a comment - I just committed this to trunk and branch-0.23. Thanks Devaraj (K)!
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #928 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/928/)
          MAPREDUCE-2925. Fixed Yarn+MR client code to behave saner with completed jobs. Contributed by Devaraj K.

          vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1172587
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ClientCache.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ClientServiceDelegate.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestClientServiceDelegate.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #928 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/928/ ) MAPREDUCE-2925 . Fixed Yarn+MR client code to behave saner with completed jobs. Contributed by Devaraj K. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1172587 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ClientCache.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ClientServiceDelegate.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestClientServiceDelegate.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #988 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/988/)
          MAPREDUCE-2925. Fixed Yarn+MR client code to behave saner with completed jobs. Contributed by Devaraj K.

          vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1172587
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ClientCache.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ClientServiceDelegate.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestClientServiceDelegate.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #988 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/988/ ) MAPREDUCE-2925 . Fixed Yarn+MR client code to behave saner with completed jobs. Contributed by Devaraj K. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1172587 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ClientCache.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ClientServiceDelegate.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestClientServiceDelegate.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #911 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/911/)
          MAPREDUCE-2925. Fixed Yarn+MR client code to behave saner with completed jobs. Contributed by Devaraj K.

          vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1172587
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ClientCache.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ClientServiceDelegate.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestClientServiceDelegate.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #911 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/911/ ) MAPREDUCE-2925 . Fixed Yarn+MR client code to behave saner with completed jobs. Contributed by Devaraj K. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1172587 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ClientCache.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ClientServiceDelegate.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestClientServiceDelegate.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-0.23-Build #11 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/11/)
          MAPREDUCE-2925. svn merge -c r1172587 --ignore-ancestry ../../trunk/

          vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1172588
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ClientCache.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ClientServiceDelegate.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestClientServiceDelegate.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Build #11 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/11/ ) MAPREDUCE-2925 . svn merge -c r1172587 --ignore-ancestry ../../trunk/ vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1172588 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ClientCache.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ClientServiceDelegate.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestClientServiceDelegate.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #835 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/835/)
          MAPREDUCE-2925. Fixed Yarn+MR client code to behave saner with completed jobs. Contributed by Devaraj K.

          vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1172587
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ClientCache.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ClientServiceDelegate.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestClientServiceDelegate.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #835 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/835/ ) MAPREDUCE-2925 . Fixed Yarn+MR client code to behave saner with completed jobs. Contributed by Devaraj K. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1172587 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ClientCache.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ClientServiceDelegate.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestClientServiceDelegate.java
          Hide
          Jeffrey Naisbitt added a comment -

          This change is causing the following message to be repeated LOTS of times whenever a job is run:
          11/09/19 20:14:32 INFO mapred.ClientCache: HistoryServer is not configured.

          This is caused when we run without having specified the mapreduce.jobhistory.address property. Other places in the code use the default: JHAdminConfig.DEFAULT_MR_HISTORY_ADDRESS if the property is not set.

          Show
          Jeffrey Naisbitt added a comment - This change is causing the following message to be repeated LOTS of times whenever a job is run: 11/09/19 20:14:32 INFO mapred.ClientCache: HistoryServer is not configured. This is caused when we run without having specified the mapreduce.jobhistory.address property. Other places in the code use the default: JHAdminConfig.DEFAULT_MR_HISTORY_ADDRESS if the property is not set.
          Hide
          Devaraj K added a comment -

          Hi Jeffry,
          For every request from job client, if the job history is null it tries to instantiate and it is logging info message if there is no history server configured.
          Default value is using only at the startup of History Server.

          I think it would be better to configure the history server from client side.

          Show
          Devaraj K added a comment - Hi Jeffry, For every request from job client, if the job history is null it tries to instantiate and it is logging info message if there is no history server configured. Default value is using only at the startup of History Server. I think it would be better to configure the history server from client side.
          Hide
          Arun C Murthy added a comment -

          No, I removed that LOG via MAPREDUCE-3040.

          Show
          Arun C Murthy added a comment - No, I removed that LOG via MAPREDUCE-3040 .
          Hide
          Devaraj K added a comment -

          I see the change. Thanks Arun..

          Show
          Devaraj K added a comment - I see the change. Thanks Arun..
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #806 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/806/)
          MAPREDUCE-2925. Fixed Yarn+MR client code to behave saner with completed jobs. Contributed by Devaraj K.

          vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1172587
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ClientCache.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ClientServiceDelegate.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestClientServiceDelegate.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #806 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/806/ ) MAPREDUCE-2925 . Fixed Yarn+MR client code to behave saner with completed jobs. Contributed by Devaraj K. vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1172587 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ClientCache.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ClientServiceDelegate.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestClientServiceDelegate.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Build #14 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/14/)
          MAPREDUCE-2925. svn merge -c r1172587 --ignore-ancestry ../../trunk/

          vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1172588
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ClientCache.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ClientServiceDelegate.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestClientServiceDelegate.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #14 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/14/ ) MAPREDUCE-2925 . svn merge -c r1172587 --ignore-ancestry ../../trunk/ vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1172588 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ClientCache.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ClientServiceDelegate.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestClientServiceDelegate.java

            People

            • Assignee:
              Devaraj K
              Reporter:
              Devaraj K
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development