Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6251

JobClient needs additional retries at a higher level to address not-immediately-consistent dfs corner cases

    Details

    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      The JobClient is used to get job status information for running and completed jobs. Final state and history for a job is communicated from the application master to the job history server via a distributed file system - where the history is uploaded by the application master to the dfs and then scanned/loaded by the jobhistory server. While HDFS has strong consistency guarantees not all Hadoop DFS's do. When used in conjunction with a distributed file system which does not have this guarantee there will be cases where the history server may not see an uploaded file, resulting in the dreaded "no such job" and a null value for the RunningJob in the client.

      1. MAPREDUCE-6251.0.patch
        4 kB
        Craig Welch
      2. MAPREDUCE-6251.1.patch
        7 kB
        Craig Welch
      3. MAPREDUCE-6251.2.patch
        7 kB
        Craig Welch
      4. MAPREDUCE-6251.3.patch
        7 kB
        Craig Welch
      5. MAPREDUCE-6251.4.patch
        8 kB
        Craig Welch
      6. MAPREDUCE-6251.6.patch
        10 kB
        Craig Welch
      7. MAPREDUCE-6251.7.patch
        10 kB
        Craig Welch
      8. MAPREDUCE-6251.8.patch
        10 kB
        Craig Welch
      9. MAPREDUCE-6251.8.patch
        10 kB
        Craig Welch

        Activity

        Hide
        cwelch Craig Welch added a comment -

        Attached is a patch which locates the retry where it is effective in capturing the state and which provides a configurable retry count/interval which will address this issue for most reasonable "eventual consistency" timeframes. Without chaning the overall handoff mechanism to not be based on DFS this is the best type of fix I believe we can achieve. Moving to synchronous calls to report history to the JH is another option I think we should consider, but that is a more significant change I think we will want to consider down the road - in the meantime this should work around the issue for most cases.

        Show
        cwelch Craig Welch added a comment - Attached is a patch which locates the retry where it is effective in capturing the state and which provides a configurable retry count/interval which will address this issue for most reasonable "eventual consistency" timeframes. Without chaning the overall handoff mechanism to not be based on DFS this is the best type of fix I believe we can achieve. Moving to synchronous calls to report history to the JH is another option I think we should consider, but that is a more significant change I think we will want to consider down the road - in the meantime this should work around the issue for most cases.
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12697887/MAPREDUCE-6251.0.patch
        against trunk revision d5855c0.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        -1 findbugs. The patch appears to introduce 13 new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5183//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5183//artifact/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5183//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12697887/MAPREDUCE-6251.0.patch against trunk revision d5855c0. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 13 new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5183//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5183//artifact/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5183//console This message is automatically generated.
        Hide
        cwelch Craig Welch added a comment -

        added unit test

        Show
        cwelch Craig Welch added a comment - added unit test
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 14m 34s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        -1 whitespace 0m 0s The patch has 27 line(s) that end in whitespace.
        +1 javac 7m 31s There were no new javac warning messages.
        +1 javadoc 9m 35s There were no new javadoc warning messages.
        +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 5m 24s The applied patch generated 1 additional checkstyle issues.
        +1 install 1m 33s mvn install still works.
        +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
        +1 findbugs 1m 58s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
        +1 mapreduce tests 1m 42s Tests passed in hadoop-mapreduce-client-core.
        +1 mapreduce tests 107m 5s Tests passed in hadoop-mapreduce-client-jobclient.
            150m 24s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12728001/MAPREDUCE-6251.1.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / d03dcb9
        whitespace https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5447/artifact/patchprocess/whitespace.txt
        checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5447/artifact/patchprocess/checkstyle-result-diff.txt
        hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5447/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
        hadoop-mapreduce-client-jobclient test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5447/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt
        Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5447/testReport/
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5447/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 34s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. -1 whitespace 0m 0s The patch has 27 line(s) that end in whitespace. +1 javac 7m 31s There were no new javac warning messages. +1 javadoc 9m 35s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 5m 24s The applied patch generated 1 additional checkstyle issues. +1 install 1m 33s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 1m 58s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 mapreduce tests 1m 42s Tests passed in hadoop-mapreduce-client-core. +1 mapreduce tests 107m 5s Tests passed in hadoop-mapreduce-client-jobclient.     150m 24s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12728001/MAPREDUCE-6251.1.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / d03dcb9 whitespace https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5447/artifact/patchprocess/whitespace.txt checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5447/artifact/patchprocess/checkstyle-result-diff.txt hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5447/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt hadoop-mapreduce-client-jobclient test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5447/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5447/testReport/ Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5447/console This message was automatically generated.
        Hide
        cwelch Craig Welch added a comment -

        Checkstyle not meaningful, see HADOOP-11869. Attached fix for trailing whitespace (also a questionable concern, but fixed easily enough)

        Show
        cwelch Craig Welch added a comment - Checkstyle not meaningful, see HADOOP-11869 . Attached fix for trailing whitespace (also a questionable concern, but fixed easily enough)
        Hide
        vinodkv Vinod Kumar Vavilapalli added a comment -

        It is not just getJob(). Clients may also make calls to isSuccessful() / getCounters() etc and may run into the same failure. We should wrap them all up in a generic retry..

        Does it also make sense for the server to throw a specific exception when this happens instead of retrying everytime?

        Show
        vinodkv Vinod Kumar Vavilapalli added a comment - It is not just getJob(). Clients may also make calls to isSuccessful() / getCounters() etc and may run into the same failure. We should wrap them all up in a generic retry.. Does it also make sense for the server to throw a specific exception when this happens instead of retrying everytime?
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 14m 46s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 javac 7m 42s There were no new javac warning messages.
        +1 javadoc 10m 1s There were no new javadoc warning messages.
        +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 4m 30s The applied patch generated 1 additional checkstyle issues.
        +1 install 1m 49s mvn install still works.
        +1 eclipse:eclipse 0m 37s The patch built with eclipse:eclipse.
        +1 findbugs 2m 14s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
        +1 mapreduce tests 1m 49s Tests passed in hadoop-mapreduce-client-core.
        -1 mapreduce tests 102m 38s Tests failed in hadoop-mapreduce-client-jobclient.
            146m 31s  



        Reason Tests
        Timed out tests org.apache.hadoop.mapreduce.v2.TestMRJobs



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12728064/MAPREDUCE-6251.2.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 4a3dabd
        checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5449/artifact/patchprocess/checkstyle-result-diff.txt
        hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5449/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
        hadoop-mapreduce-client-jobclient test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5449/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt
        Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5449/testReport/
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5449/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 46s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 javac 7m 42s There were no new javac warning messages. +1 javadoc 10m 1s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 4m 30s The applied patch generated 1 additional checkstyle issues. +1 install 1m 49s mvn install still works. +1 eclipse:eclipse 0m 37s The patch built with eclipse:eclipse. +1 findbugs 2m 14s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 mapreduce tests 1m 49s Tests passed in hadoop-mapreduce-client-core. -1 mapreduce tests 102m 38s Tests failed in hadoop-mapreduce-client-jobclient.     146m 31s   Reason Tests Timed out tests org.apache.hadoop.mapreduce.v2.TestMRJobs Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12728064/MAPREDUCE-6251.2.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 4a3dabd checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5449/artifact/patchprocess/checkstyle-result-diff.txt hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5449/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt hadoop-mapreduce-client-jobclient test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5449/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5449/testReport/ Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5449/console This message was automatically generated.
        Hide
        cwelch Craig Welch added a comment -

        It is not just getJob(). Clients may also make calls to isSuccessful() / getCounters() etc and may run into the same failure. We should wrap them all up in a generic retry..

        Those are all calls that happen on the job we are adding retries for here, so they are getting the benefit of the retry. In fact, anything of this sort looks to go through this getJob bit, so it is such a center point

        Does it also make sense for the server to throw a specific exception when this happens instead of retrying everytime?

        I don't believe so - callers are expecting to get a response for valid queries - e.g., a job which exists and is not aged-out by history, changing the kind of error they get won't help, they're expecting a response, this is to make sure they get one in all but the most extraordinary circumstances (not unreasonable) With this change they appear to effectively always get one, without it there are a small but significant number of cases where they do not.

        Show
        cwelch Craig Welch added a comment - It is not just getJob(). Clients may also make calls to isSuccessful() / getCounters() etc and may run into the same failure. We should wrap them all up in a generic retry.. Those are all calls that happen on the job we are adding retries for here, so they are getting the benefit of the retry. In fact, anything of this sort looks to go through this getJob bit, so it is such a center point Does it also make sense for the server to throw a specific exception when this happens instead of retrying everytime? I don't believe so - callers are expecting to get a response for valid queries - e.g., a job which exists and is not aged-out by history, changing the kind of error they get won't help, they're expecting a response, this is to make sure they get one in all but the most extraordinary circumstances (not unreasonable) With this change they appear to effectively always get one, without it there are a small but significant number of cases where they do not.
        Hide
        cwelch Craig Welch added a comment -

        TestMRJobs passing fine on my box, tried several times to confirm. Assuming intermittent/build server issue.

        Show
        cwelch Craig Welch added a comment - TestMRJobs passing fine on my box, tried several times to confirm. Assuming intermittent/build server issue.
        Hide
        vinodkv Vinod Kumar Vavilapalli added a comment -

        Those are all calls that happen on the job we are adding retries for here, so they are getting the benefit of the retry. In fact, anything of this sort looks to go through this getJob bit, so it is such a center point

        That's good to know. We should definitely rename the configuration properties then. The default retry should also be such that there is no retry - users can explicitly override if they want in the context of specific FileSystems.

        I don't believe so - callers are expecting to get a response for valid queries ...

        I guess a different problem I have is that the retry is always happening, whether going to the history-server or to the AM. Given the problem is only in the context of JobHistoryServer, should it be localized to calls to the HistoryServer?

        Show
        vinodkv Vinod Kumar Vavilapalli added a comment - Those are all calls that happen on the job we are adding retries for here, so they are getting the benefit of the retry. In fact, anything of this sort looks to go through this getJob bit, so it is such a center point That's good to know. We should definitely rename the configuration properties then. The default retry should also be such that there is no retry - users can explicitly override if they want in the context of specific FileSystems. I don't believe so - callers are expecting to get a response for valid queries ... I guess a different problem I have is that the retry is always happening, whether going to the history-server or to the AM. Given the problem is only in the context of JobHistoryServer, should it be localized to calls to the HistoryServer?
        Hide
        cwelch Craig Welch added a comment -

        We should definitely rename the configuration properties then

        Done, now named without the "getjob" part (so "jobclient" generic)

        The default retry should also be such that there is no retry

        Done

        I guess a different problem I have is that the retry is always happening, whether going to the history-server or to the AM. Given the problem is only in the context of JobHistoryServer, should it be localized to calls to the HistoryServer?

        I don't think so - for cases where there is an immediate response (AM or HistoryServer), the retries have no effect and there's no harm in the retry logic being in play. For the case in question, where there is a failure, the client/caller wants the retry so that they are insulated from failures and they want that regardless of where the failure originates (AM or JobHistory).

        Show
        cwelch Craig Welch added a comment - We should definitely rename the configuration properties then Done, now named without the "getjob" part (so "jobclient" generic) The default retry should also be such that there is no retry Done I guess a different problem I have is that the retry is always happening, whether going to the history-server or to the AM. Given the problem is only in the context of JobHistoryServer, should it be localized to calls to the HistoryServer? I don't think so - for cases where there is an immediate response (AM or HistoryServer), the retries have no effect and there's no harm in the retry logic being in play. For the case in question, where there is a failure, the client/caller wants the retry so that they are insulated from failures and they want that regardless of where the failure originates (AM or JobHistory).
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 14m 56s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 javac 7m 39s There were no new javac warning messages.
        +1 javadoc 9m 49s There were no new javadoc warning messages.
        +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 5m 38s The applied patch generated 1 additional checkstyle issues.
        +1 install 1m 36s mvn install still works.
        +1 eclipse:eclipse 0m 45s The patch built with eclipse:eclipse.
        +1 findbugs 1m 58s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
        +1 mapreduce tests 1m 46s Tests passed in hadoop-mapreduce-client-core.
        -1 mapreduce tests 113m 27s Tests failed in hadoop-mapreduce-client-jobclient.
            158m 5s  



        Reason Tests
        Failed unit tests hadoop.mapred.TestMRTimelineEventHandling
        Timed out tests org.apache.hadoop.mapred.TestClusterMRNotification



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12728604/MAPREDUCE-6251.3.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 9fc32c5
        checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5460/artifact/patchprocess/checkstyle-result-diff.txt
        hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5460/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
        hadoop-mapreduce-client-jobclient test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5460/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt
        Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5460/testReport/
        Java 1.7.0_55
        uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5460/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 56s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 javac 7m 39s There were no new javac warning messages. +1 javadoc 9m 49s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 5m 38s The applied patch generated 1 additional checkstyle issues. +1 install 1m 36s mvn install still works. +1 eclipse:eclipse 0m 45s The patch built with eclipse:eclipse. +1 findbugs 1m 58s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 mapreduce tests 1m 46s Tests passed in hadoop-mapreduce-client-core. -1 mapreduce tests 113m 27s Tests failed in hadoop-mapreduce-client-jobclient.     158m 5s   Reason Tests Failed unit tests hadoop.mapred.TestMRTimelineEventHandling Timed out tests org.apache.hadoop.mapred.TestClusterMRNotification Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12728604/MAPREDUCE-6251.3.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 9fc32c5 checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5460/artifact/patchprocess/checkstyle-result-diff.txt hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5460/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt hadoop-mapreduce-client-jobclient test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5460/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5460/testReport/ Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5460/console This message was automatically generated.
        Hide
        vinodkv Vinod Kumar Vavilapalli added a comment -

        Okay, reviewing again after your responses

        • Configuration usually goes into MRJobConfig. Unless you explicitly don't want them publicly visible to end-users naturally.
        • Once you move them to MRJobConfig, the naming convention also changes. Names there follow MR_CLIENT_, DEFAULT_ patterns
        • Document them in mapred-default.xml? Stating when they are needed, and how they should be used in contrast to the lower level retries.
        Show
        vinodkv Vinod Kumar Vavilapalli added a comment - Okay, reviewing again after your responses Configuration usually goes into MRJobConfig. Unless you explicitly don't want them publicly visible to end-users naturally. Once you move them to MRJobConfig, the naming convention also changes. Names there follow MR_CLIENT_ , DEFAULT_ patterns Document them in mapred-default.xml? Stating when they are needed, and how they should be used in contrast to the lower level retries.
        Hide
        cwelch Craig Welch added a comment -

        Updated with recommended move to MRJobConfig

        Show
        cwelch Craig Welch added a comment - Updated with recommended move to MRJobConfig
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 14m 32s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        +1 javac 7m 28s There were no new javac warning messages.
        +1 javadoc 9m 32s There were no new javadoc warning messages.
        +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 1m 10s The applied patch generated 21 new checkstyle issues (total was 554, now 575).
        -1 whitespace 0m 1s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 install 1m 37s mvn install still works.
        +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
        +1 findbugs 1m 56s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
        +1 mapreduce tests 1m 35s Tests passed in hadoop-mapreduce-client-core.
        +1 mapreduce tests 107m 45s Tests passed in hadoop-mapreduce-client-jobclient.
            146m 51s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12730673/MAPREDUCE-6251.4.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 90b3845
        checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5649/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-core.txt
        whitespace https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5649/artifact/patchprocess/whitespace.txt
        hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5649/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
        hadoop-mapreduce-client-jobclient test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5649/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt
        Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5649/testReport/
        Java 1.7.0_55
        uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5649/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 32s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 28s There were no new javac warning messages. +1 javadoc 9m 32s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 10s The applied patch generated 21 new checkstyle issues (total was 554, now 575). -1 whitespace 0m 1s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 37s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 1m 56s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 mapreduce tests 1m 35s Tests passed in hadoop-mapreduce-client-core. +1 mapreduce tests 107m 45s Tests passed in hadoop-mapreduce-client-jobclient.     146m 51s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12730673/MAPREDUCE-6251.4.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 90b3845 checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5649/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-core.txt whitespace https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5649/artifact/patchprocess/whitespace.txt hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5649/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt hadoop-mapreduce-client-jobclient test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5649/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5649/testReport/ Java 1.7.0_55 uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5649/console This message was automatically generated.
        Hide
        vinodkv Vinod Kumar Vavilapalli added a comment -

        Document them in mapred-default.xml? Stating when they are needed, and how they should be used in contrast to the lower level retries.

        Missed? It'll be useful to note that it is really only useful when dealing other file-systems.

        Show
        vinodkv Vinod Kumar Vavilapalli added a comment - Document them in mapred-default.xml? Stating when they are needed, and how they should be used in contrast to the lower level retries. Missed? It'll be useful to note that it is really only useful when dealing other file-systems.
        Hide
        cwelch Craig Welch added a comment -

        With mapred-default.xml entries

        Show
        cwelch Craig Welch added a comment - With mapred-default.xml entries
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 14m 45s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        +1 javac 7m 35s There were no new javac warning messages.
        +1 javadoc 9m 55s There were no new javadoc warning messages.
        +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 1m 3s The applied patch generated 21 new checkstyle issues (total was 554, now 575).
        -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 install 1m 34s mvn install still works.
        +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
        +1 findbugs 1m 56s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
        -1 mapreduce tests 1m 33s Tests failed in hadoop-mapreduce-client-core.
        -1 mapreduce tests 112m 12s Tests failed in hadoop-mapreduce-client-jobclient.
            151m 45s  



        Reason Tests
        Failed unit tests hadoop.mapreduce.lib.input.TestLineRecordReader
          hadoop.mapred.TestLineRecordReader
        Timed out tests org.apache.hadoop.mapreduce.v2.TestNonExistentJob



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12730984/MAPREDUCE-6251.6.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / b725078
        checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5654/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-core.txt
        whitespace https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5654/artifact/patchprocess/whitespace.txt
        hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5654/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
        hadoop-mapreduce-client-jobclient test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5654/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt
        Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5654/testReport/
        Java 1.7.0_55
        uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5654/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 45s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 35s There were no new javac warning messages. +1 javadoc 9m 55s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 3s The applied patch generated 21 new checkstyle issues (total was 554, now 575). -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 34s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 1m 56s The patch does not introduce any new Findbugs (version 2.0.3) warnings. -1 mapreduce tests 1m 33s Tests failed in hadoop-mapreduce-client-core. -1 mapreduce tests 112m 12s Tests failed in hadoop-mapreduce-client-jobclient.     151m 45s   Reason Tests Failed unit tests hadoop.mapreduce.lib.input.TestLineRecordReader   hadoop.mapred.TestLineRecordReader Timed out tests org.apache.hadoop.mapreduce.v2.TestNonExistentJob Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12730984/MAPREDUCE-6251.6.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / b725078 checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5654/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-core.txt whitespace https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5654/artifact/patchprocess/whitespace.txt hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5654/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt hadoop-mapreduce-client-jobclient test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5654/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5654/testReport/ Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5654/console This message was automatically generated.
        Hide
        cwelch Craig Welch added a comment -

        Uploaded patch to fix mistake in mapred-default.xml which caused TestNonExistentJob to fail. TestLineRecordReader passes on my box. Whitespace error is bogus (generally, and in this case specifically as it is a line moved not changed by the patch), but tried to fix it anyway.

        Show
        cwelch Craig Welch added a comment - Uploaded patch to fix mistake in mapred-default.xml which caused TestNonExistentJob to fail. TestLineRecordReader passes on my box. Whitespace error is bogus (generally, and in this case specifically as it is a line moved not changed by the patch), but tried to fix it anyway.
        Hide
        cwelch Craig Welch added a comment -

        ...add line failed to stage for last patch.

        Show
        cwelch Craig Welch added a comment - ...add line failed to stage for last patch.
        Hide
        cwelch Craig Welch added a comment -

        Reattach as I don't see it getting built.

        Show
        cwelch Craig Welch added a comment - Reattach as I don't see it getting built.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 14m 49s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        +1 javac 7m 30s There were no new javac warning messages.
        +1 javadoc 9m 36s There were no new javadoc warning messages.
        +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 1m 6s The applied patch generated 21 new checkstyle issues (total was 554, now 575).
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 39s mvn install still works.
        +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
        +1 findbugs 1m 54s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
        +1 mapreduce tests 1m 36s Tests passed in hadoop-mapreduce-client-core.
        +1 mapreduce tests 107m 55s Tests passed in hadoop-mapreduce-client-jobclient.
            147m 16s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12731647/MAPREDUCE-6251.8.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 6471d18
        checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5690/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-core.txt
        hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5690/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
        hadoop-mapreduce-client-jobclient test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5690/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt
        Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5690/testReport/
        Java 1.7.0_55
        uname Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5690/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 49s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 30s There were no new javac warning messages. +1 javadoc 9m 36s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 6s The applied patch generated 21 new checkstyle issues (total was 554, now 575). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 39s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 1m 54s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 mapreduce tests 1m 36s Tests passed in hadoop-mapreduce-client-core. +1 mapreduce tests 107m 55s Tests passed in hadoop-mapreduce-client-jobclient.     147m 16s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12731647/MAPREDUCE-6251.8.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 6471d18 checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5690/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-core.txt hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5690/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt hadoop-mapreduce-client-jobclient test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5690/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5690/testReport/ Java 1.7.0_55 uname Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5690/console This message was automatically generated.
        Hide
        cwelch Craig Welch added a comment -

        None of the checkstyle warnings appear to be significant.

        Show
        cwelch Craig Welch added a comment - None of the checkstyle warnings appear to be significant.
        Hide
        vinodkv Vinod Kumar Vavilapalli added a comment -

        Okay, looks good. Checking this in.

        Show
        vinodkv Vinod Kumar Vavilapalli added a comment - Okay, looks good. Checking this in.
        Hide
        vinodkv Vinod Kumar Vavilapalli added a comment -

        Committed this to trunk, branch-2 and 2.7. Thanks Craig!

        Show
        vinodkv Vinod Kumar Vavilapalli added a comment - Committed this to trunk, branch-2 and 2.7. Thanks Craig!
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #7809 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7809/)
        MAPREDUCE-6251. Added a new config for JobClient to retry JobStatus calls so that they don't fail on history-server backed by DFSes with not so strong guarantees. Contributed by Craig Welch. (vinodkv: rev f24452d14e9ba48cdb82e5e6e5c10ce5b1407308)

        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/JobClientUnitTest.java
        • hadoop-mapreduce-project/CHANGES.txt
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7809 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7809/ ) MAPREDUCE-6251 . Added a new config for JobClient to retry JobStatus calls so that they don't fail on history-server backed by DFSes with not so strong guarantees. Contributed by Craig Welch. (vinodkv: rev f24452d14e9ba48cdb82e5e6e5c10ce5b1407308) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/JobClientUnitTest.java hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #195 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/195/)
        MAPREDUCE-6251. Added a new config for JobClient to retry JobStatus calls so that they don't fail on history-server backed by DFSes with not so strong guarantees. Contributed by Craig Welch. (vinodkv: rev f24452d14e9ba48cdb82e5e6e5c10ce5b1407308)

        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/JobClientUnitTest.java
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java
        • hadoop-mapreduce-project/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #195 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/195/ ) MAPREDUCE-6251 . Added a new config for JobClient to retry JobStatus calls so that they don't fail on history-server backed by DFSes with not so strong guarantees. Contributed by Craig Welch. (vinodkv: rev f24452d14e9ba48cdb82e5e6e5c10ce5b1407308) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/JobClientUnitTest.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java hadoop-mapreduce-project/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Yarn-trunk #926 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/926/)
        MAPREDUCE-6251. Added a new config for JobClient to retry JobStatus calls so that they don't fail on history-server backed by DFSes with not so strong guarantees. Contributed by Craig Welch. (vinodkv: rev f24452d14e9ba48cdb82e5e6e5c10ce5b1407308)

        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
        • hadoop-mapreduce-project/CHANGES.txt
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/JobClientUnitTest.java
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #926 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/926/ ) MAPREDUCE-6251 . Added a new config for JobClient to retry JobStatus calls so that they don't fail on history-server backed by DFSes with not so strong guarantees. Contributed by Craig Welch. (vinodkv: rev f24452d14e9ba48cdb82e5e6e5c10ce5b1407308) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/JobClientUnitTest.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk #2124 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2124/)
        MAPREDUCE-6251. Added a new config for JobClient to retry JobStatus calls so that they don't fail on history-server backed by DFSes with not so strong guarantees. Contributed by Craig Welch. (vinodkv: rev f24452d14e9ba48cdb82e5e6e5c10ce5b1407308)

        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/JobClientUnitTest.java
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
        • hadoop-mapreduce-project/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2124 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2124/ ) MAPREDUCE-6251 . Added a new config for JobClient to retry JobStatus calls so that they don't fail on history-server backed by DFSes with not so strong guarantees. Contributed by Craig Welch. (vinodkv: rev f24452d14e9ba48cdb82e5e6e5c10ce5b1407308) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/JobClientUnitTest.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java hadoop-mapreduce-project/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #184 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/184/)
        MAPREDUCE-6251. Added a new config for JobClient to retry JobStatus calls so that they don't fail on history-server backed by DFSes with not so strong guarantees. Contributed by Craig Welch. (vinodkv: rev f24452d14e9ba48cdb82e5e6e5c10ce5b1407308)

        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/JobClientUnitTest.java
        • hadoop-mapreduce-project/CHANGES.txt
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #184 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/184/ ) MAPREDUCE-6251 . Added a new config for JobClient to retry JobStatus calls so that they don't fail on history-server backed by DFSes with not so strong guarantees. Contributed by Craig Welch. (vinodkv: rev f24452d14e9ba48cdb82e5e6e5c10ce5b1407308) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/JobClientUnitTest.java hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #194 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/194/)
        MAPREDUCE-6251. Added a new config for JobClient to retry JobStatus calls so that they don't fail on history-server backed by DFSes with not so strong guarantees. Contributed by Craig Welch. (vinodkv: rev f24452d14e9ba48cdb82e5e6e5c10ce5b1407308)

        • hadoop-mapreduce-project/CHANGES.txt
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/JobClientUnitTest.java
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #194 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/194/ ) MAPREDUCE-6251 . Added a new config for JobClient to retry JobStatus calls so that they don't fail on history-server backed by DFSes with not so strong guarantees. Contributed by Craig Welch. (vinodkv: rev f24452d14e9ba48cdb82e5e6e5c10ce5b1407308) hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/JobClientUnitTest.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk #2142 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2142/)
        MAPREDUCE-6251. Added a new config for JobClient to retry JobStatus calls so that they don't fail on history-server backed by DFSes with not so strong guarantees. Contributed by Craig Welch. (vinodkv: rev f24452d14e9ba48cdb82e5e6e5c10ce5b1407308)

        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/JobClientUnitTest.java
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
        • hadoop-mapreduce-project/CHANGES.txt
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2142 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2142/ ) MAPREDUCE-6251 . Added a new config for JobClient to retry JobStatus calls so that they don't fail on history-server backed by DFSes with not so strong guarantees. Contributed by Craig Welch. (vinodkv: rev f24452d14e9ba48cdb82e5e6e5c10ce5b1407308) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/JobClientUnitTest.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml

          People

          • Assignee:
            cwelch Craig Welch
            Reporter:
            cwelch Craig Welch
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development