Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-1318

Do not fail completed maps on lost tasktrackers if '-reducer NONE' is specified

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 0.12.3
    • Fix Version/s: 0.18.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      If '-reducer NONE' is specified the map outputs directly go to hdfs, hence there is no reason to fail completed maps on 'lost' tasktrackers.

      1. patch-1318.txt
        2 kB
        Amareshwari Sriramadasu
      2. patch-1318.txt
        2 kB
        Amareshwari Sriramadasu

        Activity

        Hide
        amareshwari Amareshwari Sriramadasu added a comment -

        This issue requires to simulate lost task trackers. And cannot be simulated by just killing the task trackers, because we have to make sure that one of the maps running on the tasktrackers is complete. And the tasktracker is running more tasks. Thus it is difficult to write a testcase for the issue. I simulated the scenario on a cluster and tested the patch.

        Show
        amareshwari Amareshwari Sriramadasu added a comment - This issue requires to simulate lost task trackers. And cannot be simulated by just killing the task trackers, because we have to make sure that one of the maps running on the tasktrackers is complete. And the tasktracker is running more tasks. Thus it is difficult to write a testcase for the issue. I simulated the scenario on a cluster and tested the patch.
        Hide
        hudson Hudson added a comment -
        Show
        hudson Hudson added a comment - Integrated in Hadoop-trunk #499 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/499/ )
        Hide
        devaraj Devaraj Das added a comment -

        I just committed this. Thanks, Amareshwari!

        Show
        devaraj Devaraj Das added a comment - I just committed this. Thanks, Amareshwari!
        Hide
        amareshwari Amareshwari Sriramadasu added a comment -

        Testfailure is due to timeout and not related to the patch. And all the tests passed on my machine.

        Show
        amareshwari Amareshwari Sriramadasu added a comment - Testfailure is due to timeout and not related to the patch. And all the tests passed on my machine.
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12382347/patch-1318.txt
        against trunk revision 658704.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no tests are needed for this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2509/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2509/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2509/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2509/console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12382347/patch-1318.txt against trunk revision 658704. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2509/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2509/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2509/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2509/console This message is automatically generated.
        Hide
        amareshwari Amareshwari Sriramadasu added a comment -

        Patch after incorporating Devaraj's comment.

        Show
        amareshwari Amareshwari Sriramadasu added a comment - Patch after incorporating Devaraj's comment.
        Hide
        amareshwari Amareshwari Sriramadasu added a comment -

        canceling patch to incorporate comments from Devaraj (offline) to remove the changes in TaskInProgress.

        Show
        amareshwari Amareshwari Sriramadasu added a comment - canceling patch to incorporate comments from Devaraj (offline) to remove the changes in TaskInProgress.
        Hide
        amareshwari Amareshwari Sriramadasu added a comment -

        Test failure org.apache.hadoop.dfs.TestFsck.testCorruptBlock is not related to the patch.

        Show
        amareshwari Amareshwari Sriramadasu added a comment - Test failure org.apache.hadoop.dfs.TestFsck.testCorruptBlock is not related to the patch.
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12382158/patch-1318.txt
        against trunk revision 656939.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no tests are needed for this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2486/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2486/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2486/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2486/console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12382158/patch-1318.txt against trunk revision 656939. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2486/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2486/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2486/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2486/console This message is automatically generated.
        Hide
        amareshwari Amareshwari Sriramadasu added a comment -

        Here is patch that dont fail completed maps on lost tasktrackers if '-reducer NONE' is specified.
        This is tested patch.

        Show
        amareshwari Amareshwari Sriramadasu added a comment - Here is patch that dont fail completed maps on lost tasktrackers if '-reducer NONE' is specified. This is tested patch.

          People

          • Assignee:
            amareshwari Amareshwari Sriramadasu
            Reporter:
            acmurthy Arun C Murthy
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development