Hadoop Common
  1. Hadoop Common
  2. HADOOP-1318

Do not fail completed maps on lost tasktrackers if '-reducer NONE' is specified

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.12.3
    • Fix Version/s: 0.18.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      If '-reducer NONE' is specified the map outputs directly go to hdfs, hence there is no reason to fail completed maps on 'lost' tasktrackers.

      1. patch-1318.txt
        2 kB
        Amareshwari Sriramadasu
      2. patch-1318.txt
        2 kB
        Amareshwari Sriramadasu

        Activity

        Hide
        Amareshwari Sriramadasu added a comment -

        Here is patch that dont fail completed maps on lost tasktrackers if '-reducer NONE' is specified.
        This is tested patch.

        Show
        Amareshwari Sriramadasu added a comment - Here is patch that dont fail completed maps on lost tasktrackers if '-reducer NONE' is specified. This is tested patch.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12382158/patch-1318.txt
        against trunk revision 656939.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no tests are needed for this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2486/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2486/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2486/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2486/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12382158/patch-1318.txt against trunk revision 656939. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2486/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2486/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2486/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2486/console This message is automatically generated.
        Hide
        Amareshwari Sriramadasu added a comment -

        Test failure org.apache.hadoop.dfs.TestFsck.testCorruptBlock is not related to the patch.

        Show
        Amareshwari Sriramadasu added a comment - Test failure org.apache.hadoop.dfs.TestFsck.testCorruptBlock is not related to the patch.
        Hide
        Amareshwari Sriramadasu added a comment -

        canceling patch to incorporate comments from Devaraj (offline) to remove the changes in TaskInProgress.

        Show
        Amareshwari Sriramadasu added a comment - canceling patch to incorporate comments from Devaraj (offline) to remove the changes in TaskInProgress.
        Hide
        Amareshwari Sriramadasu added a comment -

        Patch after incorporating Devaraj's comment.

        Show
        Amareshwari Sriramadasu added a comment - Patch after incorporating Devaraj's comment.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12382347/patch-1318.txt
        against trunk revision 658704.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no tests are needed for this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2509/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2509/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2509/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2509/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12382347/patch-1318.txt against trunk revision 658704. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2509/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2509/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2509/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2509/console This message is automatically generated.
        Hide
        Amareshwari Sriramadasu added a comment -

        Testfailure is due to timeout and not related to the patch. And all the tests passed on my machine.

        Show
        Amareshwari Sriramadasu added a comment - Testfailure is due to timeout and not related to the patch. And all the tests passed on my machine.
        Hide
        Devaraj Das added a comment -

        I just committed this. Thanks, Amareshwari!

        Show
        Devaraj Das added a comment - I just committed this. Thanks, Amareshwari!
        Hide
        Hudson added a comment -
        Show
        Hudson added a comment - Integrated in Hadoop-trunk #499 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/499/ )
        Hide
        Amareshwari Sriramadasu added a comment -

        This issue requires to simulate lost task trackers. And cannot be simulated by just killing the task trackers, because we have to make sure that one of the maps running on the tasktrackers is complete. And the tasktracker is running more tasks. Thus it is difficult to write a testcase for the issue. I simulated the scenario on a cluster and tested the patch.

        Show
        Amareshwari Sriramadasu added a comment - This issue requires to simulate lost task trackers. And cannot be simulated by just killing the task trackers, because we have to make sure that one of the maps running on the tasktrackers is complete. And the tasktracker is running more tasks. Thus it is difficult to write a testcase for the issue. I simulated the scenario on a cluster and tested the patch.

          People

          • Assignee:
            Amareshwari Sriramadasu
            Reporter:
            Arun C Murthy
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development