Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1225

TT successfully localizes a task even though the corresponding cache-file has already changed on DFS.

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.21.0, 0.22.0
    • Fix Version/s: 0.22.0
    • Component/s: tasktracker
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      This happens with the first task of a job being localized on this TT. TT doesn't check if the file on DFS is fresh according to the timestamps set in job-conf during submission. After the first task incorrectly gets localized, all further tasks fail on this TT as expected.

      Found this issue while trying to improve test-case for MAPREUCE-913.

      1. MAPREDUCE-1225.patch
        2 kB
        Zhong Wang
      2. MAPREDUCE-1225.patch
        2 kB
        Zhong Wang
      3. MAPREDUCE-1225.patch
        10 kB
        Zhong Wang
      4. MAPREDUCE-1225.patch
        10 kB
        Zhong Wang
      5. MAPREDUCE-1225.patch
        10 kB
        Chris Douglas

        Activity

        Hide
        Zhong Wang added a comment -

        We should check cache validity even if the cache has not inited.

        Show
        Zhong Wang added a comment - We should check cache validity even if the cache has not inited.
        Hide
        Zhong Wang added a comment -

        Patch has attached.

        Show
        Zhong Wang added a comment - Patch has attached.
        Hide
        Zhong Wang added a comment -

        Grant license to ASF for inclusion in ASF works

        Show
        Zhong Wang added a comment - Grant license to ASF for inclusion in ASF works
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12442143/MAPREDUCE-1225.patch
        against trunk revision 935427.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/118/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/118/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/118/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/118/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12442143/MAPREDUCE-1225.patch against trunk revision 935427. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/118/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/118/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/118/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/118/console This message is automatically generated.
        Hide
        Amareshwari Sriramadasu added a comment -

        Comments on the patch :

        • Calling the method checkCacheStatusValidity even for the cache that has not inited does not solve the problem. checkCacheStatusValidity does the check by calling isExistsAndFresh(). isExistsAndFresh does two checks :
          1. dfsFileStamp != confFileStamp
          2. dfsFileStamp != lcacheStatus.mtime
            Here, the second check is not valid for the file that is not localized. So, isExistsAndFresh() always returns false after the patch. We should refactor the first check into a different method and call only that for cache that is not yet inited. Does that make sense?
        • Can you add a unit test in TestTrackerDistributedCacheManager which fails without the patch and passes after the patch ?
        Show
        Amareshwari Sriramadasu added a comment - Comments on the patch : Calling the method checkCacheStatusValidity even for the cache that has not inited does not solve the problem. checkCacheStatusValidity does the check by calling isExistsAndFresh(). isExistsAndFresh does two checks : dfsFileStamp != confFileStamp dfsFileStamp != lcacheStatus.mtime Here, the second check is not valid for the file that is not localized. So, isExistsAndFresh() always returns false after the patch. We should refactor the first check into a different method and call only that for cache that is not yet inited. Does that make sense? Can you add a unit test in TestTrackerDistributedCacheManager which fails without the patch and passes after the patch ?
        Hide
        Zhong Wang added a comment -

        Thanks Amareshwari! I will attach a new patch soon.

        Show
        Zhong Wang added a comment - Thanks Amareshwari! I will attach a new patch soon.
        Hide
        Zhong Wang added a comment -

        Updated patch from Amareshwari. Patch does the following:
        1. Refactor isExistsAndFresh(). Move the first check into checkStampSinceJobStarted().
        2. Remove the fileStatus argument from localizeCache(), since this arg is never used.
        3. Add a test case in testFreshness() to test whether changing stamp of uninited cache could fail a job.
        4. Modify calls of getLocalCache in TestTrackerDistributedCacheManager to use the right timestamp.

        Thanks a lot!

        Show
        Zhong Wang added a comment - Updated patch from Amareshwari. Patch does the following: 1. Refactor isExistsAndFresh(). Move the first check into checkStampSinceJobStarted(). 2. Remove the fileStatus argument from localizeCache(), since this arg is never used. 3. Add a test case in testFreshness() to test whether changing stamp of uninited cache could fail a job. 4. Modify calls of getLocalCache in TestTrackerDistributedCacheManager to use the right timestamp. Thanks a lot!
        Hide
        Amareshwari Sriramadasu added a comment -

        +1 Patch looks good.
        I verified the new test in TestTrackerDistributedCacheManager is indeed a regression test. It failed without the fix.

        Running the patch through hudson.

        Show
        Amareshwari Sriramadasu added a comment - +1 Patch looks good. I verified the new test in TestTrackerDistributedCacheManager is indeed a regression test. It failed without the fix. Running the patch through hudson.
        Hide
        Zhong Wang added a comment -

        Fix an error dir name in test case. The local path of workDir2 should be TEST_ROOT_DIR/workdir2.

        Show
        Zhong Wang added a comment - Fix an error dir name in test case. The local path of workDir2 should be TEST_ROOT_DIR/workdir2.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12442309/MAPREDUCE-1225.patch
        against trunk revision 936166.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/123/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/123/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/123/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/123/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12442309/MAPREDUCE-1225.patch against trunk revision 936166. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/123/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/123/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/123/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/123/console This message is automatically generated.
        Hide
        Zhong Wang added a comment -

        -1 core tests. The patch failed core unit tests.

        What's wrong with the core tests result? It seems not caused by this patch?

        Show
        Zhong Wang added a comment - -1 core tests. The patch failed core unit tests. What's wrong with the core tests result? It seems not caused by this patch?
        Hide
        Amareshwari Sriramadasu added a comment -

        Submitting latest patch for hudson.

        Show
        Amareshwari Sriramadasu added a comment - Submitting latest patch for hudson.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12442402/MAPREDUCE-1225.patch
        against trunk revision 936166.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/360/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/360/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/360/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/360/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12442402/MAPREDUCE-1225.patch against trunk revision 936166. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/360/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/360/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/360/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/360/console This message is automatically generated.
        Hide
        Zhong Wang added a comment -

        This patch has passed hudson test and people review for about half of a month. Should we commit this to mapreduce trunk?

        Show
        Zhong Wang added a comment - This patch has passed hudson test and people review for about half of a month. Should we commit this to mapreduce trunk?
        Hide
        Chris Douglas added a comment -

        Patch went stale; merged with trunk

        Show
        Chris Douglas added a comment - Patch went stale; merged with trunk
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12446428/MAPREDUCE-1225.patch
        against trunk revision 951793.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/224/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/224/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/224/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/224/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12446428/MAPREDUCE-1225.patch against trunk revision 951793. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/224/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/224/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/224/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/224/console This message is automatically generated.
        Hide
        Amareshwari Sriramadasu added a comment -

        Could not see the test report. Re-running through hudson.

        Show
        Amareshwari Sriramadasu added a comment - Could not see the test report. Re-running through hudson.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12446428/MAPREDUCE-1225.patch
        against trunk revision 955198.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/575/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/575/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/575/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/575/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12446428/MAPREDUCE-1225.patch against trunk revision 955198. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/575/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/575/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/575/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/575/console This message is automatically generated.
        Hide
        Amareshwari Sriramadasu added a comment -

        -1 contrib tests.

        Could not see the test report still. The failure is mostly because of MAPREDUCE-1834. Taking Giri's help for confirmation.

        Show
        Amareshwari Sriramadasu added a comment - -1 contrib tests. Could not see the test report still. The failure is mostly because of MAPREDUCE-1834 . Taking Giri's help for confirmation.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12446428/MAPREDUCE-1225.patch
        against trunk revision 955198.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/365/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/365/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/365/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/365/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12446428/MAPREDUCE-1225.patch against trunk revision 955198. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/365/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/365/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/365/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/365/console This message is automatically generated.
        Hide
        Amareshwari Sriramadasu added a comment -

        Latest hudson run confirms the failure is because of MAPREDUCE-1834. I will check this in.

        Show
        Amareshwari Sriramadasu added a comment - Latest hudson run confirms the failure is because of MAPREDUCE-1834 . I will check this in.
        Hide
        Amareshwari Sriramadasu added a comment -

        I just committed this. Thanks Zhong !

        Show
        Amareshwari Sriramadasu added a comment - I just committed this. Thanks Zhong !

          People

          • Assignee:
            Zhong Wang
            Reporter:
            Vinod Kumar Vavilapalli
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development