Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.20.2
    • Fix Version/s: 0.23.0
    • Component/s: distributed-cache
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      If I pass an invalid symlink as -Dmapred.cache.files=/user/knoguchi/onerecord.txt#abc/abc

      Task only reports a WARN and goes on.

       
      2010-09-16 21:38:49,782 INFO org.apache.hadoop.mapred.TaskRunner: Creating symlink: /0/tmp/mapred-local/taskTracker/knoguchi/distcache/-5031501808205559510_-128488332_1354038698/abc-nn1.def.com/user/knoguchi/onerecord.txt <- /0/tmp/mapred-local/taskTracker/knoguchi/jobcache/job_201008310107_15105/attempt_201008310107_15105_m_000000_0/work/./abc/abc
      2010-09-16 21:38:49,789 WARN org.apache.hadoop.mapred.TaskRunner: Failed to create symlink: /0/tmp/mapred-local/taskTracker/knoguchi/distcache/-5031501808205559510_-128488332_1354038698/abc-nn1.def.com/user/knoguchi/onerecord.txt <- /0/tmp/mapred-local/taskTracker/knoguchi/jobcache/job_201008310107_15105/attempt_201008310107_15105_m_000000_0/work/./abc/abc
      

      I believe we should fail the task at this point.

      1. MAPREDUCE-2074.txt
        8 kB
        Priyo Mustafi
      2. MAPREDUCE-2074.txt
        6 kB
        Priyo Mustafi
      3. MAPREDUCE-2074.txt
        6 kB
        Priyo Mustafi

        Activity

        Koji Noguchi created issue -
        Nigel Daley made changes -
        Field Original Value New Value
        Fix Version/s 0.22.0 [ 12314184 ]
        Priyo Mustafi made changes -
        Assignee Priyo Mustafi [ priyomustafi ]
        Hide
        Priyo Mustafi added a comment -

        Throw IOException so that tasks can fail for invalid symlinks

        Show
        Priyo Mustafi added a comment - Throw IOException so that tasks can fail for invalid symlinks
        Priyo Mustafi made changes -
        Attachment MAPREDUCE-2074.txt [ 12468703 ]
        Hide
        Konstantin Shvachko added a comment -

        Looks like TestSetupWorkDir fails

        Testcase: testSetupWorkDir took 0.414 sec
        Testcase: testSetupWorkDirDistCacheSymlinkValid took 0.08 sec
        	FAILED
        null
        junit.framework.AssertionFailedError: null
        	at org.apache.hadoop.mapred.TestSetupWorkDir.testSetupWorkDirDistCacheSymlinkValid(TestSetupWorkDir.java:173)
        
        Show
        Konstantin Shvachko added a comment - Looks like TestSetupWorkDir fails Testcase: testSetupWorkDir took 0.414 sec Testcase: testSetupWorkDirDistCacheSymlinkValid took 0.08 sec FAILED null junit.framework.AssertionFailedError: null at org.apache.hadoop.mapred.TestSetupWorkDir.testSetupWorkDirDistCacheSymlinkValid(TestSetupWorkDir.java:173)
        Hide
        Priyo Mustafi added a comment -

        Fixed issue

        Show
        Priyo Mustafi added a comment - Fixed issue
        Priyo Mustafi made changes -
        Attachment MAPREDUCE-2074.txt [ 12469618 ]
        Hide
        Priyo Mustafi added a comment -

        This fix seems important for 0.22

        Show
        Priyo Mustafi added a comment - This fix seems important for 0.22
        Priyo Mustafi made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Fix Version/s 0.22.0 [ 12314184 ]
        Hide
        Priyo Mustafi added a comment -

        Fixed a test failure

        Show
        Priyo Mustafi added a comment - Fixed a test failure
        Priyo Mustafi made changes -
        Attachment MAPREDUCE-2074.txt [ 12470657 ]
        Priyo Mustafi made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Priyo Mustafi made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12470657/MAPREDUCE-2074.txt
        against trunk revision 1074251.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 6 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        +1 system test framework. The patch passed system test framework compile.

        Test results: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/66//testReport/
        Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/66//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/66//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12470657/MAPREDUCE-2074.txt against trunk revision 1074251. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/66//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/66//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/66//console This message is automatically generated.
        Hide
        Tom White added a comment -

        I've just committed this. Thanks Priyo!

        Show
        Tom White added a comment - I've just committed this. Thanks Priyo!
        Tom White made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags [Reviewed]
        Resolution Fixed [ 1 ]
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #617 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/617/)
        MAPREDUCE-2074. Task should fail when symlink creation fails. Contributed by Priyo Mustafi.

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #617 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/617/ ) MAPREDUCE-2074 . Task should fail when symlink creation fails. Contributed by Priyo Mustafi.
        Tom White made changes -
        Fix Version/s 0.23.0 [ 12315570 ]
        Fix Version/s 0.22.0 [ 12314184 ]
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #643 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/643/)

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #643 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/643/ )
        Arun C Murthy made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Priyo Mustafi
            Reporter:
            Koji Noguchi
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development