Hadoop YARN
  1. Hadoop YARN
  2. YARN-14

Symlinks to peer distributed cache files no longer work

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.0-alpha
    • Fix Version/s: 2.0.2-alpha, 0.23.3
    • Component/s: nodemanager
    • Labels:
      None

      Description

      Trying to create a symlink to another file that is specified for the distributed cache will fail to create the link. For example:

      hadoop jar ... -files "x,y,x#z"

      will localize the files x and y as x and y, but the z symlink for x will not be created. This is a regression from 1.x behavior.

      1. MAPREDUCE-4514.patch
        15 kB
        Jason Lowe
      2. YARN-14.patch
        15 kB
        Jason Lowe

        Activity

        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Patch Available Patch Available
        2h 8m 1 Jason Lowe 04/Aug/12 01:07
        Patch Available Patch Available Open Open
        4d 20h 46m 1 Jason Lowe 08/Aug/12 21:53
        Open Open Resolved Resolved
        22h 29m 1 Robert Joseph Evans 09/Aug/12 20:23
        Resolved Resolved Closed Closed
        62d 22h 24m 1 Arun C Murthy 11/Oct/12 18:48
        Arun C Murthy made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Arun C Murthy made changes -
        Affects Version/s 2.0.0-alpha [ 12323271 ]
        Affects Version/s 2.1.0-alpha [ 12322547 ]
        Arun C Murthy made changes -
        Fix Version/s 2.1.0-alpha [ 12322547 ]
        Gavin made changes -
        Workflow jira [ 12720104 ] no-reopen-closed, patch-avail [ 12722147 ]
        Robert Joseph Evans made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Fix Version/s 2.1.0-alpha [ 12322547 ]
        Fix Version/s 2.2.0-alpha [ 12322548 ]
        Fix Version/s 0.23.3 [ 12322841 ]
        Resolution Fixed [ 1 ]
        Hide
        Robert Joseph Evans added a comment -

        Thanks Jason I put this into trunk, branch-2, branch-2.1.0-alpha and branch-0.23

        Show
        Robert Joseph Evans added a comment - Thanks Jason I put this into trunk, branch-2, branch-2.1.0-alpha and branch-0.23
        Hide
        Robert Joseph Evans added a comment -

        Changes look good and Jenkins from before passed so +1.

        Show
        Robert Joseph Evans added a comment - Changes look good and Jenkins from before passed so +1.
        Jason Lowe made changes -
        Attachment YARN-14.patch [ 12539922 ]
        Hide
        Jason Lowe added a comment -

        Updated patch for YARN relocation that occurred on trunk. Same patch as before except patch paths have changed.

        Can't mark this as Patch Available until the YARN workflow is fixed.

        Show
        Jason Lowe added a comment - Updated patch for YARN relocation that occurred on trunk. Same patch as before except patch paths have changed. Can't mark this as Patch Available until the YARN workflow is fixed.
        Jason Lowe made changes -
        Project Hadoop Map/Reduce [ 12310941 ] Hadoop YARN [ 12313722 ]
        Key MAPREDUCE-4514 YARN-14
        Workflow no-reopen-closed, patch-avail [ 12719250 ] jira [ 12720104 ]
        Status Patch Available [ 10002 ] Open [ 1 ]
        Affects Version/s 2.1.0-alpha [ 12322547 ]
        Affects Version/s 0.23.3 [ 12320060 ]
        Affects Version/s 2.0.1-alpha [ 12322466 ]
        Target Version/s 0.23.3, 2.2.0-alpha [ 12320060, 12322471 ]
        Component/s nodemanager [ 12319323 ]
        Component/s distributed-cache [ 12313271 ]
        Component/s mrv2 [ 12314301 ]
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12539121/MAPREDUCE-4514.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2707//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2707//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12539121/MAPREDUCE-4514.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2707//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2707//console This message is automatically generated.
        Jason Lowe made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Target Version/s 0.23.3, 2.2.0-alpha [ 12320060, 12322471 ]
        Jason Lowe made changes -
        Field Original Value New Value
        Attachment MAPREDUCE-4514.patch [ 12539121 ]
        Hide
        Jason Lowe added a comment -

        Patch that changes Container to map pending and localized resources to List<String> instead of String so resources can have multiple symlink destinations.

        Show
        Jason Lowe added a comment - Patch that changes Container to map pending and localized resources to List<String> instead of String so resources can have multiple symlink destinations.
        Hide
        Jason Lowe added a comment -

        This also breaks when trying to create multiple symlinks to the same file, e.g.: x#a,x#b,x#c only creates the symlink for a instead of all three.

        The problem is Container holds a map from resource Path to symlink String, but there could be multiple symlinks to the same source Path.

        Show
        Jason Lowe added a comment - This also breaks when trying to create multiple symlinks to the same file, e.g.: x#a,x#b,x#c only creates the symlink for a instead of all three. The problem is Container holds a map from resource Path to symlink String, but there could be multiple symlinks to the same source Path.
        Jason Lowe created issue -

          People

          • Assignee:
            Jason Lowe
            Reporter:
            Jason Lowe
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development