Hadoop YARN
  1. Hadoop YARN
  2. YARN-14

Symlinks to peer distributed cache files no longer work

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.0-alpha
    • Fix Version/s: 2.0.2-alpha, 0.23.3
    • Component/s: nodemanager
    • Labels:
      None

      Description

      Trying to create a symlink to another file that is specified for the distributed cache will fail to create the link. For example:

      hadoop jar ... -files "x,y,x#z"

      will localize the files x and y as x and y, but the z symlink for x will not be created. This is a regression from 1.x behavior.

      1. MAPREDUCE-4514.patch
        15 kB
        Jason Lowe
      2. YARN-14.patch
        15 kB
        Jason Lowe

        Activity

        Hide
        Jason Lowe added a comment -

        This also breaks when trying to create multiple symlinks to the same file, e.g.: x#a,x#b,x#c only creates the symlink for a instead of all three.

        The problem is Container holds a map from resource Path to symlink String, but there could be multiple symlinks to the same source Path.

        Show
        Jason Lowe added a comment - This also breaks when trying to create multiple symlinks to the same file, e.g.: x#a,x#b,x#c only creates the symlink for a instead of all three. The problem is Container holds a map from resource Path to symlink String, but there could be multiple symlinks to the same source Path.
        Hide
        Jason Lowe added a comment -

        Patch that changes Container to map pending and localized resources to List<String> instead of String so resources can have multiple symlink destinations.

        Show
        Jason Lowe added a comment - Patch that changes Container to map pending and localized resources to List<String> instead of String so resources can have multiple symlink destinations.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12539121/MAPREDUCE-4514.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2707//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2707//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12539121/MAPREDUCE-4514.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2707//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2707//console This message is automatically generated.
        Hide
        Jason Lowe added a comment -

        Updated patch for YARN relocation that occurred on trunk. Same patch as before except patch paths have changed.

        Can't mark this as Patch Available until the YARN workflow is fixed.

        Show
        Jason Lowe added a comment - Updated patch for YARN relocation that occurred on trunk. Same patch as before except patch paths have changed. Can't mark this as Patch Available until the YARN workflow is fixed.
        Hide
        Robert Joseph Evans added a comment -

        Changes look good and Jenkins from before passed so +1.

        Show
        Robert Joseph Evans added a comment - Changes look good and Jenkins from before passed so +1.
        Hide
        Robert Joseph Evans added a comment -

        Thanks Jason I put this into trunk, branch-2, branch-2.1.0-alpha and branch-0.23

        Show
        Robert Joseph Evans added a comment - Thanks Jason I put this into trunk, branch-2, branch-2.1.0-alpha and branch-0.23

          People

          • Assignee:
            Jason Lowe
            Reporter:
            Jason Lowe
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development