Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2839

MR Jobs fail on a secure cluster with viewfs

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.23.0
    • Component/s: None
    • Labels:
      None

      Description

      TokenCache needs to use the new FileSystem.getDelegationTokens api for it to work with viewfs.

      1. MR2839_0.patch
        6 kB
        Siddharth Seth
      2. MR2839_279_0.patch
        6 kB
        Siddharth Seth
      3. MR2839_trunk_1.patch
        5 kB
        Siddharth Seth

        Activity

        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #742 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/742/)
        MAPREDUCE-2839. Fixed TokenCache to get delegation tokens using both new and old apis. Contributed by Siddharth Seth.

        acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1157420
        Files :

        • /hadoop/common/trunk/mapreduce/src/test/mapred/org/apache/hadoop/mapreduce/security/TestTokenCache.java
        • /hadoop/common/trunk/mapreduce/CHANGES.txt
        • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/security/TokenCache.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #742 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/742/ ) MAPREDUCE-2839 . Fixed TokenCache to get delegation tokens using both new and old apis. Contributed by Siddharth Seth. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1157420 Files : /hadoop/common/trunk/mapreduce/src/test/mapred/org/apache/hadoop/mapreduce/security/TestTokenCache.java /hadoop/common/trunk/mapreduce/CHANGES.txt /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/security/TokenCache.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #754 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/754/)
        MAPREDUCE-2839. Fixed TokenCache to get delegation tokens using both new and old apis. Contributed by Siddharth Seth.

        acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1157420
        Files :

        • /hadoop/common/trunk/mapreduce/src/test/mapred/org/apache/hadoop/mapreduce/security/TestTokenCache.java
        • /hadoop/common/trunk/mapreduce/CHANGES.txt
        • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/security/TokenCache.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #754 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/754/ ) MAPREDUCE-2839 . Fixed TokenCache to get delegation tokens using both new and old apis. Contributed by Siddharth Seth. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1157420 Files : /hadoop/common/trunk/mapreduce/src/test/mapred/org/apache/hadoop/mapreduce/security/TestTokenCache.java /hadoop/common/trunk/mapreduce/CHANGES.txt /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/security/TokenCache.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #766 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/766/)
        MAPREDUCE-2839. Fixed TokenCache to get delegation tokens using both new and old apis. Contributed by Siddharth Seth.

        acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1157420
        Files :

        • /hadoop/common/trunk/mapreduce/src/test/mapred/org/apache/hadoop/mapreduce/security/TestTokenCache.java
        • /hadoop/common/trunk/mapreduce/CHANGES.txt
        • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/security/TokenCache.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #766 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/766/ ) MAPREDUCE-2839 . Fixed TokenCache to get delegation tokens using both new and old apis. Contributed by Siddharth Seth. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1157420 Files : /hadoop/common/trunk/mapreduce/src/test/mapred/org/apache/hadoop/mapreduce/security/TestTokenCache.java /hadoop/common/trunk/mapreduce/CHANGES.txt /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/security/TokenCache.java
        Hide
        Arun C Murthy added a comment -

        I just committed this to both trunk and MR-279. Thanks Sid!

        Show
        Arun C Murthy added a comment - I just committed this to both trunk and MR-279. Thanks Sid!
        Hide
        Siddharth Seth added a comment -

        Patch for the MR-279 branch.

        Show
        Siddharth Seth added a comment - Patch for the MR-279 branch.
        Hide
        Siddharth Seth added a comment -

        Updated patch to apply cleanly against trunk after yesterday's commits.

        Two test failures - both not related to the patch.
        TEST-org.apache.hadoop.cli.TestMRCLI.txt:Tests run: 1, Failures: 1, Errors: 0, Time elapsed: 28.994 sec
        TEST-org.apache.hadoop.mapred.TestCombineOutputCollector.txt:Tests run: 2, Failures: 2, Errors: 0, Time elapsed: 1.605 sec

        1 new FindBug warning while running test-patch - not related.

        Haven't been able to get ant binary-system (FI build) to work. Seems to pick up an old version of common without NetUtils.verifyHostnames

        Will post a patch for the MR-279 branch shortly.

        Show
        Siddharth Seth added a comment - Updated patch to apply cleanly against trunk after yesterday's commits. Two test failures - both not related to the patch. TEST-org.apache.hadoop.cli.TestMRCLI.txt:Tests run: 1, Failures: 1, Errors: 0, Time elapsed: 28.994 sec TEST-org.apache.hadoop.mapred.TestCombineOutputCollector.txt:Tests run: 2, Failures: 2, Errors: 0, Time elapsed: 1.605 sec 1 new FindBug warning while running test-patch - not related. Haven't been able to get ant binary-system (FI build) to work. Seems to pick up an old version of common without NetUtils.verifyHostnames Will post a patch for the MR-279 branch shortly.
        Hide
        Siddharth Seth added a comment -

        the call to getDelegationToken is retained for now, for hftp to work. This will however end up getting the delegation token twice for other filesystem which implement both APIs.

        Show
        Siddharth Seth added a comment - the call to getDelegationToken is retained for now, for hftp to work. This will however end up getting the delegation token twice for other filesystem which implement both APIs.

          People

          • Assignee:
            Siddharth Seth
            Reporter:
            Siddharth Seth
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development