Hadoop Common
  1. Hadoop Common
  2. HADOOP-7963

test failures: TestViewFileSystemWithAuthorityLocalFileSystem and TestViewFileSystemLocalFileSystem

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.23.1
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Fix ViewFS to catch a null canonical service-name and pass tests TestViewFileSystem*

      Description

      The following tests are failing:

      org.apache.hadoop.fs.viewfs.TestViewFileSystemLocalFileSystem.testGetDelegationTokensWithCredentials
      org.apache.hadoop.fs.viewfs.TestViewFileSystemWithAuthorityLocalFileSystem.testGetDelegationTokensWithCredentials

      See hudson:
      https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-Common-trunk/lastCompletedBuild/testReport/

      1. HADOOP7963.txt
        0.8 kB
        Siddharth Seth

        Activity

        Thomas Graves created issue -
        Hide
        Siddharth Seth added a comment -

        Simple patch to fix this.

        HADOOP-7808 changed the FileSystem.getCanonicalServiceName() API (and internally SecurityUtil.buildDTServiceName to return null for FileSystems which do not use delegation tokens.
        That along with viewfs changes in HADOOP-7933 breaks ViewFs.
        The change to return null would break TokenCache in MR as well - creating a separate jira for that.

        Also, SecurityUtil.buildDTServiceName seems to be used elsewhere.
        @Daryn, is the change to return null necessary - and are the other callers taking care of the changed behaviour.

        Show
        Siddharth Seth added a comment - Simple patch to fix this. HADOOP-7808 changed the FileSystem.getCanonicalServiceName() API (and internally SecurityUtil.buildDTServiceName to return null for FileSystems which do not use delegation tokens. That along with viewfs changes in HADOOP-7933 breaks ViewFs. The change to return null would break TokenCache in MR as well - creating a separate jira for that. Also, SecurityUtil.buildDTServiceName seems to be used elsewhere. @Daryn, is the change to return null necessary - and are the other callers taking care of the changed behaviour.
        Siddharth Seth made changes -
        Field Original Value New Value
        Attachment HADOOP7963.txt [ 12509765 ]
        Siddharth Seth made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12509765/HADOOP7963.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        -1 javadoc. The javadoc tool appears to have generated 7 warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in .

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/500//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/500//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12509765/HADOOP7963.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javadoc. The javadoc tool appears to have generated 7 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/500//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/500//console This message is automatically generated.
        Hide
        Daryn Sharp added a comment -

        The SecurityUtils change was a port from 205. My memory is fuzzy, but I seem to recall it as related to a har issue in 205. As you discovered, there was a dependent TokenCache change which hasn't been ported yet. I recall it was a few lines if you want to bring it in.

        (As an aside, in the near future I plan to eliminate the cross-dependency in FileSystem and TokenCache which is the only reason getCanonicalServiceName name even exists. I just haven't had time to test the patch.)

        Show
        Daryn Sharp added a comment - The SecurityUtils change was a port from 205. My memory is fuzzy, but I seem to recall it as related to a har issue in 205. As you discovered, there was a dependent TokenCache change which hasn't been ported yet. I recall it was a few lines if you want to bring it in. (As an aside, in the near future I plan to eliminate the cross-dependency in FileSystem and TokenCache which is the only reason getCanonicalServiceName name even exists. I just haven't had time to test the patch.)
        Hide
        Daryn Sharp added a comment -

        I forgot to add my +1.

        Show
        Daryn Sharp added a comment - I forgot to add my +1.
        Vinod Kumar Vavilapalli made changes -
        Assignee Siddharth Seth [ sseth ]
        Hide
        Vinod Kumar Vavilapalli added a comment -

        +1 from my side too. Pushing this in.

        Show
        Vinod Kumar Vavilapalli added a comment - +1 from my side too. Pushing this in.
        Hide
        Vinod Kumar Vavilapalli added a comment -

        I just committed this to trunk and branch-0.23. Thanks Sid!

        Show
        Vinod Kumar Vavilapalli added a comment - I just committed this to trunk and branch-0.23. Thanks Sid!
        Vinod Kumar Vavilapalli made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags Reviewed [ 10343 ]
        Release Note Fix ViewFS to catch a null canonical service-name and pass tests TestViewFileSystem*
        Resolution Fixed [ 1 ]
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Commit #344 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/344/)
        HADOOP-7963. Fix ViewFS to catch a null canonical service-name and pass tests TestViewFileSystem* (Siddharth Seth via vinodkv)
        svn merge --ignore-ancestry -c 1229379 ../../trunk/

        vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1229382
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Commit #344 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/344/ ) HADOOP-7963 . Fix ViewFS to catch a null canonical service-name and pass tests TestViewFileSystem* (Siddharth Seth via vinodkv) svn merge --ignore-ancestry -c 1229379 ../../trunk/ vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1229382 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #1514 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1514/)
        HADOOP-7963. Fix ViewFS to catch a null canonical service-name and pass tests TestViewFileSystem* (Siddharth Seth via vinodkv)

        vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1229379
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1514 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1514/ ) HADOOP-7963 . Fix ViewFS to catch a null canonical service-name and pass tests TestViewFileSystem* (Siddharth Seth via vinodkv) vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1229379 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #1587 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1587/)
        HADOOP-7963. Fix ViewFS to catch a null canonical service-name and pass tests TestViewFileSystem* (Siddharth Seth via vinodkv)

        vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1229379
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1587 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1587/ ) HADOOP-7963 . Fix ViewFS to catch a null canonical service-name and pass tests TestViewFileSystem* (Siddharth Seth via vinodkv) vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1229379 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-0.23-Commit #354 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/354/)
        HADOOP-7963. Fix ViewFS to catch a null canonical service-name and pass tests TestViewFileSystem* (Siddharth Seth via vinodkv)
        svn merge --ignore-ancestry -c 1229379 ../../trunk/

        vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1229382
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-0.23-Commit #354 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/354/ ) HADOOP-7963 . Fix ViewFS to catch a null canonical service-name and pass tests TestViewFileSystem* (Siddharth Seth via vinodkv) svn merge --ignore-ancestry -c 1229379 ../../trunk/ vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1229382 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-0.23-Commit #365 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/365/)
        HADOOP-7963. Fix ViewFS to catch a null canonical service-name and pass tests TestViewFileSystem* (Siddharth Seth via vinodkv)
        svn merge --ignore-ancestry -c 1229379 ../../trunk/

        vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1229382
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Commit #365 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/365/ ) HADOOP-7963 . Fix ViewFS to catch a null canonical service-name and pass tests TestViewFileSystem* (Siddharth Seth via vinodkv) svn merge --ignore-ancestry -c 1229379 ../../trunk/ vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1229382 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #1534 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1534/)
        HADOOP-7963. Fix ViewFS to catch a null canonical service-name and pass tests TestViewFileSystem* (Siddharth Seth via vinodkv)

        vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1229379
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1534 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1534/ ) HADOOP-7963 . Fix ViewFS to catch a null canonical service-name and pass tests TestViewFileSystem* (Siddharth Seth via vinodkv) vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1229379 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #921 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/921/)
        HADOOP-7963. Fix ViewFS to catch a null canonical service-name and pass tests TestViewFileSystem* (Siddharth Seth via vinodkv)

        vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1229379
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #921 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/921/ ) HADOOP-7963 . Fix ViewFS to catch a null canonical service-name and pass tests TestViewFileSystem* (Siddharth Seth via vinodkv) vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1229379 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Build #134 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/134/)
        HADOOP-7963. Fix ViewFS to catch a null canonical service-name and pass tests TestViewFileSystem* (Siddharth Seth via vinodkv)
        svn merge --ignore-ancestry -c 1229379 ../../trunk/

        vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1229382
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #134 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/134/ ) HADOOP-7963 . Fix ViewFS to catch a null canonical service-name and pass tests TestViewFileSystem* (Siddharth Seth via vinodkv) svn merge --ignore-ancestry -c 1229379 ../../trunk/ vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1229382 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-0.23-Build #156 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/156/)
        HADOOP-7963. Fix ViewFS to catch a null canonical service-name and pass tests TestViewFileSystem* (Siddharth Seth via vinodkv)
        svn merge --ignore-ancestry -c 1229379 ../../trunk/

        vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1229382
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Build #156 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/156/ ) HADOOP-7963 . Fix ViewFS to catch a null canonical service-name and pass tests TestViewFileSystem* (Siddharth Seth via vinodkv) svn merge --ignore-ancestry -c 1229379 ../../trunk/ vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1229382 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #954 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/954/)
        HADOOP-7963. Fix ViewFS to catch a null canonical service-name and pass tests TestViewFileSystem* (Siddharth Seth via vinodkv)

        vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1229379
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #954 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/954/ ) HADOOP-7963 . Fix ViewFS to catch a null canonical service-name and pass tests TestViewFileSystem* (Siddharth Seth via vinodkv) vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1229379 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
        Arun C Murthy made changes -
        Fix Version/s 0.23.1 [ 12318884 ]
        Fix Version/s 0.23.0 [ 12315569 ]
        Target Version/s 0.23.0 [ 12315569 ] 1.0.0 [ 12318244 ]
        Arun C Murthy made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Patch Available Patch Available
        6h 28m 1 Siddharth Seth 07/Jan/12 04:27
        Patch Available Patch Available Resolved Resolved
        2d 17h 28m 1 Vinod Kumar Vavilapalli 09/Jan/12 21:56
        Resolved Resolved Closed Closed
        55d 4h 53m 1 Arun C Murthy 05/Mar/12 02:49

          People

          • Assignee:
            Siddharth Seth
            Reporter:
            Thomas Graves
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development