Hadoop Common
  1. Hadoop Common
  2. HADOOP-8408

MR doesn't work with a non-default ViewFS mount table and security enabled

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.0-alpha
    • Fix Version/s: 2.0.2-alpha
    • Component/s: viewfs
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      With security enabled, if one sets up a ViewFS mount table using the default mount table name, everything works as expected. However, if you try to create a ViewFS mount table with a non-default name, you'll end up getting an error like the following (in this case "vfs-cluster" was the name of the mount table) when running an MR job:

      java.lang.IllegalArgumentException: java.net.UnknownHostException: vfs-cluster
      
      1. HDFS-8408.patch
        4 kB
        Aaron T. Myers
      2. HADOOP-8408-amendment.patch
        1 kB
        Aaron T. Myers
      3. HADOOP-8408-amendment.patch
        2 kB
        Aaron T. Myers

        Activity

        Hide
        Aaron T. Myers added a comment -

        Here's a patch which addresses the issue. The trouble is that ViewFileSystem didn't override FileSystem#getCanonicalServiceName, and thus used the default which assumes that the host name of the URI can be turned into an IP address. Without this patch, running an MR job on a view filesystem with a non-default mount table name yields the following exception:

        java.lang.IllegalArgumentException: java.net.UnknownHostException: vfs-cluster
        	at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:400)
        	at org.apache.hadoop.security.SecurityUtil.buildDTServiceName(SecurityUtil.java:281)
        	at org.apache.hadoop.fs.FileSystem.getCanonicalServiceName(FileSystem.java:235)
        	at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:124)
        	at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:101)
        	at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:81)
        	at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:137)
        	at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:410)
        	at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:325)
        	at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1226)
        	at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1)
        	at java.security.AccessController.doPrivileged(Native Method)
        	at javax.security.auth.Subject.doAs(Subject.java:396)
        	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
        	at org.apache.hadoop.mapreduce.Job.submit(Job.java:1223)
        	at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1244)
        	at org.apache.hadoop.examples.QuasiMonteCarlo.estimatePi(QuasiMonteCarlo.java:306)
        	at org.apache.hadoop.examples.QuasiMonteCarlo.run(QuasiMonteCarlo.java:351)
        	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        	at org.apache.hadoop.examples.QuasiMonteCarlo.main(QuasiMonteCarlo.java:360)
        	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        	at java.lang.reflect.Method.invoke(Method.java:597)
        	at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
        	at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
        	at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:68)
        	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        	at java.lang.reflect.Method.invoke(Method.java:597)
        	at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
        Caused by: java.net.UnknownHostException: vfs-cluster
        	at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:401)
        	... 31 more
        

        In addition to the automated tests in this patch, I also verified manually that I can successfully run MR jobs on a secure cluster with a non-default mount table name.

        Show
        Aaron T. Myers added a comment - Here's a patch which addresses the issue. The trouble is that ViewFileSystem didn't override FileSystem#getCanonicalServiceName, and thus used the default which assumes that the host name of the URI can be turned into an IP address. Without this patch, running an MR job on a view filesystem with a non-default mount table name yields the following exception: java.lang.IllegalArgumentException: java.net.UnknownHostException: vfs-cluster at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:400) at org.apache.hadoop.security.SecurityUtil.buildDTServiceName(SecurityUtil.java:281) at org.apache.hadoop.fs.FileSystem.getCanonicalServiceName(FileSystem.java:235) at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:124) at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:101) at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:81) at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:137) at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:410) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:325) at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1226) at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1223) at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1244) at org.apache.hadoop.examples.QuasiMonteCarlo.estimatePi(QuasiMonteCarlo.java:306) at org.apache.hadoop.examples.QuasiMonteCarlo.run(QuasiMonteCarlo.java:351) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.examples.QuasiMonteCarlo.main(QuasiMonteCarlo.java:360) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72) at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144) at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:68) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:208) Caused by: java.net.UnknownHostException: vfs-cluster at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:401) ... 31 more In addition to the automated tests in this patch, I also verified manually that I can successfully run MR jobs on a secure cluster with a non-default mount table name.
        Hide
        Aaron T. Myers added a comment -

        I should've mentioned: this is basically the ViewFS analog of what HDFS-3062 was for HA.

        Show
        Aaron T. Myers added a comment - I should've mentioned: this is basically the ViewFS analog of what HDFS-3062 was for HA.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12527975/HDFS-8408.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 javadoc. The javadoc tool appears to have generated 2 warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1005//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1005//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12527975/HDFS-8408.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 javadoc. The javadoc tool appears to have generated 2 warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1005//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1005//console This message is automatically generated.
        Hide
        Aaron T. Myers added a comment -

        The javadoc warnings are unrelated.

        Show
        Aaron T. Myers added a comment - The javadoc warnings are unrelated.
        Hide
        Eli Collins added a comment -

        +1 looks great

        Show
        Eli Collins added a comment - +1 looks great
        Hide
        Eli Collins added a comment -

        I've committed this to trunk and merged to branch-2. Thanks ATM.

        Show
        Eli Collins added a comment - I've committed this to trunk and merged to branch-2. Thanks ATM.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #2340 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2340/)
        HADOOP-8408. MR doesn't work with a non-default ViewFS mount table and security enabled. Contributed by Aaron T. Myers (Revision 1339970)

        Result = SUCCESS
        eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1339970
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFileSystemDelegationTokenSupport.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2340 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2340/ ) HADOOP-8408 . MR doesn't work with a non-default ViewFS mount table and security enabled. Contributed by Aaron T. Myers (Revision 1339970) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1339970 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFileSystemDelegationTokenSupport.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #2267 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2267/)
        HADOOP-8408. MR doesn't work with a non-default ViewFS mount table and security enabled. Contributed by Aaron T. Myers (Revision 1339970)

        Result = SUCCESS
        eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1339970
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFileSystemDelegationTokenSupport.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2267 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2267/ ) HADOOP-8408 . MR doesn't work with a non-default ViewFS mount table and security enabled. Contributed by Aaron T. Myers (Revision 1339970) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1339970 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFileSystemDelegationTokenSupport.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #2285 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2285/)
        HADOOP-8408. MR doesn't work with a non-default ViewFS mount table and security enabled. Contributed by Aaron T. Myers (Revision 1339970)

        Result = FAILURE
        eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1339970
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFileSystemDelegationTokenSupport.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2285 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2285/ ) HADOOP-8408 . MR doesn't work with a non-default ViewFS mount table and security enabled. Contributed by Aaron T. Myers (Revision 1339970) Result = FAILURE eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1339970 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFileSystemDelegationTokenSupport.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #1049 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1049/)
        HADOOP-8408. MR doesn't work with a non-default ViewFS mount table and security enabled. Contributed by Aaron T. Myers (Revision 1339970)

        Result = SUCCESS
        eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1339970
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFileSystemDelegationTokenSupport.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1049 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1049/ ) HADOOP-8408 . MR doesn't work with a non-default ViewFS mount table and security enabled. Contributed by Aaron T. Myers (Revision 1339970) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1339970 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFileSystemDelegationTokenSupport.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1083 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1083/)
        HADOOP-8408. MR doesn't work with a non-default ViewFS mount table and security enabled. Contributed by Aaron T. Myers (Revision 1339970)

        Result = SUCCESS
        eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1339970
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFileSystemDelegationTokenSupport.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1083 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1083/ ) HADOOP-8408 . MR doesn't work with a non-default ViewFS mount table and security enabled. Contributed by Aaron T. Myers (Revision 1339970) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1339970 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFileSystemDelegationTokenSupport.java
        Hide
        Daryn Sharp added a comment -

        This actually isn't the right fix. ViewFileSystem#getCanonicalServiceName should be returning null since it doesn't have any intrinsic tokens of its own. Returning the raw uri's hostname is really returning the mount table name. It "accidentally works".

        Show
        Daryn Sharp added a comment - This actually isn't the right fix. ViewFileSystem#getCanonicalServiceName should be returning null since it doesn't have any intrinsic tokens of its own. Returning the raw uri's hostname is really returning the mount table name. It "accidentally works".
        Hide
        Aaron T. Myers added a comment -

        Thanks for the input, Daryn. I'll try changing it as you describe, and we can amend this JIRA if it works. Do you think the fix for HDFS-3062 should also be corrected?

        It's not super clear to me what the flow of control is expected to be for file systems which don't have their own support for tokens, but rather delegate to underlying FS implementations. Would you mind explaining it? I see this comment in FileSystem, but it doesn't make it clear how tokens will end up getting cached for a file system which delegates token fetching:

        Get a canonical service name for this file system. The token cache is
        the only user of this value, and uses it to lookup this filesystem's
        service tokens. The token cache will not attempt to acquire tokens if the
        service is null.

        Show
        Aaron T. Myers added a comment - Thanks for the input, Daryn. I'll try changing it as you describe, and we can amend this JIRA if it works. Do you think the fix for HDFS-3062 should also be corrected? It's not super clear to me what the flow of control is expected to be for file systems which don't have their own support for tokens, but rather delegate to underlying FS implementations. Would you mind explaining it? I see this comment in FileSystem, but it doesn't make it clear how tokens will end up getting cached for a file system which delegates token fetching: Get a canonical service name for this file system. The token cache is the only user of this value, and uses it to lookup this filesystem's service tokens. The token cache will not attempt to acquire tokens if the service is null.
        Hide
        Daryn Sharp added a comment -

        Yes, I noticed awhile back that HDFS-3062 was problematic, so it was superseded by my HDFS-3268. The handling of tokens for multi-token filesystem is rather funky right now. HADOOP-7967 and MAPREDUCE-3825 aim to fix the problem.

        The way it's intended to work is a filesystem with no tokens is supposed to return null for its service name. The TokenCache used to skip a fs with a null service. This was problematic for multi-token filesystems like ViewFileSystem, so someone changed the TokenCache to proceed to acquire tokens from a fs with a null service. A better approach is to request tokens from a filesystem: a "normal" fs returns its token if its service is non-null, and a multi-token filesystem gets the tokens for the leaf fs-es that don't have a null service. I hope that makes some sense...

        Show
        Daryn Sharp added a comment - Yes, I noticed awhile back that HDFS-3062 was problematic, so it was superseded by my HDFS-3268 . The handling of tokens for multi-token filesystem is rather funky right now. HADOOP-7967 and MAPREDUCE-3825 aim to fix the problem. The way it's intended to work is a filesystem with no tokens is supposed to return null for its service name. The TokenCache used to skip a fs with a null service. This was problematic for multi-token filesystems like ViewFileSystem , so someone changed the TokenCache to proceed to acquire tokens from a fs with a null service. A better approach is to request tokens from a filesystem: a "normal" fs returns its token if its service is non-null, and a multi-token filesystem gets the tokens for the leaf fs-es that don't have a null service. I hope that makes some sense...
        Hide
        Aaron T. Myers added a comment -

        Re-opening to amend the patch per Daryn's feedback.

        Show
        Aaron T. Myers added a comment - Re-opening to amend the patch per Daryn's feedback.
        Hide
        Aaron T. Myers added a comment -

        Here's an amendment patch which should address Daryn's comments. Posting just this diff seems easier than reverting/posting the whole patch again.

        In addition to modifying the automated test, I also tested this patch manually on a secure cluster with both a default and non-default ViewFS mount table and confirmed that it works as expected.

        Daryn, does this patch look OK to you?

        Show
        Aaron T. Myers added a comment - Here's an amendment patch which should address Daryn's comments. Posting just this diff seems easier than reverting/posting the whole patch again. In addition to modifying the automated test, I also tested this patch manually on a secure cluster with both a default and non-default ViewFS mount table and confirmed that it works as expected. Daryn, does this patch look OK to you?
        Hide
        Aaron T. Myers added a comment -

        Also, yes, that explanation makes sense. Thanks a lot for helping me out with this, and for your valuable feedback.

        Show
        Aaron T. Myers added a comment - Also, yes, that explanation makes sense. Thanks a lot for helping me out with this, and for your valuable feedback.
        Hide
        Daryn Sharp added a comment -

        +1 Perfect!

        Show
        Daryn Sharp added a comment - +1 Perfect!
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12528487/HADOOP-8408-amendment.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common:

        org.apache.hadoop.fs.viewfs.TestViewFsTrash

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1015//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1015//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12528487/HADOOP-8408-amendment.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common: org.apache.hadoop.fs.viewfs.TestViewFsTrash +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1015//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1015//console This message is automatically generated.
        Hide
        Aaron T. Myers added a comment -

        The test failure is unrelated.

        Show
        Aaron T. Myers added a comment - The test failure is unrelated.
        Hide
        Eli Collins added a comment -

        +1 Replace the comment "Ensure that a canonical service name can be determined for ViewFileSystem instances configured with a non-default mount table name" with rationale for why a filesystem with no tokens is supposed to return null for its service name. Since it's just a comment no need for a new patch.

        Thanks Daryn!

        Show
        Eli Collins added a comment - +1 Replace the comment "Ensure that a canonical service name can be determined for ViewFileSystem instances configured with a non-default mount table name" with rationale for why a filesystem with no tokens is supposed to return null for its service name. Since it's just a comment no need for a new patch. Thanks Daryn!
        Hide
        Aaron T. Myers added a comment -

        Thanks a lot for the reviews, Daryn and Eli. Here's an updated patch which includes Eli's feedback. I'm going to commit this momentarily based on Eli's +1.

        Show
        Aaron T. Myers added a comment - Thanks a lot for the reviews, Daryn and Eli. Here's an updated patch which includes Eli's feedback. I'm going to commit this momentarily based on Eli's +1.
        Hide
        Aaron T. Myers added a comment -

        I've just committed the amendment patch to trunk and branch-2.

        Show
        Aaron T. Myers added a comment - I've just committed the amendment patch to trunk and branch-2.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #2352 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2352/)
        Amend HADOOP-8408 per Daryn's post-commit feedback. (Revision 1341338)

        Result = SUCCESS
        atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1341338
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFileSystemDelegationTokenSupport.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2352 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2352/ ) Amend HADOOP-8408 per Daryn's post-commit feedback. (Revision 1341338) Result = SUCCESS atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1341338 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFileSystemDelegationTokenSupport.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #2279 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2279/)
        Amend HADOOP-8408 per Daryn's post-commit feedback. (Revision 1341338)

        Result = SUCCESS
        atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1341338
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFileSystemDelegationTokenSupport.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2279 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2279/ ) Amend HADOOP-8408 per Daryn's post-commit feedback. (Revision 1341338) Result = SUCCESS atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1341338 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFileSystemDelegationTokenSupport.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #2297 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2297/)
        Amend HADOOP-8408 per Daryn's post-commit feedback. (Revision 1341338)

        Result = FAILURE
        atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1341338
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFileSystemDelegationTokenSupport.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2297 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2297/ ) Amend HADOOP-8408 per Daryn's post-commit feedback. (Revision 1341338) Result = FAILURE atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1341338 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFileSystemDelegationTokenSupport.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #1053 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1053/)
        Amend HADOOP-8408 per Daryn's post-commit feedback. (Revision 1341338)

        Result = SUCCESS
        atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1341338
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFileSystemDelegationTokenSupport.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1053 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1053/ ) Amend HADOOP-8408 per Daryn's post-commit feedback. (Revision 1341338) Result = SUCCESS atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1341338 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFileSystemDelegationTokenSupport.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1087 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1087/)
        Amend HADOOP-8408 per Daryn's post-commit feedback. (Revision 1341338)

        Result = SUCCESS
        atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1341338
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFileSystemDelegationTokenSupport.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1087 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1087/ ) Amend HADOOP-8408 per Daryn's post-commit feedback. (Revision 1341338) Result = SUCCESS atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1341338 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFileSystemDelegationTokenSupport.java

          People

          • Assignee:
            Aaron T. Myers
            Reporter:
            Aaron T. Myers
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development