Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-6665

Add tests for XAttrs in combination with viewfs

    Details

    • Type: Test Test
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.5.0
    • Fix Version/s: 2.6.0
    • Component/s: hdfs-client
    • Labels:
      None

      Description

      This is similar to HDFS-5624 (Add tests for ACLs in combination with viewfs)

      We should verify that XAttr operations work properly with viewfs, and that XAttr commands are routed to the correct namenode in a federated deployment.

      Also, we should make sure that the behavior of XAttr commands on internal dirs is consistent with other commands. For example, setPermission will throw the readonly AccessControlException for paths above the root mount entry.

      1. HDFS-6665.2.patch
        13 kB
        Stephen Chu
      2. HDFS-6665.1.patch
        13 kB
        Stephen Chu

        Issue Links

          Activity

          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Patch Available Patch Available
          14d 41m 1 Stephen Chu 24/Jul/14 22:10
          Patch Available Patch Available Resolved Resolved
          5d 8h 43m 1 Andrew Wang 30/Jul/14 06:54
          Resolved Resolved Closed Closed
          123d 21h 16m 1 Arun C Murthy 01/Dec/14 03:10
          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1847 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1847/)
          HDFS-6665. Add tests for XAttrs in combination with viewfs. Contributed by Stephen Chu. (wang: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1614545)

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFileSystemWithXAttrs.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFsWithXAttrs.java
          Show
          Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1847 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1847/ ) HDFS-6665 . Add tests for XAttrs in combination with viewfs. Contributed by Stephen Chu. (wang: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1614545 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFileSystemWithXAttrs.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFsWithXAttrs.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #1821 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1821/)
          HDFS-6665. Add tests for XAttrs in combination with viewfs. Contributed by Stephen Chu. (wang: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1614545)

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFileSystemWithXAttrs.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFsWithXAttrs.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #1821 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1821/ ) HDFS-6665 . Add tests for XAttrs in combination with viewfs. Contributed by Stephen Chu. (wang: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1614545 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFileSystemWithXAttrs.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFsWithXAttrs.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #628 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/628/)
          HDFS-6665. Add tests for XAttrs in combination with viewfs. Contributed by Stephen Chu. (wang: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1614545)

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFileSystemWithXAttrs.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFsWithXAttrs.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #628 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/628/ ) HDFS-6665 . Add tests for XAttrs in combination with viewfs. Contributed by Stephen Chu. (wang: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1614545 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFileSystemWithXAttrs.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFsWithXAttrs.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #5984 (See https://builds.apache.org/job/Hadoop-trunk-Commit/5984/)
          HDFS-6665. Add tests for XAttrs in combination with viewfs. Contributed by Stephen Chu. (wang: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1614545)

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFileSystemWithXAttrs.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFsWithXAttrs.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #5984 (See https://builds.apache.org/job/Hadoop-trunk-Commit/5984/ ) HDFS-6665 . Add tests for XAttrs in combination with viewfs. Contributed by Stephen Chu. (wang: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1614545 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFileSystemWithXAttrs.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFsWithXAttrs.java
          Hide
          Stephen Chu added a comment -

          Thank you, Andrew!

          Show
          Stephen Chu added a comment - Thank you, Andrew!
          Andrew Wang made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Fix Version/s 2.6.0 [ 12327181 ]
          Resolution Fixed [ 1 ]
          Hide
          Andrew Wang added a comment -

          Committed to trunk and branch-2, thanks Stephen!

          Show
          Andrew Wang added a comment - Committed to trunk and branch-2, thanks Stephen!
          Hide
          Stephen Chu added a comment -

          Ah, yes, seems to be the same too many open files issue talked about in that JIRA. Thanks for pointing me to it, Vinay.

          Show
          Stephen Chu added a comment - Ah, yes, seems to be the same too many open files issue talked about in that JIRA. Thanks for pointing me to it, Vinay.
          Hide
          Vinayakumar B added a comment -

          TestPipelinesFailover failure might be related to HDFS-6694

          Show
          Vinayakumar B added a comment - TestPipelinesFailover failure might be related to HDFS-6694
          Hide
          Stephen Chu added a comment -

          TestPipelinesFailover failure is not due to the patch changes. I re-ran the test locally successfully a couple times to be sure. Other than that, the Hadoop QA job looks good.

          Show
          Stephen Chu added a comment - TestPipelinesFailover failure is not due to the patch changes. I re-ran the test locally successfully a couple times to be sure. Other than that, the Hadoop QA job looks good.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12657727/HDFS-6665.2.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/7459//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7459//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12657727/HDFS-6665.2.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/7459//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7459//console This message is automatically generated.
          Stephen Chu made changes -
          Attachment HDFS-6665.2.patch [ 12657727 ]
          Hide
          Stephen Chu added a comment -

          Thanks for the review and catching that, Andrew Wang!

          Uploading a new patch to fix the comment.

          Looked into test results of the TestBlockTokenWithDFS and TestNamenodeCapacityReport, and they're not related to these WebHDFS test changes. Re-ran them locally successfully.

          Show
          Stephen Chu added a comment - Thanks for the review and catching that, Andrew Wang ! Uploading a new patch to fix the comment. Looked into test results of the TestBlockTokenWithDFS and TestNamenodeCapacityReport, and they're not related to these WebHDFS test changes. Re-ran them locally successfully.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12657676/HDFS-6665.1.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS
          org.apache.hadoop.hdfs.server.namenode.TestNamenodeCapacityReport

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/7457//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7457//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12657676/HDFS-6665.1.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS org.apache.hadoop.hdfs.server.namenode.TestNamenodeCapacityReport +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/7457//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7457//console This message is automatically generated.
          Hide
          Andrew Wang added a comment -

          Hi Stephen, thanks for the patch, just nitty comments again:

          • "Remove the ACL entries on the first namespace" probably meant xattr, in both files

          +1 pending that though, nice work.

          Show
          Andrew Wang added a comment - Hi Stephen, thanks for the patch, just nitty comments again: "Remove the ACL entries on the first namespace" probably meant xattr, in both files +1 pending that though, nice work.
          Stephen Chu made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Target Version/s 2.6.0 [ 12327181 ] 3.0.0, 2.6.0 [ 12320356, 12327181 ]
          Stephen Chu made changes -
          Attachment HDFS-6665.1.patch [ 12657676 ]
          Hide
          Stephen Chu added a comment -

          Attaching patch to add two tests that verify XAttrs with ViewFs and ViewFileSystem. They verify that the XAttr operations are routed to the correct NameNode.

          Show
          Stephen Chu added a comment - Attaching patch to add two tests that verify XAttrs with ViewFs and ViewFileSystem. They verify that the XAttr operations are routed to the correct NameNode.
          Hide
          Stephen Chu added a comment -

          I've submitted a patch for HADOOP-10887, similar to what we did for ACLs in HADOOP-10845.

          Once that's resolved, we can add HDFS tests for XAttrs + ViewFileSystem and ViewFs.

          Show
          Stephen Chu added a comment - I've submitted a patch for HADOOP-10887 , similar to what we did for ACLs in HADOOP-10845 . Once that's resolved, we can add HDFS tests for XAttrs + ViewFileSystem and ViewFs.
          Stephen Chu made changes -
          Field Original Value New Value
          Link This issue requires HADOOP-10887 [ HADOOP-10887 ]
          Stephen Chu created issue -

            People

            • Assignee:
              Stephen Chu
              Reporter:
              Stephen Chu
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development