Hadoop Common
  1. Hadoop Common
  2. HADOOP-8129

ViewFileSystemTestSetup setupForViewFileSystem is erring when the user's home directory is somewhere other than /home (eg. /User) etc.

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.3, 2.0.2-alpha
    • Component/s: fs, test
    • Labels:

      Description

      All TestFSMainOperationsLocalFileSystem tests (99 in all) fail saying:

      java.io.FileNotFoundException: /home
      	at org.apache.hadoop.fs.viewfs.InodeTree.resolve(InodeTree.java:403)
      	at org.apache.hadoop.fs.viewfs.ViewFileSystem.mkdirs(ViewFileSystem.java:373)
      	at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1684)
      	at org.apache.hadoop.fs.FSMainOperationsBaseTest.setUp(FSMainOperationsBaseTest.java:90)
      	at org.apache.hadoop.fs.viewfs.TestFSMainOperationsLocalFileSystem.setUp(TestFSMainOperationsLocalFileSystem.java:42)
      
      1. HADOOP-8129.patch
        1 kB
        Ahmed Radwan
      2. HADOOP-8129.patch
        1 kB
        Ravi Prakash

        Activity

        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1122 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1122/)
        HADOOP-8129. ViewFileSystemTestSetup setupForViewFileSystem is erring (Ahmed Radwan and Ravi Prakash via bobby) (Revision 1354093)

        Result = FAILURE
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1354093
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFileSystemTestSetup.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1122 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1122/ ) HADOOP-8129 . ViewFileSystemTestSetup setupForViewFileSystem is erring (Ahmed Radwan and Ravi Prakash via bobby) (Revision 1354093) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1354093 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFileSystemTestSetup.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Build #299 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/299/)
        svn merge -c 1354093. FIXES: HADOOP-8129. ViewFileSystemTestSetup setupForViewFileSystem is erring (Ahmed Radwan and Ravi Prakash via bobby) (Revision 1354099)

        Result = UNSTABLE
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1354099
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFileSystemTestSetup.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #299 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/299/ ) svn merge -c 1354093. FIXES: HADOOP-8129 . ViewFileSystemTestSetup setupForViewFileSystem is erring (Ahmed Radwan and Ravi Prakash via bobby) (Revision 1354099) Result = UNSTABLE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1354099 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFileSystemTestSetup.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #1089 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1089/)
        HADOOP-8129. ViewFileSystemTestSetup setupForViewFileSystem is erring (Ahmed Radwan and Ravi Prakash via bobby) (Revision 1354093)

        Result = FAILURE
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1354093
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFileSystemTestSetup.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1089 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1089/ ) HADOOP-8129 . ViewFileSystemTestSetup setupForViewFileSystem is erring (Ahmed Radwan and Ravi Prakash via bobby) (Revision 1354093) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1354093 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFileSystemTestSetup.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #2408 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2408/)
        HADOOP-8129. ViewFileSystemTestSetup setupForViewFileSystem is erring (Ahmed Radwan and Ravi Prakash via bobby) (Revision 1354093)

        Result = FAILURE
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1354093
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFileSystemTestSetup.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2408 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2408/ ) HADOOP-8129 . ViewFileSystemTestSetup setupForViewFileSystem is erring (Ahmed Radwan and Ravi Prakash via bobby) (Revision 1354093) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1354093 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFileSystemTestSetup.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #2389 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2389/)
        HADOOP-8129. ViewFileSystemTestSetup setupForViewFileSystem is erring (Ahmed Radwan and Ravi Prakash via bobby) (Revision 1354093)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1354093
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFileSystemTestSetup.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2389 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2389/ ) HADOOP-8129 . ViewFileSystemTestSetup setupForViewFileSystem is erring (Ahmed Radwan and Ravi Prakash via bobby) (Revision 1354093) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1354093 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFileSystemTestSetup.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #2458 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2458/)
        HADOOP-8129. ViewFileSystemTestSetup setupForViewFileSystem is erring (Ahmed Radwan and Ravi Prakash via bobby) (Revision 1354093)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1354093
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFileSystemTestSetup.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2458 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2458/ ) HADOOP-8129 . ViewFileSystemTestSetup setupForViewFileSystem is erring (Ahmed Radwan and Ravi Prakash via bobby) (Revision 1354093) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1354093 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFileSystemTestSetup.java
        Hide
        Robert Joseph Evans added a comment -

        Thanks Ahmed for driving this over the finish line. I put this into trunk, branch-2, and branch-0.23

        Show
        Robert Joseph Evans added a comment - Thanks Ahmed for driving this over the finish line. I put this into trunk, branch-2, and branch-0.23
        Hide
        Robert Joseph Evans added a comment -

        The change looks good to me too +1, so with 3 +1s I will be checking this in.

        Show
        Robert Joseph Evans added a comment - The change looks good to me too +1, so with 3 +1s I will be checking this in.
        Hide
        Tom White added a comment -

        Thanks for the explanation, Ahmed. I tried running TestFSMainOperationsLocalFileSystem and TestViewFsTrash from outside my home directory and they failed without the patch, and passed with it. Both still passed when run from within my home directory.

        +1

        Show
        Tom White added a comment - Thanks for the explanation, Ahmed. I tried running TestFSMainOperationsLocalFileSystem and TestViewFsTrash from outside my home directory and they failed without the patch, and passed with it. Both still passed when run from within my home directory. +1
        Hide
        Eli Collins added a comment -

        +1 Looks good to me, I think HADOOP-8036 erroneously removed this code (establishing a link to the first component of the test dir).

        Show
        Eli Collins added a comment - +1 Looks good to me, I think HADOOP-8036 erroneously removed this code (establishing a link to the first component of the test dir).
        Hide
        Ahmed Radwan added a comment -

        Yes, I brought back some of the changes that HADOOP-8036 deleted. Basically to be able to run the tests outside the home directory, we need to add back the link to the test root directory and not just the home directory. So HADOOP-8036 needed to add an additional link and not to replace the one I brought back.

        Regarding Daryn's comment on HADOOP-8110, this is part of the logic in the test that should fail and the test is confirming that it'll fail.

        Show
        Ahmed Radwan added a comment - Yes, I brought back some of the changes that HADOOP-8036 deleted. Basically to be able to run the tests outside the home directory, we need to add back the link to the test root directory and not just the home directory. So HADOOP-8036 needed to add an additional link and not to replace the one I brought back. Regarding Daryn's comment on HADOOP-8110 , this is part of the logic in the test that should fail and the test is confirming that it'll fail.
        Hide
        Tom White added a comment -

        It looks like this patch is basically reverting some of the changes HADOOP-8036 made, e.g. to ViewFileSystemTestSetup. Is that right? Perhaps Colin and Eli could comment.

        Also, see HADOOP-8110, particularly Daryn's comment about how TestViewFsTrash tries to delete the user's home directory. It would be good if this patch fixed that too.

        Show
        Tom White added a comment - It looks like this patch is basically reverting some of the changes HADOOP-8036 made, e.g. to ViewFileSystemTestSetup . Is that right? Perhaps Colin and Eli could comment. Also, see HADOOP-8110 , particularly Daryn's comment about how TestViewFsTrash tries to delete the user's home directory. It would be good if this patch fixed that too.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12533177/HADOOP-8129.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 javadoc. The javadoc tool appears to have generated 13 warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1137//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1137//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12533177/HADOOP-8129.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 javadoc. The javadoc tool appears to have generated 13 warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1137//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1137//console This message is automatically generated.
        Hide
        Ahmed Radwan added a comment -

        The issue can be reproduced by trying to run TestFSMainOperationsLocalFileSystem or TestViewFSTrash from any location not under the system home directory.

        Here is a patch. I have confirmed that both tests successfully run from locations outside or under the system home directory.

        Show
        Ahmed Radwan added a comment - The issue can be reproduced by trying to run TestFSMainOperationsLocalFileSystem or TestViewFSTrash from any location not under the system home directory. Here is a patch. I have confirmed that both tests successfully run from locations outside or under the system home directory.
        Hide
        Ravi Prakash added a comment -

        Hi Ahmed! Unfortunately this slipped in my priorities, and probably will remain so until we find the conviction to go fix all the issues in the implementation of Trash + ViewFS. I'll unassign myself from this.

        I don't know if ignoring the tests which fail (signalling a legitimate problem) is a good long term solution, but if you want to make it so, to fix your builds, I'll not stand in the way.

        Show
        Ravi Prakash added a comment - Hi Ahmed! Unfortunately this slipped in my priorities, and probably will remain so until we find the conviction to go fix all the issues in the implementation of Trash + ViewFS. I'll unassign myself from this. I don't know if ignoring the tests which fail (signalling a legitimate problem) is a good long term solution, but if you want to make it so, to fix your builds, I'll not stand in the way.
        Hide
        Ahmed Radwan added a comment -

        Any updates on this issue?
        I guess we can just automatically ignore this test when detecting that it is running from outside /home.

        Show
        Ahmed Radwan added a comment - Any updates on this issue? I guess we can just automatically ignore this test when detecting that it is running from outside /home.
        Hide
        Ravi Prakash added a comment -

        There's a very illuminating discussion on https://issues.apache.org/jira/browse/HADOOP-7284. I'm not sure what the right way going forward is. I'm suspending work on this JIRA for now. If someone wants to take over, please feel free

        Show
        Ravi Prakash added a comment - There's a very illuminating discussion on https://issues.apache.org/jira/browse/HADOOP-7284 . I'm not sure what the right way going forward is. I'm suspending work on this JIRA for now. If someone wants to take over, please feel free
        Hide
        Arun C Murthy added a comment -

        Cancelling patch while we brainstorm the fix to the tests.

        Show
        Arun C Murthy added a comment - Cancelling patch while we brainstorm the fix to the tests.
        Hide
        Ravi Prakash added a comment -

        This is not good. TestViewFSTrash is writing into ~/.Trash . It has no business writing in there. Can we all agree? Seems like LocalFileSystem will need to be changed. Thoughts anybody?

        Show
        Ravi Prakash added a comment - This is not good. TestViewFSTrash is writing into ~/.Trash . It has no business writing in there. Can we all agree? Seems like LocalFileSystem will need to be changed. Thoughts anybody?
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12516887/HADOOP-8129.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.ipc.TestRPCCallBenchmark
        org.apache.hadoop.fs.viewfs.TestViewFsTrash

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/663//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/663//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12516887/HADOOP-8129.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.ipc.TestRPCCallBenchmark org.apache.hadoop.fs.viewfs.TestViewFsTrash +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/663//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/663//console This message is automatically generated.
        Hide
        Ravi Prakash added a comment -

        Simple and small patch. Can someone please review and commit

        Show
        Ravi Prakash added a comment - Simple and small patch. Can someone please review and commit

          People

          • Assignee:
            Ahmed Radwan
            Reporter:
            Ravi Prakash
          • Votes:
            0 Vote for this issue
            Watchers:
            11 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development