Hadoop Common
  1. Hadoop Common
  2. HADOOP-7257

A client side mount table to give per-application/per-job file system view

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      viewfs - client-side mount table.

      Description

      This jira proposes a client side mount table to allow application-centric (or job-centric) filesystem views.

      1. viewfs1.patch
        105 kB
        Sanjay Radia
      2. viewFs2.patch
        214 kB
        Sanjay Radia
      3. viewFs3.patch
        218 kB
        Sanjay Radia
      4. viewFs4.patch
        225 kB
        Sanjay Radia
      5. viewFs5.patch
        226 kB
        Sanjay Radia
      6. ViewFsJavaDoc.pdf
        140 kB
        Sanjay Radia
      7. ViewFs - javdoc2.pdf
        317 kB
        Sanjay Radia

        Issue Links

          Activity

          Hide
          Sanjay Radia added a comment -

          A client side mount table allows application centric filesystem view. Such a feature is very useful
          in an environment that contains multiple namspaces. These can arise today with multiple independent HDFS clusters. It can also arise when a HDFS cluster has muitiple namenodes/namepsaced as proposed in HDFS-1052.

          Several file systems/naming systems have provided this in the past, especially in an environment of
          remote execution.

          Show
          Sanjay Radia added a comment - A client side mount table allows application centric filesystem view. Such a feature is very useful in an environment that contains multiple namspaces. These can arise today with multiple independent HDFS clusters. It can also arise when a HDFS cluster has muitiple namenodes/namepsaced as proposed in HDFS-1052 . Several file systems/naming systems have provided this in the past, especially in an environment of remote execution. Plan 9: http://portal.acm.org/citation.cfm?id=506413&dl=GUIDE&coll=GUIDE&CFID=82715774&CFTOKEN=20109739 The Per-Process View of Naming and Remote Execution: http://portal.acm.org/citation.cfm?id=613822 The Spring system: http://www2.informatik.hu-berlin.de/~mint/Library/Spring/spring-namingpolicy.pshttp://www2.informatik.hu-berlin.de/~mint/Library/Spring/spring-namingpolicy.ps
          Hide
          Milind Bhandarkar added a comment -

          Sanjay, do you have a proposal / requirements doc for this ?

          Show
          Milind Bhandarkar added a comment - Sanjay, do you have a proposal / requirements doc for this ?
          Hide
          Sanjay Radia added a comment -

          BTW this jira should be in hadoop common, not hdfs.

          See attached patch which provides an implementation of the mount table called ViewFs. See the JavaDoc for ViewFs.

          ViewFs extends AbstractFileSystem which implements the mount table in memory in the client side. It reads the
          mountable from the config. This allows the mount table to be passed to jobs easily.

          I still need to implement ViewFileSystem which extends FileSystem - this will allow one to have mount tables with the old file system.

          The mount table is implemented as in-memory inode tree (see InodeTreeImpl<T> where T is AbstractFileSystem or FileSystem

          • ie both can shared the core impl). The ViewFs javadoc gives some config examples.

          As part of this I have implemented Chrooted file system - see ChRootedFs; it is used to represent the target of a mount link.

          FAQ.
          1) Why didn't you use symbolic links?
          Answer:
          1a) We (Yahoo) need an impl for FileSystem that we plan to use for release 20.x FileSystem does not have sym links.
          1b) Symlinks would require all resolutions to go through a central HDFS making it a central point of failure even when the target
          file systems are up.

          Show
          Sanjay Radia added a comment - BTW this jira should be in hadoop common, not hdfs. See attached patch which provides an implementation of the mount table called ViewFs. See the JavaDoc for ViewFs. ViewFs extends AbstractFileSystem which implements the mount table in memory in the client side. It reads the mountable from the config. This allows the mount table to be passed to jobs easily. I still need to implement ViewFileSystem which extends FileSystem - this will allow one to have mount tables with the old file system. The mount table is implemented as in-memory inode tree (see InodeTreeImpl<T> where T is AbstractFileSystem or FileSystem ie both can shared the core impl). The ViewFs javadoc gives some config examples. As part of this I have implemented Chrooted file system - see ChRootedFs; it is used to represent the target of a mount link. FAQ. 1) Why didn't you use symbolic links? Answer: 1a) We (Yahoo) need an impl for FileSystem that we plan to use for release 20.x FileSystem does not have sym links. 1b) Symlinks would require all resolutions to go through a central HDFS making it a central point of failure even when the target file systems are up.
          Hide
          Sanjay Radia added a comment -

          The patch provides an implementation for viewfs ie. client-side mount tables.
          They are implemented as Hadoop file systems and hence can be transparently used where ever a Hadoop file systems is used.
          (e.g. all the commandline tools work with it).
          The patch has an implementation for both FileSystem and AbstractFileSystem called ViewFileSystem and ViewFs
          I have attach the pdf of the javadoc for viewfs which explains how to use this feature.
          I will later post a user guide document.

          The patch contains a large number of additional tests and improves some of the existing file system tests.
          It also fixes a bug in AbstractFileSystem in how it deals with URIs.

          Show
          Sanjay Radia added a comment - The patch provides an implementation for viewfs ie. client-side mount tables. They are implemented as Hadoop file systems and hence can be transparently used where ever a Hadoop file systems is used. (e.g. all the commandline tools work with it). The patch has an implementation for both FileSystem and AbstractFileSystem called ViewFileSystem and ViewFs I have attach the pdf of the javadoc for viewfs which explains how to use this feature. I will later post a user guide document. The patch contains a large number of additional tests and improves some of the existing file system tests. It also fixes a bug in AbstractFileSystem in how it deals with URIs.
          Hide
          Konstantin Boudnik added a comment -

          Quickly skimming over the patch I have noticed that assertions in the tests do not have any messages which will result in Assert(null) error messages if tests fail. This makes failure analyze harder. Please have them added.

          Show
          Konstantin Boudnik added a comment - Quickly skimming over the patch I have noticed that assertions in the tests do not have any messages which will result in Assert(null) error messages if tests fail. This makes failure analyze harder. Please have them added.
          Hide
          Suresh Srinivas added a comment -

          > result in Assert(null) error messages if tests fail
          I am not sure if this makes analysis any harder. One needs to look at the stack trace, go to the place where the exception happened, understand what is going on and fix the issue.

          If this was an exception seen by the user and not a developer, adding text makes sense. Given that this is in unit tests, I am not sure about the value of this.

          Show
          Suresh Srinivas added a comment - > result in Assert(null) error messages if tests fail I am not sure if this makes analysis any harder. One needs to look at the stack trace, go to the place where the exception happened, understand what is going on and fix the issue. If this was an exception seen by the user and not a developer, adding text makes sense. Given that this is in unit tests, I am not sure about the value of this.
          Hide
          Konstantin Boudnik added a comment -

          The value of this is that every time out "unit" tests are failing in the Hudson ppl need to go and do exactly as you said - look into the code, dig the context, etc. Messages are easy to add and facilitate test failure analysis.

          Show
          Konstantin Boudnik added a comment - The value of this is that every time out "unit" tests are failing in the Hudson ppl need to go and do exactly as you said - look into the code, dig the context, etc. Messages are easy to add and facilitate test failure analysis.
          Hide
          Suresh Srinivas added a comment -

          I am not sure I understand what you mean here. Say, I have a descriptive message, say it says, "file exists() must return true". Can you describe how this will facilitate test failure analysis.

          I think this adds unnecessary bloat to the code, because you have to look at the code any way to understand the failure. I do not mind if people have added this kind of messages in their patch. However, I think mandating that it needs to be added seems unnecessary.

          Show
          Suresh Srinivas added a comment - I am not sure I understand what you mean here. Say, I have a descriptive message, say it says, "file exists() must return true". Can you describe how this will facilitate test failure analysis. I think this adds unnecessary bloat to the code, because you have to look at the code any way to understand the failure. I do not mind if people have added this kind of messages in their patch. However, I think mandating that it needs to be added seems unnecessary.
          Hide
          Sanjay Radia added a comment -

          Attach patch addresses the Assert error message issue that cos raised. Also a minor improvement in the javadoc.

          Show
          Sanjay Radia added a comment - Attach patch addresses the Assert error message issue that cos raised. Also a minor improvement in the javadoc.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12478021/viewFs3.patch
          against trunk revision 1098888.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 54 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/447//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12478021/viewFs3.patch against trunk revision 1098888. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 54 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/447//console This message is automatically generated.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12478021/viewFs3.patch
          against trunk revision 1098840.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 54 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          -1 javac. The applied patch generated 1077 javac compiler warnings (more than the trunk's current 1070 warnings).

          -1 findbugs. The patch appears to introduce 8 new Findbugs (version 1.3.9) warnings.

          -1 release audit. The applied patch generated 8 release audit warnings (more than the trunk's current 1 warnings).

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/395//testReport/
          Release audit warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/395//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
          Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/395//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/395//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12478021/viewFs3.patch against trunk revision 1098840. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 54 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 1077 javac compiler warnings (more than the trunk's current 1070 warnings). -1 findbugs. The patch appears to introduce 8 new Findbugs (version 1.3.9) warnings. -1 release audit. The applied patch generated 8 release audit warnings (more than the trunk's current 1 warnings). +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/395//testReport/ Release audit warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/395//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/395//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/395//console This message is automatically generated.
          Hide
          Sanjay Radia added a comment -

          Updated patch: Fixed findbug and compiler warnings

          Show
          Sanjay Radia added a comment - Updated patch: Fixed findbug and compiler warnings
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12478200/viewFs4.patch
          against trunk revision 1099284.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 54 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          -1 javac. The applied patch generated 1072 javac compiler warnings (more than the trunk's current 1070 warnings).

          -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/398//testReport/
          Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/398//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/398//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12478200/viewFs4.patch against trunk revision 1099284. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 54 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 1072 javac compiler warnings (more than the trunk's current 1070 warnings). -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/398//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/398//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/398//console This message is automatically generated.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12478238/viewFs5.patch
          against trunk revision 1099633.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 54 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/402//testReport/
          Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/402//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/402//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12478238/viewFs5.patch against trunk revision 1099633. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 54 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/402//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/402//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/402//console This message is automatically generated.
          Hide
          Jitendra Nath Pandey added a comment -

          +1 for the patch.

          Show
          Jitendra Nath Pandey added a comment - +1 for the patch.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #573 (See https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/573/)
          HADOOP-7257 Client side mount tables (sanjay)

          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #573 (See https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/573/ ) HADOOP-7257 Client side mount tables (sanjay)
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk #680 (See https://builds.apache.org/hudson/job/Hadoop-Common-trunk/680/)
          HADOOP-7257 Client side mount tables (sanjay)

          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk #680 (See https://builds.apache.org/hudson/job/Hadoop-Common-trunk/680/ ) HADOOP-7257 Client side mount tables (sanjay)
          Hide
          Ravi Prakash added a comment -

          @Sanjay: ViewFs.java:624

              static private void checkPathIsSlash(final Path f) throws IOException {
                if (f != InodeTree.SlashPath) {
                  throw new IOException (
                  "Internal implementation error: expected file name to be /" );
                }
              }
          

          Should the condition be

          if (f.equals(Inode.SlashPath)
          

          My small program

          	Path p = new Path("/");
          	Path q = new Path("/");
          	System.out.println("Equals? " + (p == q) );
          

          Prints "Equals? false"

          Show
          Ravi Prakash added a comment - @Sanjay: ViewFs.java:624 static private void checkPathIsSlash(final Path f) throws IOException { if (f != InodeTree.SlashPath) { throw new IOException ( "Internal implementation error: expected file name to be /" ); } } Should the condition be if (f.equals(Inode.SlashPath) My small program Path p = new Path("/"); Path q = new Path("/"); System.out.println("Equals? " + (p == q) ); Prints "Equals? false"
          Hide
          Eli Collins added a comment -

          I have attach the pdf of the javadoc for viewfs which explains how to use this feature. I will later post a user guide document.

          Filed HADOOP-7953.

          Show
          Eli Collins added a comment - I have attach the pdf of the javadoc for viewfs which explains how to use this feature. I will later post a user guide document. Filed HADOOP-7953 .

            People

            • Assignee:
              Sanjay Radia
              Reporter:
              Sanjay Radia
            • Votes:
              0 Vote for this issue
              Watchers:
              19 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development