Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-11058

Implement 'hadoop fs -df' command for ViewFileSystem

    Details

    • Type: Task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0-alpha1
    • Fix Version/s: 3.0.0-alpha2
    • Component/s: None
    • Labels:

      Description

      Df command doesn't seem to work well with ViewFileSystem. It always reports used data as 0. Here is the client mount table configuration I am using against a federated clusters of 2 NameNodes and 2 DataNoes.

        1 <?xml version="1.0" ?>
        2 <configuration>
        3   <property>
        4     <name>fs.defaultFS</name>
        5     <value>viewfs://ClusterX/</value>
        6   </property>
        ..
       11   <property>
       12     <name>fs.default.name</name>
       13     <value>viewfs://ClusterX/</value>
       14   </property>
       ..
       23   <property>
       24     <name>fs.viewfs.mounttable.ClusterX.link./nn0</name>
       25     <value>hdfs://127.0.0.1:50001/</value>
       26   </property>
       27   <property>
       28     <name>fs.viewfs.mounttable.ClusterX.link./nn1</name>
       29     <value>hdfs://127.0.0.1:51001/</value>
       30   </property>
       31   <property>
       32     <name>fs.viewfs.mounttable.ClusterX.link./nn2</name>
       33     <value>hdfs://127.0.0.1:52001/nn2</value>
       34   </property>
       35   <property>
       36     <name>fs.viewfs.mounttable.ClusterX.link./nn3</name>
       37     <value>hdfs://127.0.0.1:52001/nn3</value>
       38   </property>
       39   <property>
       40     <name>fs.viewfs.mounttable.ClusterY.linkMergeSlash</name>
       41     <value>hdfs://127.0.0.1:50001/</value>
       42   </property>
       43 </configuration>
      

      Df command always reports Size/Available as 8.0E and the usage as 0 for any federated cluster.

      # hadoop fs -fs viewfs://ClusterX/ -df  /
      Filesystem                         Size  Used            Available  Use%
      viewfs://ClusterX/  9223372036854775807     0  9223372036854775807    0%
      
      # hadoop fs -fs viewfs://ClusterX/ -df  -h /
      Filesystem           Size  Used  Available  Use%
      viewfs://ClusterX/  8.0 E     0      8.0 E    0%
      
      # hadoop fs -fs viewfs://ClusterY/ -df  -h /
      Filesystem           Size  Used  Available  Use%
      viewfs://ClusterY/  8.0 E     0      8.0 E    0%
      

      Whereas Du command seems to work as expected even with ViewFileSystem.

      # hadoop fs -fs viewfs://ClusterY/ -du -h /
      10.6 K  31.8 K  /build.log.16y
      0       0       /user
      
      # hadoop fs -fs viewfs://ClusterX/ -du -h /
      10.6 K  31.8 K  /nn0
      0       0       /nn1
      20.2 K  35.8 K  /nn3
      40.6 K  34.3 K  /nn4
      
      
      1. HDFS-11058.01.patch
        31 kB
        Manoj Govindassamy
      2. HDFS-11058.02.patch
        30 kB
        Manoj Govindassamy
      3. HDFS-11058.03.patch
        27 kB
        Manoj Govindassamy
      4. HDFS-11058.04.patch
        27 kB
        Manoj Govindassamy

        Issue Links

          Activity

          Hide
          manojg Manoj Govindassamy added a comment -

          Current df command behavior for DFS

          # hadoop fs -fs hdfs://127.0.0.1:50001/ -df -h /
          Filesystem                 Size  Used  Available  Use%
          hdfs://127.0.0.1:50001  930.6 G  96 K    652.6 G    0%  
          

          Current df command behavior for ViewFileSystem

          # hadoop fs -fs viewfs://ClusterX/ -df  -h /
          Filesystem           Size  Used  Available  Use%
          viewfs://ClusterX/  8.0 E     0      8.0 E    0%
          

          Just like df in traditional linux systems, here is my proposal for df command when run against ViewFileSystem

          # hadoop fs -fs viewfs://ClusterX/ -df  -h /
          Filesystem                     Mounted on     Size  Used  Available  Use%
          hdfs://NN0.example.com:port/         /nn0  100.0 G     0     80.0 G    0%  
          hdfs://NN1.example.com:port/         /nn1  100.0 G     0     80.0 G    0%  
          hdfs://NN2.example.com:port/         /nn2  100.0 G     0     80.0 G    0%  
          hdfs://NN3.example.com:port/         /nn3  100.0 G     0     80.0 G    0%  
          

          Andrew Wang, Lei (Eddy) Xu, Zhe Zhang, any thoughts on the above df proposal for viewfs:// ?

          Show
          manojg Manoj Govindassamy added a comment - Current df command behavior for DFS # hadoop fs -fs hdfs://127.0.0.1:50001/ -df -h / Filesystem Size Used Available Use% hdfs://127.0.0.1:50001 930.6 G 96 K 652.6 G 0% Current df command behavior for ViewFileSystem # hadoop fs -fs viewfs://ClusterX/ -df -h / Filesystem Size Used Available Use% viewfs://ClusterX/ 8.0 E 0 8.0 E 0% Just like df in traditional linux systems, here is my proposal for df command when run against ViewFileSystem # hadoop fs -fs viewfs://ClusterX/ -df -h / Filesystem Mounted on Size Used Available Use% hdfs://NN0.example.com:port/ /nn0 100.0 G 0 80.0 G 0% hdfs://NN1.example.com:port/ /nn1 100.0 G 0 80.0 G 0% hdfs://NN2.example.com:port/ /nn2 100.0 G 0 80.0 G 0% hdfs://NN3.example.com:port/ /nn3 100.0 G 0 80.0 G 0% Andrew Wang , Lei (Eddy) Xu , Zhe Zhang , any thoughts on the above df proposal for viewfs:// ?
          Hide
          andrew.wang Andrew Wang added a comment -

          Seems reasonable to me, though I'd prefer we put the new "Mounted on" as the last column and left-aligned it (which is how it shows on my local df).

          Adding this "Mounted on" column would be an incompatible change though. This is fine, but do you intend to only show it when it's a ViewFileSystem?

          Show
          andrew.wang Andrew Wang added a comment - Seems reasonable to me, though I'd prefer we put the new "Mounted on" as the last column and left-aligned it (which is how it shows on my local df). Adding this "Mounted on" column would be an incompatible change though. This is fine, but do you intend to only show it when it's a ViewFileSystem?
          Hide
          manojg Manoj Govindassamy added a comment -
          • 'Mounted on' as the last column sounds good to me.
          • Yes, this additional column and change in behavior is only when the FS is ViewFileSystem which can be detected at FsCommand level from PathData.fs
          Show
          manojg Manoj Govindassamy added a comment - 'Mounted on' as the last column sounds good to me. Yes, this additional column and change in behavior is only when the FS is ViewFileSystem which can be detected at FsCommand level from PathData.fs
          Hide
          manojg Manoj Govindassamy added a comment -

          Attaching v01 patch which addresses the following:

          1. hadoop fs -df command can now work for ViewFileSystem paths. It can detect ViewFileSystem paths and accordingly display enhanced df information based on ViewFileSystem mount points configured. Just like traditional linux systems, df for viewfs will display target file systems and FsStatus specific to each mount points.
          2. Unit tests to verify the new df behavior, and no regression for other filesystems

          Andrew Wang, can you please review the patch ?

          Show
          manojg Manoj Govindassamy added a comment - Attaching v01 patch which addresses the following: hadoop fs -df command can now work for ViewFileSystem paths. It can detect ViewFileSystem paths and accordingly display enhanced df information based on ViewFileSystem mount points configured. Just like traditional linux systems, df for viewfs will display target file systems and FsStatus specific to each mount points. Unit tests to verify the new df behavior, and no regression for other filesystems Andrew Wang , can you please review the patch ?
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 25s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 0m 17s Maven dependency ordering for branch
          +1 mvninstall 8m 26s trunk passed
          +1 compile 8m 5s trunk passed
          +1 checkstyle 1m 31s trunk passed
          +1 mvnsite 1m 51s trunk passed
          +1 mvneclipse 0m 25s trunk passed
          +1 findbugs 3m 1s trunk passed
          +1 javadoc 1m 19s trunk passed
          0 mvndep 0m 15s Maven dependency ordering for patch
          +1 mvninstall 1m 22s the patch passed
          +1 compile 7m 15s the patch passed
          -1 javac 7m 15s root generated 3 new + 700 unchanged - 1 fixed = 703 total (was 701)
          +1 checkstyle 1m 34s root: The patch generated 0 new + 220 unchanged - 4 fixed = 220 total (was 224)
          +1 mvnsite 1m 49s the patch passed
          +1 mvneclipse 0m 26s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 25s the patch passed
          +1 javadoc 1m 22s the patch passed
          -1 unit 8m 9s hadoop-common in the patch failed.
          -1 unit 91m 52s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 30s The patch does not generate ASF License warnings.
          144m 13s



          Reason Tests
          Failed junit tests hadoop.fs.viewfs.TestFSMainOperationsLocalFileSystem
            hadoop.hdfs.TestPersistBlocks
            hadoop.hdfs.TestEncryptionZones
            hadoop.metrics2.sink.TestRollingFileSystemSinkWithHdfs



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HDFS-11058
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12836506/HDFS-11058.01.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux b826995761fb 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / cb5cc0d
          Default Java 1.8.0_101
          findbugs v3.0.0
          javac https://builds.apache.org/job/PreCommit-HDFS-Build/17374/artifact/patchprocess/diff-compile-javac-root.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/17374/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/17374/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17374/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17374/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 25s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 0m 17s Maven dependency ordering for branch +1 mvninstall 8m 26s trunk passed +1 compile 8m 5s trunk passed +1 checkstyle 1m 31s trunk passed +1 mvnsite 1m 51s trunk passed +1 mvneclipse 0m 25s trunk passed +1 findbugs 3m 1s trunk passed +1 javadoc 1m 19s trunk passed 0 mvndep 0m 15s Maven dependency ordering for patch +1 mvninstall 1m 22s the patch passed +1 compile 7m 15s the patch passed -1 javac 7m 15s root generated 3 new + 700 unchanged - 1 fixed = 703 total (was 701) +1 checkstyle 1m 34s root: The patch generated 0 new + 220 unchanged - 4 fixed = 220 total (was 224) +1 mvnsite 1m 49s the patch passed +1 mvneclipse 0m 26s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 25s the patch passed +1 javadoc 1m 22s the patch passed -1 unit 8m 9s hadoop-common in the patch failed. -1 unit 91m 52s hadoop-hdfs in the patch failed. +1 asflicense 0m 30s The patch does not generate ASF License warnings. 144m 13s Reason Tests Failed junit tests hadoop.fs.viewfs.TestFSMainOperationsLocalFileSystem   hadoop.hdfs.TestPersistBlocks   hadoop.hdfs.TestEncryptionZones   hadoop.metrics2.sink.TestRollingFileSystemSinkWithHdfs Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-11058 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12836506/HDFS-11058.01.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux b826995761fb 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / cb5cc0d Default Java 1.8.0_101 findbugs v3.0.0 javac https://builds.apache.org/job/PreCommit-HDFS-Build/17374/artifact/patchprocess/diff-compile-javac-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17374/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17374/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17374/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17374/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          manojg Manoj Govindassamy added a comment -

          Andrew Wang, org.apache.hadoop.fs.viewfs.TestFSMainOperationsLocalFileSystem.testFsStatus failure does look like this patch issue. Will fix this along with other review comments when i get them.

          Show
          manojg Manoj Govindassamy added a comment - Andrew Wang , org.apache.hadoop.fs.viewfs.TestFSMainOperationsLocalFileSystem.testFsStatus failure does look like this patch issue. Will fix this along with other review comments when i get them.
          Hide
          andrew.wang Andrew Wang added a comment -

          Hi Manoj, thanks for working on this. As a high-level comment, I'd prefer to do renames / refactors as separate JIRAs to keep each one focused.

          • Looks like there was a refactor to replace MountPoint with ViewFsMountPoint. These have different fields though, so I don't think they substitute for each other. As an example, MountPoint has multiple targets which seems to be designed for merge mounts. This required the change in testGetMountPoints.
          • This patch adds new public APIs via a ViewFsUtil class. I commented about MountPoint vs. ViewFsMountPoint, but we also need to annotate whatever we expose as public/evolving also. This doesn't really relate to "df" also, so could be a separate JIRA.

          What is the intent of separating out responsive and non-responsive filesystems? In the single FS case, it'll throw an exception and a non-zero exit code. But with this patch, viewfs will lead to exceptions being swallowed. This is surprising and makes it harder to script around DF+viewfs.

          What does unix df do in this situation? I'm also okay just letting it throw, and having users query on a mount-by-mount basis (assuming we expose the list of mounts somewhere easily accessible).

          Show
          andrew.wang Andrew Wang added a comment - Hi Manoj, thanks for working on this. As a high-level comment, I'd prefer to do renames / refactors as separate JIRAs to keep each one focused. Looks like there was a refactor to replace MountPoint with ViewFsMountPoint. These have different fields though, so I don't think they substitute for each other. As an example, MountPoint has multiple targets which seems to be designed for merge mounts. This required the change in testGetMountPoints. This patch adds new public APIs via a ViewFsUtil class. I commented about MountPoint vs. ViewFsMountPoint, but we also need to annotate whatever we expose as public/evolving also. This doesn't really relate to "df" also, so could be a separate JIRA. What is the intent of separating out responsive and non-responsive filesystems? In the single FS case, it'll throw an exception and a non-zero exit code. But with this patch, viewfs will lead to exceptions being swallowed. This is surprising and makes it harder to script around DF+viewfs. What does unix df do in this situation? I'm also okay just letting it throw, and having users query on a mount-by-mount basis (assuming we expose the list of mounts somewhere easily accessible).
          Hide
          manojg Manoj Govindassamy added a comment - - edited

          Thanks for the review Andrew Wang.

          1. ViewFsMountPoint : Yes, this replaces ViewFileSystem#MountPoint. This is more than a refactor as the new definition is slightly different from the older one. So far there has not been any real outside user for ViewFileSystem#MountPoint. As part of HDFS-5684, I added annotations in MountPoint and extended the test case to print MountPoints. DfUsage for ViewFileSystem is the first real outside user for MountPoint, and I thought this would be right time for defining a proper ViewFsMountPoint. IMHO, ViewFsMountPoint should be abstracted and expose only the needed attributes – the MountedOn path and its target FileSystem. The FileSystem could be a hdfs:// or it could be a one for MergeFs, but I don't see a need for exposing all the NameServices, at least for now.
          1. ViewFsUtil : This is the new helper routine for ViewFileSystem. Already added the annotation InterfaceAudience.Public, InterfaceStability.Evolving to it. Utility functions inside this class are introduced only for DfUsage. Is it still worth to separate this out to a new patch when there are no other callers ?
          1. Responsive and non-responsive filesystems: Yes, this is a new behavior for DF, but this is only in the context of ViewFileSystem. I was contemplating on Availability vs Consistency for the command and inclined towards the Availability. When any one of the NameNode or backing filesystem is not reachable, then DF command will error out normally and none of the information would be printed. Whereas by skipping the unreachable ones, at least the reachable FileSystems are printed out. I have seen the unix df command getting stuck at times when NFS servers are not reachable. But, I am totally ok to remove this extra feature and error out when any of the backing NameServices are not reachable. May I will propse this as a separate patch and not part of this issue.

          Please let me know your thoughts.

          Show
          manojg Manoj Govindassamy added a comment - - edited Thanks for the review Andrew Wang . ViewFsMountPoint : Yes, this replaces ViewFileSystem#MountPoint . This is more than a refactor as the new definition is slightly different from the older one. So far there has not been any real outside user for ViewFileSystem#MountPoint . As part of HDFS-5684 , I added annotations in MountPoint and extended the test case to print MountPoints. DfUsage for ViewFileSystem is the first real outside user for MountPoint, and I thought this would be right time for defining a proper ViewFsMountPoint . IMHO, ViewFsMountPoint should be abstracted and expose only the needed attributes – the MountedOn path and its target FileSystem. The FileSystem could be a hdfs:// or it could be a one for MergeFs , but I don't see a need for exposing all the NameServices, at least for now. ViewFsUtil : This is the new helper routine for ViewFileSystem. Already added the annotation InterfaceAudience.Public, InterfaceStability.Evolving to it. Utility functions inside this class are introduced only for DfUsage . Is it still worth to separate this out to a new patch when there are no other callers ? Responsive and non-responsive filesystems: Yes, this is a new behavior for DF , but this is only in the context of ViewFileSystem. I was contemplating on Availability vs Consistency for the command and inclined towards the Availability. When any one of the NameNode or backing filesystem is not reachable, then DF command will error out normally and none of the information would be printed. Whereas by skipping the unreachable ones, at least the reachable FileSystems are printed out. I have seen the unix df command getting stuck at times when NFS servers are not reachable. But, I am totally ok to remove this extra feature and error out when any of the backing NameServices are not reachable. May I will propse this as a separate patch and not part of this issue. Please let me know your thoughts.
          Hide
          andrew.wang Andrew Wang added a comment -

          Thanks for the thoughtful response Manoj. I need to think about this some more, but a few ideas for discussion:

          IMHO, ViewFsMountPoint should be abstracted and expose only the needed attributes – the MountedOn path and its target FileSystem. The FileSystem could be a hdfs:// or it could be a one for MergeFs, but I don't see a need for exposing all the NameServices, at least for now.

          I think the intent was to implement merging in ViewFileSystem itself, rather than a new FileSystem. So we'd need to return an array here, like in the original MountPoint.

          Our user API for referring to a FileSystem is also by URI, not by object reference. Yes, the user can always call getUri, but there is global state in a FileSystem like file handles and statistics, and it might be better to not share that by handing out a FileSystem object which they can poke at. Also, since it looks like we allow mounting subdirectories, FileSystem#getUri by itself is underspecified without the path component.

          Finally, what is the reason for using generics? getTargetFileSystem will always return a FileSystem right?

          <ViewFsUtil>

          Not worth to separate out, though we should think about this some more.

          As a semi-side note, I'm quite surprised that ViewFileSystem is annotated @Public. My impression from DistributedFileSystem is that the FileSystem subclasses are private, and are only used when casted as a FileSystem. This is why we have HdfsAdmin, which lets you do DFS-specific operations.

          ViewFsUtil is similar to HdfsAdmin, but used to examine an already created ViewFileSystem instance. However, since getStatus takes a ViewFileSystem, it forces the user to downcast which is unfortunate. Instead, we could have an isViewFileSystem API, and having getStatus take a FileSystem and throwing UnsupportedOperation if the passed FS is not a VFS.

          Finally, we should probably also name this ViewFileSystemUtil since ViewFs is the FileContext implementation.

          I have seen the unix df command getting stuck at times when NFS servers are not reachable. But, I am totally ok to remove this extra feature and error out when any of the backing NameServices are not reachable.

          Good point, I've seen similar behavior as well. Let's tackle this in a separate JIRA though, and maybe put the behavior behind a flag. I do think we should return non-zero in this case, and think about how scripts will be able to parse the output.

          Show
          andrew.wang Andrew Wang added a comment - Thanks for the thoughtful response Manoj. I need to think about this some more, but a few ideas for discussion: IMHO, ViewFsMountPoint should be abstracted and expose only the needed attributes – the MountedOn path and its target FileSystem. The FileSystem could be a hdfs:// or it could be a one for MergeFs, but I don't see a need for exposing all the NameServices, at least for now. I think the intent was to implement merging in ViewFileSystem itself, rather than a new FileSystem. So we'd need to return an array here, like in the original MountPoint. Our user API for referring to a FileSystem is also by URI, not by object reference. Yes, the user can always call getUri , but there is global state in a FileSystem like file handles and statistics, and it might be better to not share that by handing out a FileSystem object which they can poke at. Also, since it looks like we allow mounting subdirectories, FileSystem#getUri by itself is underspecified without the path component. Finally, what is the reason for using generics? getTargetFileSystem will always return a FileSystem right? <ViewFsUtil> Not worth to separate out, though we should think about this some more. As a semi-side note, I'm quite surprised that ViewFileSystem is annotated @Public. My impression from DistributedFileSystem is that the FileSystem subclasses are private, and are only used when casted as a FileSystem. This is why we have HdfsAdmin, which lets you do DFS-specific operations. ViewFsUtil is similar to HdfsAdmin, but used to examine an already created ViewFileSystem instance. However, since getStatus takes a ViewFileSystem, it forces the user to downcast which is unfortunate. Instead, we could have an isViewFileSystem API, and having getStatus take a FileSystem and throwing UnsupportedOperation if the passed FS is not a VFS. Finally, we should probably also name this ViewFileSystemUtil since ViewFs is the FileContext implementation. I have seen the unix df command getting stuck at times when NFS servers are not reachable. But, I am totally ok to remove this extra feature and error out when any of the backing NameServices are not reachable. Good point, I've seen similar behavior as well. Let's tackle this in a separate JIRA though, and maybe put the behavior behind a flag. I do think we should return non-zero in this case, and think about how scripts will be able to parse the output.
          Hide
          manojg Manoj Govindassamy added a comment -

          Thanks for the detailed review Andrew Wang. Much appreciated.

          Our user API for referring to a FileSystem is also by URI, not by object reference. es, the user can always call getUri, but there is global state in a FileSystem like file handles and statistics, and it might be better to not share that by handing out a FileSystem object which they can poke at. .

          Very valid point. Exposing the whole FS object is not a good thing. Will expose an array of NN URIs.

          what is the reason for using generics? getTargetFileSystem will always return a FileSystem right?

          The source for ViewFsMountPoint is T in the generic class InodeTree. So to be in sync with the data source, made the dependent class also a generic one. But, now that T is not going to be exposed in ViewFsMountPoint, I can remove the generics also.

          Instead, we could have an isViewFileSystem API, and having getStatus take a FileSystem and throwing UnsupportedOperation if the passed FS is not a VFS...we should probably also name this ViewFileSystemUtil since ViewFs is the FileContext implementation.

          Sounds good. Will do as you suggested.

          Unresponsive FileSystem .. Let's tackle this in a separate JIRA though, and maybe put the behavior behind a flag.

          Sure, will track this as a new Jira.

          Show
          manojg Manoj Govindassamy added a comment - Thanks for the detailed review Andrew Wang . Much appreciated. Our user API for referring to a FileSystem is also by URI, not by object reference. es, the user can always call getUri, but there is global state in a FileSystem like file handles and statistics, and it might be better to not share that by handing out a FileSystem object which they can poke at. . Very valid point. Exposing the whole FS object is not a good thing. Will expose an array of NN URIs. what is the reason for using generics? getTargetFileSystem will always return a FileSystem right? The source for ViewFsMountPoint is T in the generic class InodeTree . So to be in sync with the data source, made the dependent class also a generic one. But, now that T is not going to be exposed in ViewFsMountPoint, I can remove the generics also. Instead, we could have an isViewFileSystem API, and having getStatus take a FileSystem and throwing UnsupportedOperation if the passed FS is not a VFS...we should probably also name this ViewFileSystemUtil since ViewFs is the FileContext implementation. Sounds good. Will do as you suggested. Unresponsive FileSystem .. Let's tackle this in a separate JIRA though, and maybe put the behavior behind a flag. Sure, will track this as a new Jira.
          Hide
          manojg Manoj Govindassamy added a comment -

          Attached v02 patch to address previous review comments.

          1. Fixed a test failure
          2. Refactored ViewFsMountPoint to carry URIs instead of FileSystem object, removed the generics as ther is no more need.
          3. Renamed ViewFsUtil to ViewFileSystemUtil
          4. Removed code pertaining to 'Unresponsive FileSystem' from DfUsage. Will propose this as part of a new task - HDFS-11109
            Andrew Wang, kindly take a look at the updated patch.

          Sample output of Df command when run against Federated Cluster of 2 NameSpaces.

          # hadoop fs -df -h /
          Filesystem                Size  Used  Available  Use%  Mounted on
          hdfs://127.0.0.1:51001/  1.4 T  48 K    942.3 G    0%  /nn1
          hdfs://127.0.0.1:50001/  1.4 T  48 K    942.3 G    0%  /nn0
          
          # hadoop fs -df -h /nn0
          Filesystem                Size  Used  Available  Use%  Mounted on
          hdfs://127.0.0.1:50001/  1.4 T  48 K    942.3 G    0%  /nn0
          manoj@~/work/test/hadev-mg(master): hadoop fs -df -h /nn0/user.
          df: `/nn0/user.': No such file or directory
          
          # hadoop fs -df -h /nn0/user/
          Filesystem                Size  Used  Available  Use%  Mounted on
          hdfs://127.0.0.1:50001/  1.4 T  48 K    942.3 G    0%  /nn0
          
          # hadoop fs -df -h /nn0/user/manoj
          Filesystem                Size  Used  Available  Use%  Mounted on
          hdfs://127.0.0.1:50001/  1.4 T  48 K    942.3 G    0%  /nn0
          
          #hadoop fs -df -h /abc
          df: `/abc': No such file or directory
          
          # hadoop fs -df -h /nn0/abc
          df: `/nn0/abc': No such file or directory
          
          
          Show
          manojg Manoj Govindassamy added a comment - Attached v02 patch to address previous review comments. Fixed a test failure Refactored ViewFsMountPoint to carry URIs instead of FileSystem object, removed the generics as ther is no more need. Renamed ViewFsUtil to ViewFileSystemUtil Removed code pertaining to 'Unresponsive FileSystem' from DfUsage. Will propose this as part of a new task - HDFS-11109 Andrew Wang , kindly take a look at the updated patch. Sample output of Df command when run against Federated Cluster of 2 NameSpaces. # hadoop fs -df -h / Filesystem Size Used Available Use% Mounted on hdfs://127.0.0.1:51001/ 1.4 T 48 K 942.3 G 0% /nn1 hdfs://127.0.0.1:50001/ 1.4 T 48 K 942.3 G 0% /nn0 # hadoop fs -df -h /nn0 Filesystem Size Used Available Use% Mounted on hdfs://127.0.0.1:50001/ 1.4 T 48 K 942.3 G 0% /nn0 manoj@~/work/test/hadev-mg(master): hadoop fs -df -h /nn0/user. df: `/nn0/user.': No such file or directory # hadoop fs -df -h /nn0/user/ Filesystem Size Used Available Use% Mounted on hdfs://127.0.0.1:50001/ 1.4 T 48 K 942.3 G 0% /nn0 # hadoop fs -df -h /nn0/user/manoj Filesystem Size Used Available Use% Mounted on hdfs://127.0.0.1:50001/ 1.4 T 48 K 942.3 G 0% /nn0 #hadoop fs -df -h /abc df: `/abc': No such file or directory # hadoop fs -df -h /nn0/abc df: `/nn0/abc': No such file or directory
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 0m 15s Maven dependency ordering for branch
          +1 mvninstall 6m 49s trunk passed
          +1 compile 6m 58s trunk passed
          +1 checkstyle 1m 32s trunk passed
          +1 mvnsite 1m 47s trunk passed
          +1 mvneclipse 0m 25s trunk passed
          +1 findbugs 2m 58s trunk passed
          +1 javadoc 1m 21s trunk passed
          0 mvndep 0m 14s Maven dependency ordering for patch
          +1 mvninstall 1m 21s the patch passed
          +1 compile 6m 51s the patch passed
          -1 javac 6m 51s root generated 3 new + 691 unchanged - 3 fixed = 694 total (was 694)
          +1 checkstyle 1m 28s root: The patch generated 0 new + 220 unchanged - 4 fixed = 220 total (was 224)
          +1 mvnsite 1m 44s the patch passed
          +1 mvneclipse 0m 25s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 16s the patch passed
          +1 javadoc 1m 21s the patch passed
          +1 unit 7m 19s hadoop-common in the patch passed.
          -1 unit 72m 43s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 25s The patch does not generate ASF License warnings.
          120m 23s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.namenode.TestCacheDirectives



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HDFS-11058
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12837295/HDFS-11058.02.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux bc5115b97c2f 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 6bb741f
          Default Java 1.8.0_101
          findbugs v3.0.0
          javac https://builds.apache.org/job/PreCommit-HDFS-Build/17437/artifact/patchprocess/diff-compile-javac-root.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/17437/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17437/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17437/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 0m 15s Maven dependency ordering for branch +1 mvninstall 6m 49s trunk passed +1 compile 6m 58s trunk passed +1 checkstyle 1m 32s trunk passed +1 mvnsite 1m 47s trunk passed +1 mvneclipse 0m 25s trunk passed +1 findbugs 2m 58s trunk passed +1 javadoc 1m 21s trunk passed 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 1m 21s the patch passed +1 compile 6m 51s the patch passed -1 javac 6m 51s root generated 3 new + 691 unchanged - 3 fixed = 694 total (was 694) +1 checkstyle 1m 28s root: The patch generated 0 new + 220 unchanged - 4 fixed = 220 total (was 224) +1 mvnsite 1m 44s the patch passed +1 mvneclipse 0m 25s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 16s the patch passed +1 javadoc 1m 21s the patch passed +1 unit 7m 19s hadoop-common in the patch passed. -1 unit 72m 43s hadoop-hdfs in the patch failed. +1 asflicense 0m 25s The patch does not generate ASF License warnings. 120m 23s Reason Tests Failed junit tests hadoop.hdfs.server.namenode.TestCacheDirectives Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-11058 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12837295/HDFS-11058.02.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux bc5115b97c2f 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 6bb741f Default Java 1.8.0_101 findbugs v3.0.0 javac https://builds.apache.org/job/PreCommit-HDFS-Build/17437/artifact/patchprocess/diff-compile-javac-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17437/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17437/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17437/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          andrew.wang Andrew Wang added a comment -

          Thanks for working on this Manoj. I think we're getting close, I only have small comments:

          • Like the Util class, we should rename ViewFsMountPoint to ViewFileSystemMountPoint to distinguish it from ViewFs. It might be easier to leave it as a nested class in ViewFileSystem, since this is a verbose name.
          • If VFSMountPoint is meant to be a public class, it needs InterfaceAudience and InterfaceStability annotations, and javadoc on the getters.
          • I'd consider renaming InodeTreeMountPoint back to InodeTree.MountPoint, since the names are equally as verbose, and the nested one can sometimes be referred to as just MountPoint.
          • Could we also add a test that also tries the "human readable" behavior for df?
          • Extra space in VFSUtil "static void updateMountPointFsStatus"
          • VFSUtil#getStatus could use some unit tests too, the behavior looks reasonably complex based on the javadoc description.
          Show
          andrew.wang Andrew Wang added a comment - Thanks for working on this Manoj. I think we're getting close, I only have small comments: Like the Util class, we should rename ViewFsMountPoint to ViewFileSystemMountPoint to distinguish it from ViewFs. It might be easier to leave it as a nested class in ViewFileSystem, since this is a verbose name. If VFSMountPoint is meant to be a public class, it needs InterfaceAudience and InterfaceStability annotations, and javadoc on the getters. I'd consider renaming InodeTreeMountPoint back to InodeTree.MountPoint, since the names are equally as verbose, and the nested one can sometimes be referred to as just MountPoint. Could we also add a test that also tries the "human readable" behavior for df? Extra space in VFSUtil "static void updateMountPointFsStatus" VFSUtil#getStatus could use some unit tests too, the behavior looks reasonably complex based on the javadoc description.
          Hide
          manojg Manoj Govindassamy added a comment -

          Thanks for the review Andrew Wang. Incorporated your review comments. Please take a look.

          1. Sure, moved back ViewFsMountPoint as a nested class under ViewFileSystem.
          2. Now that ViewFileSystemMountPoint is not going to be a public class, interface annotations are not needed.
          3. The reason I chose to rename them is because there are three versions of MountPoint (ViewFileSystem, ViewFs, InodeTree) and it could be confusing on which one being used. I made the external caller FsUsage to explicitly specify prefix the parent class name so as to avoid confusion.
          4. yes, already added TestViewFileSystemHdfs#testDf to verify the human readable behavior of Df command. Test verifies the following
            1. When the fs is not viewfs://, Df command should not display _Mounted on column
            2. When the fs is viewfs://, Df command should display _Mounted on column
            3. Various Paths and their corresponding mounted filesystem usages
          5. Removed the extra white space in the definition for updateMountPointFsStatus.
          6. Added one more test ViewFileSystemBaseTest#testtestViewFileSystemUtil to verify all ViewFileSystemUtil contracts
          Show
          manojg Manoj Govindassamy added a comment - Thanks for the review Andrew Wang . Incorporated your review comments. Please take a look. Sure, moved back ViewFsMountPoint as a nested class under ViewFileSystem. Now that ViewFileSystemMountPoint is not going to be a public class, interface annotations are not needed. The reason I chose to rename them is because there are three versions of MountPoint (ViewFileSystem, ViewFs, InodeTree) and it could be confusing on which one being used. I made the external caller FsUsage to explicitly specify prefix the parent class name so as to avoid confusion. yes, already added TestViewFileSystemHdfs#testDf to verify the human readable behavior of Df command. Test verifies the following When the fs is not viewfs:// , Df command should not display _Mounted on column When the fs is viewfs:// , Df command should display _Mounted on column Various Paths and their corresponding mounted filesystem usages Removed the extra white space in the definition for updateMountPointFsStatus. Added one more test ViewFileSystemBaseTest#testtestViewFileSystemUtil to verify all ViewFileSystemUtil contracts
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 21s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 1m 40s Maven dependency ordering for branch
          +1 mvninstall 8m 38s trunk passed
          +1 compile 10m 37s trunk passed
          +1 checkstyle 1m 45s trunk passed
          +1 mvnsite 2m 12s trunk passed
          +1 mvneclipse 0m 42s trunk passed
          +1 findbugs 3m 17s trunk passed
          +1 javadoc 1m 35s trunk passed
          0 mvndep 0m 14s Maven dependency ordering for patch
          +1 mvninstall 1m 23s the patch passed
          +1 compile 10m 8s the patch passed
          -1 javac 10m 8s root generated 3 new + 688 unchanged - 3 fixed = 691 total (was 691)
          +1 checkstyle 1m 41s root: The patch generated 0 new + 143 unchanged - 4 fixed = 143 total (was 147)
          +1 mvnsite 2m 6s the patch passed
          +1 mvneclipse 0m 38s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 33s the patch passed
          +1 javadoc 1m 39s the patch passed
          -1 unit 9m 6s hadoop-common in the patch failed.
          +1 unit 80m 43s hadoop-hdfs in the patch passed.
          +1 asflicense 0m 33s The patch does not generate ASF License warnings.
          143m 42s



          Reason Tests
          Failed junit tests hadoop.fs.viewfs.TestViewfsFileStatus



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-11058
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12839089/HDFS-11058.03.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux cf9dcb9792d8 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 61c0bed
          Default Java 1.8.0_101
          findbugs v3.0.0
          javac https://builds.apache.org/job/PreCommit-HDFS-Build/17578/artifact/patchprocess/diff-compile-javac-root.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/17578/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17578/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17578/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 21s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 1m 40s Maven dependency ordering for branch +1 mvninstall 8m 38s trunk passed +1 compile 10m 37s trunk passed +1 checkstyle 1m 45s trunk passed +1 mvnsite 2m 12s trunk passed +1 mvneclipse 0m 42s trunk passed +1 findbugs 3m 17s trunk passed +1 javadoc 1m 35s trunk passed 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 1m 23s the patch passed +1 compile 10m 8s the patch passed -1 javac 10m 8s root generated 3 new + 688 unchanged - 3 fixed = 691 total (was 691) +1 checkstyle 1m 41s root: The patch generated 0 new + 143 unchanged - 4 fixed = 143 total (was 147) +1 mvnsite 2m 6s the patch passed +1 mvneclipse 0m 38s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 33s the patch passed +1 javadoc 1m 39s the patch passed -1 unit 9m 6s hadoop-common in the patch failed. +1 unit 80m 43s hadoop-hdfs in the patch passed. +1 asflicense 0m 33s The patch does not generate ASF License warnings. 143m 42s Reason Tests Failed junit tests hadoop.fs.viewfs.TestViewfsFileStatus Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11058 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12839089/HDFS-11058.03.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux cf9dcb9792d8 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 61c0bed Default Java 1.8.0_101 findbugs v3.0.0 javac https://builds.apache.org/job/PreCommit-HDFS-Build/17578/artifact/patchprocess/diff-compile-javac-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17578/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17578/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17578/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 18m 22s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          0 mvndep 1m 46s Maven dependency ordering for branch
          +1 mvninstall 6m 56s trunk passed
          +1 compile 9m 25s trunk passed
          +1 checkstyle 1m 32s trunk passed
          +1 mvnsite 1m 58s trunk passed
          +1 mvneclipse 0m 37s trunk passed
          +1 findbugs 3m 9s trunk passed
          +1 javadoc 1m 32s trunk passed
          0 mvndep 0m 14s Maven dependency ordering for patch
          +1 mvninstall 1m 21s the patch passed
          +1 compile 9m 5s the patch passed
          -1 javac 9m 5s root generated 3 new + 688 unchanged - 3 fixed = 691 total (was 691)
          +1 checkstyle 1m 31s root: The patch generated 0 new + 143 unchanged - 4 fixed = 143 total (was 147)
          +1 mvnsite 1m 57s the patch passed
          +1 mvneclipse 0m 35s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 25s the patch passed
          +1 javadoc 1m 29s the patch passed
          +1 unit 7m 24s hadoop-common in the patch passed.
          +1 unit 54m 10s hadoop-hdfs in the patch passed.
          +1 asflicense 0m 33s The patch does not generate ASF License warnings.
          128m 8s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-11058
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12839236/HDFS-11058.03.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 02143dc175eb 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / c90891e
          Default Java 1.8.0_111
          findbugs v3.0.0
          javac https://builds.apache.org/job/PreCommit-HDFS-Build/17591/artifact/patchprocess/diff-compile-javac-root.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17591/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17591/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 18m 22s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 1m 46s Maven dependency ordering for branch +1 mvninstall 6m 56s trunk passed +1 compile 9m 25s trunk passed +1 checkstyle 1m 32s trunk passed +1 mvnsite 1m 58s trunk passed +1 mvneclipse 0m 37s trunk passed +1 findbugs 3m 9s trunk passed +1 javadoc 1m 32s trunk passed 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 1m 21s the patch passed +1 compile 9m 5s the patch passed -1 javac 9m 5s root generated 3 new + 688 unchanged - 3 fixed = 691 total (was 691) +1 checkstyle 1m 31s root: The patch generated 0 new + 143 unchanged - 4 fixed = 143 total (was 147) +1 mvnsite 1m 57s the patch passed +1 mvneclipse 0m 35s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 25s the patch passed +1 javadoc 1m 29s the patch passed +1 unit 7m 24s hadoop-common in the patch passed. +1 unit 54m 10s hadoop-hdfs in the patch passed. +1 asflicense 0m 33s The patch does not generate ASF License warnings. 128m 8s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11058 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12839236/HDFS-11058.03.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 02143dc175eb 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / c90891e Default Java 1.8.0_111 findbugs v3.0.0 javac https://builds.apache.org/job/PreCommit-HDFS-Build/17591/artifact/patchprocess/diff-compile-javac-root.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17591/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17591/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 23m 15s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          0 mvndep 0m 17s Maven dependency ordering for branch
          +1 mvninstall 7m 1s trunk passed
          +1 compile 11m 22s trunk passed
          +1 checkstyle 1m 37s trunk passed
          +1 mvnsite 2m 14s trunk passed
          +1 mvneclipse 0m 36s trunk passed
          +1 findbugs 3m 31s trunk passed
          +1 javadoc 1m 46s trunk passed
          0 mvndep 0m 21s Maven dependency ordering for patch
          +1 mvninstall 1m 53s the patch passed
          +1 compile 11m 34s the patch passed
          -1 javac 11m 34s root generated 3 new + 688 unchanged - 3 fixed = 691 total (was 691)
          +1 checkstyle 1m 49s root: The patch generated 0 new + 144 unchanged - 4 fixed = 144 total (was 148)
          +1 mvnsite 2m 14s the patch passed
          +1 mvneclipse 0m 41s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 4m 0s the patch passed
          +1 javadoc 1m 52s the patch passed
          +1 unit 9m 44s hadoop-common in the patch passed.
          -1 unit 97m 57s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 36s The patch does not generate ASF License warnings.
          185m 56s



          Reason Tests
          Failed junit tests hadoop.hdfs.TestPersistBlocks
            hadoop.hdfs.server.blockmanagement.TestReconstructStripedBlocksWithRackAwareness



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-11058
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12839236/HDFS-11058.03.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux cb841daf20f7 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / c90891e
          Default Java 1.8.0_111
          findbugs v3.0.0
          javac https://builds.apache.org/job/PreCommit-HDFS-Build/17590/artifact/patchprocess/diff-compile-javac-root.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/17590/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17590/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17590/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 23m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 0m 17s Maven dependency ordering for branch +1 mvninstall 7m 1s trunk passed +1 compile 11m 22s trunk passed +1 checkstyle 1m 37s trunk passed +1 mvnsite 2m 14s trunk passed +1 mvneclipse 0m 36s trunk passed +1 findbugs 3m 31s trunk passed +1 javadoc 1m 46s trunk passed 0 mvndep 0m 21s Maven dependency ordering for patch +1 mvninstall 1m 53s the patch passed +1 compile 11m 34s the patch passed -1 javac 11m 34s root generated 3 new + 688 unchanged - 3 fixed = 691 total (was 691) +1 checkstyle 1m 49s root: The patch generated 0 new + 144 unchanged - 4 fixed = 144 total (was 148) +1 mvnsite 2m 14s the patch passed +1 mvneclipse 0m 41s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 4m 0s the patch passed +1 javadoc 1m 52s the patch passed +1 unit 9m 44s hadoop-common in the patch passed. -1 unit 97m 57s hadoop-hdfs in the patch failed. +1 asflicense 0m 36s The patch does not generate ASF License warnings. 185m 56s Reason Tests Failed junit tests hadoop.hdfs.TestPersistBlocks   hadoop.hdfs.server.blockmanagement.TestReconstructStripedBlocksWithRackAwareness Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11058 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12839236/HDFS-11058.03.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux cb841daf20f7 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / c90891e Default Java 1.8.0_111 findbugs v3.0.0 javac https://builds.apache.org/job/PreCommit-HDFS-Build/17590/artifact/patchprocess/diff-compile-javac-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17590/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17590/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17590/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          andrew.wang Andrew Wang added a comment -

          LGTM, +1 though it looks like this patch has gone stale. Could you rebase? I promise to commit soon.

          Show
          andrew.wang Andrew Wang added a comment - LGTM, +1 though it looks like this patch has gone stale. Could you rebase? I promise to commit soon.
          Hide
          manojg Manoj Govindassamy added a comment -

          Thanks for the review Andrew Wang. Attaching v04 patch after the rebase to latest trunk.

          Show
          manojg Manoj Govindassamy added a comment - Thanks for the review Andrew Wang . Attaching v04 patch after the rebase to latest trunk.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 11s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          0 mvndep 2m 22s Maven dependency ordering for branch
          +1 mvninstall 7m 23s trunk passed
          +1 compile 10m 13s trunk passed
          +1 checkstyle 1m 39s trunk passed
          +1 mvnsite 2m 13s trunk passed
          +1 mvneclipse 0m 39s trunk passed
          +1 findbugs 3m 23s trunk passed
          +1 javadoc 1m 39s trunk passed
          0 mvndep 0m 15s Maven dependency ordering for patch
          +1 mvninstall 1m 23s the patch passed
          +1 compile 9m 42s the patch passed
          -1 javac 9m 42s root generated 3 new + 711 unchanged - 3 fixed = 714 total (was 714)
          +1 checkstyle 1m 33s root: The patch generated 0 new + 143 unchanged - 4 fixed = 143 total (was 147)
          +1 mvnsite 2m 12s the patch passed
          +1 mvneclipse 0m 35s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 4m 3s the patch passed
          +1 javadoc 1m 43s the patch passed
          +1 unit 7m 56s hadoop-common in the patch passed.
          -1 unit 69m 42s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 33s The patch does not generate ASF License warnings.
          130m 27s



          Reason Tests
          Failed junit tests hadoop.hdfs.TestEncryptionZones



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-11058
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12840168/HDFS-11058.04.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 05dd024b5601 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 2bf9a15
          Default Java 1.8.0_111
          findbugs v3.0.0
          javac https://builds.apache.org/job/PreCommit-HDFS-Build/17642/artifact/patchprocess/diff-compile-javac-root.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/17642/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17642/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17642/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 11s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 2m 22s Maven dependency ordering for branch +1 mvninstall 7m 23s trunk passed +1 compile 10m 13s trunk passed +1 checkstyle 1m 39s trunk passed +1 mvnsite 2m 13s trunk passed +1 mvneclipse 0m 39s trunk passed +1 findbugs 3m 23s trunk passed +1 javadoc 1m 39s trunk passed 0 mvndep 0m 15s Maven dependency ordering for patch +1 mvninstall 1m 23s the patch passed +1 compile 9m 42s the patch passed -1 javac 9m 42s root generated 3 new + 711 unchanged - 3 fixed = 714 total (was 714) +1 checkstyle 1m 33s root: The patch generated 0 new + 143 unchanged - 4 fixed = 143 total (was 147) +1 mvnsite 2m 12s the patch passed +1 mvneclipse 0m 35s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 4m 3s the patch passed +1 javadoc 1m 43s the patch passed +1 unit 7m 56s hadoop-common in the patch passed. -1 unit 69m 42s hadoop-hdfs in the patch failed. +1 asflicense 0m 33s The patch does not generate ASF License warnings. 130m 27s Reason Tests Failed junit tests hadoop.hdfs.TestEncryptionZones Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11058 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12840168/HDFS-11058.04.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 05dd024b5601 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 2bf9a15 Default Java 1.8.0_111 findbugs v3.0.0 javac https://builds.apache.org/job/PreCommit-HDFS-Build/17642/artifact/patchprocess/diff-compile-javac-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17642/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17642/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17642/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          manojg Manoj Govindassamy added a comment -

          Test failure is not related to the patch.

          Show
          manojg Manoj Govindassamy added a comment - Test failure is not related to the patch.
          Hide
          andrew.wang Andrew Wang added a comment -

          Committed to trunk, thanks for working on this Manoj!

          Show
          andrew.wang Andrew Wang added a comment - Committed to trunk, thanks for working on this Manoj!

            People

            • Assignee:
              manojg Manoj Govindassamy
              Reporter:
              manojg Manoj Govindassamy
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development