Hadoop Common
  1. Hadoop Common
  2. HADOOP-6569

FsShell#cat should avoid calling unecessary getFileStatus before opening a file to read

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: fs
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed

      Description

      Since FileSystem#open throws a FileNotFoundException when the file to be read does not exist, there is no need to check if the file is a directory or not before open. In case of HDFS, this could reduce one getFileInfo RPC to NameNode.

      1. optimizeCat.patch
        0.6 kB
        Hairong Kuang
      2. optimizeCat-yahoo2.patch
        3 kB
        Hairong Kuang
      3. optimizeCat-yahoo1.patch
        2 kB
        Hairong Kuang
      4. optimizeCat-yahoo.patch
        0.7 kB
        Hairong Kuang
      5. optimizeCat.patch
        0.6 kB
        Hairong Kuang

        Issue Links

          Activity

          Hide
          Eli Collins added a comment -

          +1 looks good to me.

          Show
          Eli Collins added a comment - +1 looks good to me.
          Hide
          Hairong Kuang added a comment -

          A patch for yahoo 0.20 branch.

          Show
          Hairong Kuang added a comment - A patch for yahoo 0.20 branch.
          Hide
          Hairong Kuang added a comment -

          The change made to FsShell#cat makes the output of cat different when the src does not exist or is a directory. This patch made changes to testCLI configuration file to accommodate this output change.

          Show
          Hairong Kuang added a comment - The change made to FsShell#cat makes the output of cat different when the src does not exist or is a directory. This patch made changes to testCLI configuration file to accommodate this output change.
          Hide
          Hairong Kuang added a comment -

          After much thinking back and forth, I eventually decided to change the DistributedFileSystem#open as well. When the file to be open does not exist or is a directory, open throws a FileNotFoundException instead of IOException. The error message is changed from "cannot open a file" to be "File doesn't exist".

          Show
          Hairong Kuang added a comment - After much thinking back and forth, I eventually decided to change the DistributedFileSystem#open as well. When the file to be open does not exist or is a directory, open throws a FileNotFoundException instead of IOException. The error message is changed from "cannot open a file" to be "File doesn't exist".
          Hide
          Suresh Srinivas added a comment -

          +1 for the patch. This is a good change.

          Show
          Suresh Srinivas added a comment - +1 for the patch. This is a good change.
          Hide
          Hairong Kuang added a comment -

          The patch is for trunk.

          Show
          Hairong Kuang added a comment - The patch is for trunk.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12437641/optimizeCat.patch
          against trunk revision 917737.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/402/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/402/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/402/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/402/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12437641/optimizeCat.patch against trunk revision 917737. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/402/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/402/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/402/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/402/console This message is automatically generated.
          Hide
          Hairong Kuang added a comment -

          The test is provided in HDFS-1016.

          Show
          Hairong Kuang added a comment - The test is provided in HDFS-1016 .
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #194 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/194/)
          . FsShell#cat should avoid calling unecessary getFileStatus before opening a file to read. Contributed by Hairong Kuang.

          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #194 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/194/ ) . FsShell#cat should avoid calling unecessary getFileStatus before opening a file to read. Contributed by Hairong Kuang.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk #266 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/266/)
          . FsShell#cat should avoid calling unecessary getFileStatus before opening a file to read. Contributed by Hairong Kuang.

          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk #266 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/266/ ) . FsShell#cat should avoid calling unecessary getFileStatus before opening a file to read. Contributed by Hairong Kuang.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #208 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/208/)
          HDFS-1016. HDFS side change for . Contributed by Hairong Kuang.

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #208 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/208/ ) HDFS-1016 . HDFS side change for . Contributed by Hairong Kuang.
          Hide
          Hairong Kuang added a comment -

          I've committed this for a while.

          Show
          Hairong Kuang added a comment - I've committed this for a while.
          Hide
          Hudson added a comment -

          Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #146 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/146/)

          Show
          Hudson added a comment - Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #146 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/146/ )
          Hide
          Hudson added a comment -

          Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #302 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/302/)

          Show
          Hudson added a comment - Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #302 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/302/ )
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #275 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/275/)

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #275 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/275/ )

            People

            • Assignee:
              Hairong Kuang
              Reporter:
              Hairong Kuang
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development