Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-6678

Remove FileContext#isFile, isDirectory and exists

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: fs
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      1. Add a method Iterator<FileStatus> listStatus(Path), which allows HDFS client not to have the whole listing in the memory, benefit more from the iterative listing added in HDFS-985. Move the current FileStatus[] listStatus(Path) to be a utility method.
      2. Remove methods isFile(Path), isDirectory(Path), and exists.
        All these methods are implemented by calling getFileStatus(Path).But most users are not aware of this. They would write code as below:
          FileContext fc = ..;
          if (fc.exists(path)) {
            if (fc.isFile(path)) {
             ...
            } else {
            ...
            }
          }
        

        The above code adds unnecessary getFileInfo RPC to NameNode. In our production clusters, we often see that the number of getFileStatus calls is multiple times of the open calls. If we remove isFile, isDirectory, and exists from FileContext, users have to explicitly call getFileStatus first, it is more likely that they will write more efficient code as follow:

          FileContext fc = ...;
          FileStatus fstatus = fc.getFileStatus(path);
          if (fstatus.isFile() {
            ...
          } else {
            ...
          }
        

        Attachments

        1. hadoop-6678-1.patch
          67 kB
          Eli Collins
        2. hadoop-6678-2.patch
          46 kB
          Eli Collins

          Issue Links

            Activity

              People

              • Assignee:
                eli Eli Collins
                Reporter:
                hairong Hairong Kuang
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: