Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-14477

FileSystem Simplify / Optimize listStatus Method

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Patch Available
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: 2.7.3, 3.0.0-alpha3
    • Fix Version/s: None
    • Component/s: fs
    • Labels:
      None
    • Flags:
      Patch

      Description

      org.apache.hadoop.fs.FileSystem.listStatus(ArrayList<FileStatus>, Path, PathFilter)
        /*
         * Filter files/directories in the given path using the user-supplied path
         * filter. Results are added to the given array <code>results</code>.
         */
        private void listStatus(ArrayList<FileStatus> results, Path f,
            PathFilter filter) throws FileNotFoundException, IOException {
          FileStatus listing[] = listStatus(f);
          if (listing == null) {
            throw new IOException("Error accessing " + f);
          }
      
          for (int i = 0; i < listing.length; i++) {
            if (filter.accept(listing[i].getPath())) {
              results.add(listing[i]);
            }
          }
        }
      
      org.apache.hadoop.fs.FileSystem.listStatus(Path, PathFilter)
        public FileStatus[] listStatus(Path f, PathFilter filter) 
                                         throws FileNotFoundException, IOException {
          ArrayList<FileStatus> results = new ArrayList<FileStatus>();
          listStatus(results, f, filter);
          return results.toArray(new FileStatus[results.size()]);
        }
      

      We can be smarter about this:

      1. Use enhanced for-loops
      2. Optimize for the case where there are zero files in a directory, save on object instantiation
      3. More encapsulated design

        Attachments

        1. HADOOP-14477.2.patch
          3 kB
          David Mollitor
        2. HADOOP-14477.1.patch
          3 kB
          David Mollitor

          Issue Links

            Activity

              People

              • Assignee:
                belugabehr David Mollitor
                Reporter:
                belugabehr David Mollitor
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated: