Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-2187

FileSystem should return location information with byte ranges

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Duplicate
    • None
    • None
    • fs
    • None

    Description

      The FileSystem interface should provide location information with byte ranges rather than a String[][] of locations. I suggest that we deprecate FileSystem.getFileCacheHints and replace it with:

      abstract public class FileSystem {
         ...
         public static class BlockInformation implements Writable {
            public BlockInformation(long start, String[] locations) {...}
            public String[] getHosts() {...}
            public long getStartingOffset() {...}
         }
         BlockInformation[] getFileLocations(Path f, long start, long length) { ... }
      }
      

      This will allow us to fix the FileInputFormat in map/reduce to make just one call per a file to the name node instead of one per a block.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              omalley Owen O'Malley
              Votes:
              1 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: