Hadoop Common
  1. Hadoop Common
  2. HADOOP-4952

Improved files system interface for the application writer.

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.21.0
    • Fix Version/s: 0.21.0
    • Component/s: fs
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      New FileContext API introduced to replace FileSystem API. FileContext will be the version-compatible API for future releases. FileSystem API will be deprecated in the next release.

      Description

      Currently the FIleSystem interface serves two purposes:

      • an application writer's interface for using the Hadoop file system
      • a file system implementer's interface (e.g. hdfs, local file system, kfs, etc)

      This Jira proposes that we provide a simpler interfaces for the application writer and leave the FilsSystem interface for the implementer of a filesystem.

      • Filesystem interface has a confusing set of methods for the application writer
      • We could make it easier to take advantage of the URI file naming
        • Current approach is to get FileSystem instance by supplying the URI and then access that name space. It is consistent for the FileSystem instance to not accept URIs for other schemes, but we can do better.
        • The special copyFromLocalFIle can be generalized as a copyFile where the src or target can be generalized to any URI, including the local one.
        • The proposed scheme (below) simplifies this.
      • The client side config can be simplified.
        • New config() by default uses the default config. Since this is the common usage pattern, one should not need to always pass the config as a parameter when accessing the file system.
        • It does not handle multiple file systems too well. Today a site.xml is derived from a single Hadoop cluster. This does not make sense for multiple Hadoop clusters which may have different defaults.
        • Further one should need very little to configure the client side:
          • Default files system.
          • Block size
          • Replication factor
          • Scheme to class mapping
        • It should be possible to take Blocksize and replication factors defaults from the target file system, rather then the client size config. I am not suggesting we don't allow setting client side defaults, but most clients do not care and would find it simpler to take the defaults for their systems from the target file system.
      1. FileContext-common25.patch
        92 kB
        Sanjay Radia
      2. FileContext-common24.patch
        92 kB
        Sanjay Radia
      3. FileContext-common22.patch
        91 kB
        Sanjay Radia
      4. FileContext-common21.patch
        92 kB
        Sanjay Radia
      5. FileContext-common19.patch
        90 kB
        Sanjay Radia
      6. FileContext-common18.patch
        88 kB
        Sanjay Radia
      7. FileContext-common16.patch
        88 kB
        Sanjay Radia
      8. FileContext-common14.patch
        88 kB
        Sanjay Radia
      9. FileContext-common13.patch
        88 kB
        Sanjay Radia
      10. FileContext-common12.patch
        88 kB
        Sanjay Radia
      11. FileContext-hdfs11.patch
        8 kB
        Sanjay Radia
      12. FileContext-common11.patch
        83 kB
        Sanjay Radia
      13. FileContext-hdfs10.patch
        5 kB
        Sanjay Radia
      14. FileContext-common10.patch
        81 kB
        Sanjay Radia
      15. FileContext9.patch
        79 kB
        Sanjay Radia
      16. FileContext7.patch
        73 kB
        Sanjay Radia
      17. FileContext6.patch
        70 kB
        Sanjay Radia
      18. FileContext5.patch
        68 kB
        Sanjay Radia
      19. FileContext3.patch
        65 kB
        Sanjay Radia
      20. FilesContext2.patch
        66 kB
        Sanjay Radia
      21. FilesContext1.patch
        53 kB
        Sanjay Radia
      22. Files.java
        15 kB
        Sanjay Radia
      23. Files.java
        11 kB
        Sanjay Radia

        Issue Links

          Activity

            People

            • Assignee:
              Sanjay Radia
              Reporter:
              Sanjay Radia
            • Votes:
              0 Vote for this issue
              Watchers:
              27 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development