Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-15289

Allow viewfs mounts with HDFS/HCFS scheme and centralized mount table

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 3.2.0
    • Fix Version/s: None
    • Component/s: fs
    • Labels:
      None

      Description

      ViewFS provides flexibility to mount different filesystem types with mount points configuration table. This approach is solving the scalability problems, but users need to reconfigure the filesystem to ViewFS and to its scheme.  This will be problematic in the case of paths persisted in meta stores, ex: Hive. In systems like Hive, it will store uris in meta store. So, changing the file system scheme will create a burden to upgrade/recreate meta stores. In our experience many users are not ready to change that.  

      Router based federation is another implementation to provide coordinated mount points for HDFS federation clusters. Even though this provides flexibility to handle mount points easily, this will not allow other(non-HDFS) file systems to mount. So, this does not solve the purpose when users want to mount external(non-HDFS) filesystems.

      So, the problem here is: Even though many users want to adapt to the scalable fs options available, technical challenges of changing schemes (ex: in meta stores) in deployments are obstructing them.

      So, we propose to allow hdfs scheme in ViewFS like client side mount system and provision user to create mount links without changing URI paths. 
      I will upload detailed design doc shortly.

        Attachments

        1. ViewFSOverloadScheme.png
          186 kB
          Uma Maheswara Rao G
        2. ViewFSOverloadScheme - V1.0.pdf
          166 kB
          Uma Maheswara Rao G

          Issue Links

            Activity

              People

              • Assignee:
                umamaheswararao Uma Maheswara Rao G
                Reporter:
                umamaheswararao Uma Maheswara Rao G
              • Votes:
                0 Vote for this issue
                Watchers:
                39 Start watching this issue

                Dates

                • Created:
                  Updated:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 15h 10m
                  15h 10m