Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9806

Allow HDFS block replicas to be provided by an external storage system

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.1.0
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      Provided storage allows data stored outside HDFS to be mapped to and addressed from HDFS. It builds on heterogeneous storage by introducing a new storage type, PROVIDED, to the set of media in a datanode. Clients accessing data in PROVIDED storages can cache replicas in local media, enforce HDFS invariants (e.g., security, quotas), and address more data than the cluster could persist in the storage attached to DataNodes.
      Show
      Provided storage allows data stored outside HDFS to be mapped to and addressed from HDFS. It builds on heterogeneous storage by introducing a new storage type, PROVIDED, to the set of media in a datanode. Clients accessing data in PROVIDED storages can cache replicas in local media, enforce HDFS invariants (e.g., security, quotas), and address more data than the cluster could persist in the storage attached to DataNodes.

      Description

      In addition to heterogeneous media, many applications work with heterogeneous storage systems. The guarantees and semantics provided by these systems are often similar, but not identical to those of HDFS. Any client accessing multiple storage systems is responsible for reasoning about each system independently, and must propagate/and renew credentials for each store.

      Remote stores could be mounted under HDFS. Block locations could be mapped to immutable file regions, opaque IDs, or other tokens that represent a consistent view of the data. While correctness for arbitrary operations requires careful coordination between stores, in practice we can provide workable semantics with weaker guarantees.

        Attachments

        1. HDFS-9806.003.patch
          500 kB
          Virajith Jalaparti
        2. HDFS-9806.002.patch
          500 kB
          Virajith Jalaparti
        3. HDFS-9806.001.patch
          501 kB
          Virajith Jalaparti
        4. HDFS-9806-design.002.pdf
          429 kB
          Virajith Jalaparti
        5. HDFS-9806-design.001.pdf
          360 kB
          Christopher Douglas

          Issue Links

          There are no Sub-Tasks for this issue.

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                cdouglas Christopher Douglas
              • Votes:
                0 Vote for this issue
                Watchers:
                79 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: