Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9806 Allow HDFS block replicas to be provided by an external storage system
  3. HDFS-12665

[AliasMap] Create a version of the AliasMap that runs in memory in the Namenode (leveldb)

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.1.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The design of Provided Storage requires the use of an AliasMap to manage the mapping between blocks of files on the local HDFS and ranges of files on a remote storage system. To reduce load from the Namenode, this can be done using a pluggable external service (e.g. AzureTable, Cassandra, Ratis). However, to aide adoption and ease of deployment, we propose an in memory version.

      This AliasMap will be a wrapper around LevelDB (already a dependency from the Timeline Service) and use protobuf for the key (blockpool, blockid, and genstamp) and the value (url, offset, length, nonce). The in memory service will also have a configurable port on which it will listen for updates from Storage Policy Satisfier (SPS) Coordinating Datanodes (C-DN).

        Attachments

        1. HDFS-12665-HDFS-9806.001.patch
          165 kB
          Ewan Higgs
        2. HDFS-12665-HDFS-9806.002.patch
          124 kB
          Ewan Higgs
        3. HDFS-12665-HDFS-9806.003.patch
          166 kB
          Ewan Higgs
        4. HDFS-12665-HDFS-9806.004.patch
          125 kB
          Ewan Higgs
        5. HDFS-12665-HDFS-9806.005.patch
          120 kB
          Virajith Jalaparti
        6. HDFS-12665-HDFS-9806.006.patch
          137 kB
          Ewan Higgs
        7. HDFS-12665-HDFS-9806.007.patch
          115 kB
          Virajith Jalaparti
        8. HDFS-12665-HDFS-9806.008.patch
          107 kB
          Virajith Jalaparti
        9. HDFS-12665-HDFS-9806.009.patch
          108 kB
          Ewan Higgs
        10. HDFS-12665-HDFS-9806.010.patch
          107 kB
          Virajith Jalaparti
        11. HDFS-12665-HDFS-9806.011.patch
          116 kB
          Virajith Jalaparti
        12. HDFS-12665-HDFS-9806.012.patch
          107 kB
          Virajith Jalaparti

          Issue Links

            Activity

              People

              • Assignee:
                ehiggs Ewan Higgs
                Reporter:
                ehiggs Ewan Higgs
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: