Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9806 Allow HDFS block replicas to be provided by an external storage system
  3. HDFS-12665

[AliasMap] Create a version of the AliasMap that runs in memory in the Namenode (leveldb)

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 3.1.0
    • None
    • None
    • Reviewed

    Description

      The design of Provided Storage requires the use of an AliasMap to manage the mapping between blocks of files on the local HDFS and ranges of files on a remote storage system. To reduce load from the Namenode, this can be done using a pluggable external service (e.g. AzureTable, Cassandra, Ratis). However, to aide adoption and ease of deployment, we propose an in memory version.

      This AliasMap will be a wrapper around LevelDB (already a dependency from the Timeline Service) and use protobuf for the key (blockpool, blockid, and genstamp) and the value (url, offset, length, nonce). The in memory service will also have a configurable port on which it will listen for updates from Storage Policy Satisfier (SPS) Coordinating Datanodes (C-DN).

      Attachments

        1. HDFS-12665-HDFS-9806.001.patch
          165 kB
          Ewan Higgs
        2. HDFS-12665-HDFS-9806.002.patch
          124 kB
          Ewan Higgs
        3. HDFS-12665-HDFS-9806.003.patch
          166 kB
          Ewan Higgs
        4. HDFS-12665-HDFS-9806.004.patch
          125 kB
          Ewan Higgs
        5. HDFS-12665-HDFS-9806.005.patch
          120 kB
          Virajith Jalaparti
        6. HDFS-12665-HDFS-9806.006.patch
          137 kB
          Ewan Higgs
        7. HDFS-12665-HDFS-9806.007.patch
          115 kB
          Virajith Jalaparti
        8. HDFS-12665-HDFS-9806.008.patch
          107 kB
          Virajith Jalaparti
        9. HDFS-12665-HDFS-9806.009.patch
          108 kB
          Ewan Higgs
        10. HDFS-12665-HDFS-9806.010.patch
          107 kB
          Virajith Jalaparti
        11. HDFS-12665-HDFS-9806.011.patch
          116 kB
          Virajith Jalaparti
        12. HDFS-12665-HDFS-9806.012.patch
          107 kB
          Virajith Jalaparti

        Issue Links

          Activity

            People

              ehiggs Ewan Higgs
              ehiggs Ewan Higgs
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: