Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9806 Allow HDFS block replicas to be provided by an external storage system
  3. HDFS-12665

[AliasMap] Create a version of the AliasMap that runs in memory in the Namenode (leveldb)

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 3.1.0
    • None
    • None
    • Reviewed

    Description

      The design of Provided Storage requires the use of an AliasMap to manage the mapping between blocks of files on the local HDFS and ranges of files on a remote storage system. To reduce load from the Namenode, this can be done using a pluggable external service (e.g. AzureTable, Cassandra, Ratis). However, to aide adoption and ease of deployment, we propose an in memory version.

      This AliasMap will be a wrapper around LevelDB (already a dependency from the Timeline Service) and use protobuf for the key (blockpool, blockid, and genstamp) and the value (url, offset, length, nonce). The in memory service will also have a configurable port on which it will listen for updates from Storage Policy Satisfier (SPS) Coordinating Datanodes (C-DN).

      Attachments

        1. HDFS-12665-HDFS-9806.001.patch
          165 kB
          Ewan Higgs
        2. HDFS-12665-HDFS-9806.002.patch
          124 kB
          Ewan Higgs
        3. HDFS-12665-HDFS-9806.003.patch
          166 kB
          Ewan Higgs
        4. HDFS-12665-HDFS-9806.004.patch
          125 kB
          Ewan Higgs
        5. HDFS-12665-HDFS-9806.005.patch
          120 kB
          Virajith Jalaparti
        6. HDFS-12665-HDFS-9806.006.patch
          137 kB
          Ewan Higgs
        7. HDFS-12665-HDFS-9806.007.patch
          115 kB
          Virajith Jalaparti
        8. HDFS-12665-HDFS-9806.008.patch
          107 kB
          Virajith Jalaparti
        9. HDFS-12665-HDFS-9806.009.patch
          108 kB
          Ewan Higgs
        10. HDFS-12665-HDFS-9806.010.patch
          107 kB
          Virajith Jalaparti
        11. HDFS-12665-HDFS-9806.011.patch
          116 kB
          Virajith Jalaparti
        12. HDFS-12665-HDFS-9806.012.patch
          107 kB
          Virajith Jalaparti

        Issue Links

          depends upon

          Sub-task - The sub-task of the issue HDFS-11902 [READ] Merge BlockFormatProvider and FileRegionProvider.

          • Major - Major loss of function.
          • Resolved
          is depended upon by

          Sub-task - The sub-task of the issue HDFS-12591 [READ] Implement LevelDBFileRegionFormat

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          is related to

          Bug - A problem which impairs or prevents the functions of the product. HDFS-13795 Fix potential NPE in InMemoryLevelDBAliasMapServer

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-13536 [PROVIDED Storage] HA for InMemoryAliasMap

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-13796 Allow verbosity of InMemoryLevelDBAliasMapServer to be configurable

          • Trivial - Cosmetic problem like misspelt words or misaligned text.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-13798 Create InMemoryAliasMap location if missing.

          • Major - Major loss of function.
          • Patch Available

          Task - A task that needs to be done. HDFS-12944 Update NOTICE for AssertJ dependency

          • Trivial - Cosmetic problem like misspelt words or misaligned text.
          • Resolved
          requires

          Sub-task - The sub-task of the issue HDFS-12713 [READ] Refactor FileRegion and BlockAliasMap to separate out HDFS metadata and PROVIDED storage metadata

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              ehiggs Ewan Higgs
              ehiggs Ewan Higgs
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: