[HDFS-9806] Allow HDFS block replicas to be provided by an external storage system

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.1.0
• Target Version/s: 3.1.0
• Hadoop Flags: Reviewed

Release Note

Provided storage allows data stored outside HDFS to be mapped to and addressed from HDFS. It builds on heterogeneous storage by introducing a new storage type, PROVIDED, to the set of media in a datanode. Clients accessing data in PROVIDED storages can cache replicas in local media, enforce HDFS invariants (e.g., security, quotas), and address more data than the cluster could persist in the storage attached to DataNodes.

• Assignee: Unassigned
• Reporter: Christopher Douglas
• Created: 13/Feb/16 01:36
• Updated: 04/Dec/20 05:05
• Resolved: 16/Dec/17 02:15

Description

In addition to heterogeneous media, many applications work with heterogeneous storage systems. The guarantees and semantics provided by these systems are often similar, but not identical to those of HDFS. Any client accessing multiple storage systems is responsible for reasoning about each system independently, and must propagate/and renew credentials for each store.

Remote stores could be mounted under HDFS. Block locations could be mapped to immutable file regions, opaque IDs, or other tokens that represent a consistent view of the data. While correctness for arbitrary operations requires careful coordination between stores, in practice we can provide workable semantics with weaker guarantees.

Issue Links

• incorporates: HDFS-10706 [READ] Add tool generating FSImage from external store
• is depended upon by: HDFS-13069 Enable HDFS to cache data read from external storage systems
• is related to: HDFS-11125 [SPS]: Use smaller batches of BlockMovingInfo into the block storage movement command
• is related to: HDFS-15660 StorageTypeProto is not compatiable between 3.x and 2.6
• is related to: HDFS-14805 Mounting external stores in HDFS on-the-fly
• is related to: HDFS-7878 API - expose a unique file identifier
• is related to: HDFS-9807 Add an optional StorageID to writes
• is related to: HDFS-9808 Combine READ_ONLY_SHARED DatanodeStorages with the same ID
• is related to: HDFS-12882 Support full open(PathHandle) contract in HDFS
• relates to: HDFS-12090 Handling writes from HDFS to Provided storages
• relates to: HDFS-13536 [PROVIDED Storage] HA for InMemoryAliasMap
• relates to: HDFS-14390 Provide kerberos support for AliasMap service used by Provided storage
• relates to: HDFS-9810 Allow support for more than one block replica per datanode
• relates to: HDFS-9809 Abstract implementation-specific details from the datanode

Sub-Tasks

• [READ] Datanode support to read from external stores.: Resolved
• [READ] Add tool generating FSImage from external store: Resolved
• [READ] Namenode support for data stored in external stores.: Resolved
• [READ] ProvidedReplica should return an InputStream that is bounded by its length: Resolved
• [READ] Fix NullPointerException in ProvidedBlocksBuilder: Resolved
• [READ] Tests for ProvidedStorageMap: Resolved
• [READ] Handle failures of Datanode with PROVIDED storage: Resolved
• [READ] Test for increasing replication of provided files.: Resolved
• [READ] Test cases for ProvidedVolumeDF and ProviderBlockIteratorImpl: Resolved
• [READ] Datanodes should use a unique identifier when reading from external stores: Resolved
• [READ] Merge BlockFormatProvider and FileRegionProvider.: Resolved
• [READ] Check that the replicas served from a {{ProvidedVolumeImpl}} belong to the correct external storage: Resolved
• [READ] Share remoteFS between ProvidedReplica instances.: Resolved
• [READ] HDFS-12091 breaks the tests for provided block reads: Resolved
• [READ] Fix errors in image generation tool from latest rebase: Resolved
• [READ] TestNameNodeProvidedImplementation#testProvidedDatanodeFailures fails after rebase: Resolved
• [READ] Even one dead datanode with PROVIDED storage results in ProvidedStorageInfo being marked as FAILED: Resolved
• [READ] FsVolumeImpl exception when scanning Provided storage volume: Resolved
• [READ] Test NameNode restarts when PROVIDED is configured: Resolved
• [AliasMap] Create a version of the AliasMap that runs in memory in the Namenode (leveldb): Resolved
• [READ] Implement LevelDBFileRegionFormat: Resolved
• [READ] Refactor FileRegion and BlockAliasMap to separate out HDFS metadata and PROVIDED storage metadata: Resolved
• [READ] Fix reporting of Provided volumes: Resolved
• [READ] Increasing replication for PROVIDED files should create local replicas: Resolved
• [READ] Reduce memory and CPU footprint for PROVIDED volumes.: Resolved
• [READ] Report multiple locations for PROVIDED blocks: Resolved
• [READ] Allow cluster id to be specified to the Image generation tool: Resolved
• [READ] Image generation tool does not close an opened stream: Resolved
• [READ] Fix the randomized selection of locations in {{ProvidedBlocksBuilder}}.: Resolved
• [READ] Documentation for provided storage: Resolved
• Add visibility/stability annotations: Resolved
• [READ] Support replication of Provided blocks with non-default topologies.: Resolved
• [READ] Allow Datanodes with Provided volumes to start when blocks with the same id exist locally: Resolved
• [READ] Skip setting block count of ProvidedDatanodeStorageInfo on DN registration update: Resolved
• [READ] Fix closing streams in ImageWriter: Resolved
• [READ] Handle decommissioning and under-maintenance Datanodes with Provided storage.: Resolved
• [9806] Code style cleanup: Resolved
• [READ] Fix configuration and implementation of LevelDB-based alias maps: Resolved