[HDFS-12090] Handling writes from HDFS to Provided storages

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Labels: pull-request-available
• Assignee: Unassigned
• Reporter: Virajith Jalaparti
• Created: 05/Jul/17 17:47
• Updated: 08/Dec/20 01:57

Description

HDFS-9806 introduces the concept of PROVIDED storage, which makes data in external storage systems accessible through HDFS. However, HDFS-9806 is limited to data being read through HDFS. This JIRA will deal with how data can be written to such PROVIDED storages from HDFS.

Issue Links

• is part of: HDFS-15714 HDFS Provided Storage Read/Write Mount Support On-the-fly
• is related to: HDFS-9806 Allow HDFS block replicas to be provided by an external storage system
• is related to: HDFS-14805 Mounting external stores in HDFS on-the-fly
• links to: GitHub Pull Request #1726

Sub-Tasks

• [PROVIDED Phase 2] Encode the BlockAlias in the client protocol: Closed
• [PROVIDED Phase 2] Refactor FsDatasetImpl to use the BlockAlias from ClientProtocol for PROVIDED blocks.: Open
• [PROVIDED Phase 2] Command line tools for managing Provided Storage Backup mounts: Resolved
• [DISCUSS] Provided Storage BlockAlias Refactoring: Open
• [PROVIDED Phase 2] Provided Storage Mount Manager (PSMM) mount: Patch Available
• [PROVIDED Phase 2] Block Bit Field Allocation of Provided Storage: Resolved
• [PROVIDED Phase 2] Add a pluggable policy for selecting locations for Provided files.: Open
• [PROVIDED Phase 2] The DatanodeProtocol should be have DNA_BACKUP to backup blocks: Resolved
• [PROVIDED Phase 2] Implement DNA_BACKUP command in Datanode: Resolved
• [PROVIDED Phase 2] Implement pause/resume for syncservice: Open
• [PROVIDED Phase 2] Multipart Uploader API: Resolved
• [PROVIDED Phase 2] Implement full resync command for syncservice: Open
• [PROVIDED Phase 2] Provided ReplicaMap should be LRU with separate lookup from normal Replicas: Open
• Add specification of Multipart Upload API to FS specification, with contract tests: Resolved
• HDFS Classes used for implementation of Multipart uploads to move to hadoop-common JAR: Resolved
• [PROVIDED Phase 2] Scheduler in the NN for distributing DNA_BACKUP work.: Resolved
• [PROVIDED Phase 2] Teach BlockAliasMap.Writer `remove` method.: Resolved
• Multipart uploaders to be created through API call to FileSystem/FileContext, not service loader: Resolved
• multipart upload to HDFS to support 0 byte upload: Resolved
• Multipart Uploader APIs to be marked as private/unstable in 3.2.0: Resolved
• Implement Multipart-aware distcp: Open
• remove META-INF/services/org.apache.hadoop.fs.MultipartUploader file: Open