[HDFS-5477] Block manager as a service

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.0.0-alpha, 3.0.0-alpha1
• Component/s: namenode
• Assignee: Daryn Sharp
• Reporter: Daryn Sharp
• Created: 08/Nov/13 01:24
• Updated: 17/May/16 20:24

Description

The block manager needs to evolve towards having the ability to run as a standalone service to improve NN vertical and horizontal scalability. The goal is reducing the memory footprint of the NN proper to support larger namespaces, and improve overall performance by decoupling the block manager from the namespace and its lock. Ideally, a distinct BM will be transparent to clients and DNs.

Issue Links

• blocks: HDFS-9814 Advance Hybrid Cloud Architecture (AHA) - HDFS Block Level Leases (Locks)
• relates to: HDFS-10419 Building HDFS on top of new storage layer (HDDS)

Sub-Tasks

• Separate memory space between BM and NN: Open
• Refactoring to define interfaces between BM and NN and simplify the flow between them: Open
• Separate concurrency control between BM and NN: Open
• A NN-internal RPC BM service: Open
• Testing support for BM as a service: Open
• BM service as a separate process: Open