[HDDS-1266] [Ozone upgrade] Support Upgrading HDFS clusters to use Ozone

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Labels: Triaged
• Assignee: Anu Engineer
• Reporter: Anu Engineer
• Created: 14/Mar/19 00:03
• Updated: 27/Sep/20 15:47

Description

This is the master JIRA to support upgrading existing HDFS clusters to have Ozone running concurrently. One of the requirements is that we support upgrading from HDFS to Ozone, without a full data copy. This requirement is called "In Place upgrade", the end result of such an upgrade would be to have the HDFS data appear in Ozone as if Ozone has taken a snap-shot of the HDFS data. Once upgrade is complete, Ozone and HDFS will act as independent systems. I will post a design document soon.

Issue Links

• is related to: HDDS-4283 Remove unsupported upgrade command in ozone cli

Sub-Tasks

• [Ozone Upgrade] Create a tool/background service called Planner: Open
• [Ozone Upgrade] Add ability to read the FSImage from Namenode.: Open
• [Ozone Upgrade] Add the ability to read block information from Namenode.: Open
• [Ozone Upgrade] Create the protobuf definitions for persisting the block to node map: Open
• [Ozone Upgrade] Add support for persisting the blockMap to RocksDB: Open
• [Ozone Upgrade] Create HDFS cluster Status: Open
• [Ozone Upgrade] Compute data moves at various sizes: Open
• [Ozone Upgrade] Create a move plan: Open
• [Ozone Upgrade] Create a mover for executing the move plan: Open
• [Ozone Upgrade] Convert BlocktoNode map to Ozone SCM DB: Open
• [Ozone Upgrade] Add Support for HDFS containers in Datanode: Open
• [Ozone Upgrade] Add Support for HDFS Container protocol in the Ozone upgrade planner: Open
• [Ozone Upgrade] Add support for generating block tokens to communicate with data nodes for upgrade.: Open
• [Ozone Upgrade] Add support for OM and SCM to run in upgrade mode.: Open