[HDFS-7541] Upgrade Domains in HDFS

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.9.0, 3.0.0-alpha1, 2.8.2
• Assignee: Ming Ma
• Reporter: Ming Ma
• Created: 17/Dec/14 19:35
• Updated: 20/Aug/19 22:09
• Resolved: 16/Jun/16 05:09

Description

Current HDFS DN rolling upgrade step requires sequential DN restart to minimize the impact on data availability and read/write operations. The side effect is longer upgrade duration for large clusters. This might be acceptable for DN JVM quick restart to update hadoop code/configuration. However, for OS upgrade that requires machine reboot, the overall upgrade duration will be too long if we continue to do sequential DN rolling restart.

Issue Links

• is related to: HDFS-14637 Namenode may not replicate blocks to meet the policy after enabling upgradeDomain
• is related to: AMBARI-24301 API for HDFS Upgrade Domains management for scripted node additions
• is related to: HDFS-8789 Block Placement policy migrator
• is related to: HDFS-7877 [Umbrella] Support maintenance state for datanodes
• is related to: HDFS-13742 HDFS Upgrade Domains dynamic placement policy using dfsadmin commands and scripting API to be automation friendly for scripted datanode additions to clusters
• relates to: YARN-8256 Pluggable provider for node membership management
• relates to: HDFS-1431 Balancer should work with the logic of BlockPlacementPolicy
• relates to: HDFS-3566 Custom Replication Policy for Azure
• relates to: HDFS-13739 Add option to disable rack local write preference