Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-7541

Upgrade Domains in HDFS

          Details

          • Type: Improvement
          • Status: Resolved
          • Priority: Major
          • Resolution: Fixed
          • Affects Version/s: None
          • Fix Version/s: 2.9.0, 3.0.0-alpha1, 2.8.2
          • Component/s: None
          • Labels:
            None

            Description

            Current HDFS DN rolling upgrade step requires sequential DN restart to minimize the impact on data availability and read/write operations. The side effect is longer upgrade duration for large clusters. This might be acceptable for DN JVM quick restart to update hadoop code/configuration. However, for OS upgrade that requires machine reboot, the overall upgrade duration will be too long if we continue to do sequential DN rolling restart.

              Attachments

              1. HDFS-7541.patch
                67 kB
                Ming Ma
              2. HDFS-7541-2.patch
                89 kB
                Ming Ma
              3. SupportforfastHDFSdatanoderollingupgrade.pdf
                125 kB
                Ming Ma
              4. UpgradeDomains_design_v2.pdf
                288 kB
                Ming Ma
              5. UpgradeDomains_Design_v3.pdf
                242 kB
                Ming Ma

                Issue Links

                is related to

                New Feature - A new feature of the product, which has yet to be developed. AMBARI-24301 API for HDFS Upgrade Domains management for scripted node additions

                • Major - Major loss of function.
                • Open

                New Feature - A new feature of the product, which has yet to be developed. HDFS-8789 Block Placement policy migrator

                • Major - Major loss of function.
                • Open

                New Feature - A new feature of the product, which has yet to be developed. HDFS-7877 [Umbrella] Support maintenance state for datanodes

                • Major - Major loss of function.
                • Resolved

                Improvement - An improvement or enhancement to an existing feature or task. HDFS-13742 HDFS Upgrade Domains dynamic placement policy using dfsadmin commands and scripting API to be automation friendly for scripted datanode additions to clusters

                • Major - Major loss of function.
                • Open
                relates to

                New Feature - A new feature of the product, which has yet to be developed. YARN-8256 Pluggable provider for node membership management

                • Major - Major loss of function.
                • Open

                Improvement - An improvement or enhancement to an existing feature or task. HDFS-1431 Balancer should work with the logic of BlockPlacementPolicy

                • Major - Major loss of function.
                • Open

                Improvement - An improvement or enhancement to an existing feature or task. HDFS-13739 Option to disable Rack Local Write Preference to avoid 2 issues - 1. Rack-by-Rack Maintenance leaves last data replica at risk, 2. avoid Major Storage Imbalance across DataNodes caused by uneven spread of Datanodes across Racks

                • Major - Major loss of function.
                • Open

                Improvement - An improvement or enhancement to an existing feature or task. HDFS-3566 Custom Replication Policy for Azure

                • Major - Major loss of function.
                • Resolved

                  Activity

                    People

                    • Assignee:
                      mingma Ming Ma
                      Reporter:
                      mingma Ming Ma
                    • Votes:
                      0 Vote for this issue
                      Watchers:
                      40 Start watching this issue

                      Dates

                      • Created:
                        Updated:
                        Resolved: