Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-5535

Umbrella jira for improved HDFS rolling upgrades

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0, 2.2.0
    • Fix Version/s: 2.4.0
    • Component/s: datanode, ha, hdfs-client, namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      In order to roll a new HDFS release through a large cluster quickly and safely, a few enhancements are needed in HDFS. An initial High level design document will be attached to this jira, and sub-jiras will itemize the individual tasks.

      1. HDFSRollingUpgradesHighLevelDesign.pdf
        294 kB
        Suresh Srinivas
      2. h5535_20140219.patch
        435 kB
        Tsz Wo Nicholas Sze
      3. h5535_20140220-1554.patch
        438 kB
        Tsz Wo Nicholas Sze
      4. h5535_20140220b.patch
        455 kB
        Tsz Wo Nicholas Sze
      5. h5535_20140221-2031.patch
        435 kB
        Tsz Wo Nicholas Sze
      6. h5535_20140224-1931.patch
        495 kB
        Tsz Wo Nicholas Sze
      7. h5535_20140225-1225.patch
        531 kB
        Tsz Wo Nicholas Sze
      8. hdfs-5535-test-plan.pdf
        72 kB
        Haohui Mai
      9. h5535_20140226-1328.patch
        561 kB
        Tsz Wo Nicholas Sze
      10. h5535_20140226-1911.patch
        561 kB
        Tsz Wo Nicholas Sze
      11. h5535_20140227-1239.patch
        566 kB
        Tsz Wo Nicholas Sze
      12. h5535_20140228-1714.patch
        572 kB
        Tsz Wo Nicholas Sze
      13. HDFSRollingUpgradesHighLevelDesign.v2.pdf
        83 kB
        Suresh Srinivas
      14. h5535_20140304-1138.patch
        579 kB
        Tsz Wo Nicholas Sze
      15. HDFSRollingUpgradesHighLevelDesign.v3.pdf
        331 kB
        Kihwal Lee
      16. h5535_20140304-branch-2.patch
        601 kB
        Tsz Wo Nicholas Sze
      17. h5535_20140310-branch-2.patch
        602 kB
        Tsz Wo Nicholas Sze

        Issue Links

        1.
        Revist upgrade snapshots, roll back, finalize to enable rolling upgrades Sub-task Resolved Sanjay Radia
         
        2.
        Make replication queue initialization asynchronous Sub-task Resolved Vinayakumar B
         
        3.
        Improve datanode startup time Sub-task Resolved Kihwal Lee
         
        4.
        Consider supporting a mechanism to allow datanodes to drain outstanding work during rolling upgrade Sub-task Resolved Unassigned
         
        5.
        Make DN send an OOB Ack on shutdown before restarting Sub-task Resolved Kihwal Lee
         
        6.
        Add OOB upgrade response and client-side logic for reads Sub-task Resolved Unassigned
         
        7.
        Provide admin commands for data node upgrade Sub-task Resolved Kihwal Lee
         
        8.
        Add quick-restart option for datanode Sub-task Resolved Kihwal Lee
         
        9.
        Support upgrade marker in editlog streams Sub-task Resolved Tsz Wo Nicholas Sze
         
        10.
        Add a new DFSAdminCommand for rolling upgrade Sub-task Resolved Tsz Wo Nicholas Sze
         
        11.
        Support query and finalize rolling upgrade Sub-task Resolved Tsz Wo Nicholas Sze
         
        12.
        Add new NN startup options for downgrade and rollback using upgrade marker Sub-task Resolved Tsz Wo Nicholas Sze
         
        13.
        Split LayoutVerion into NamenodeLayoutVersion and DatanodeLayoutVersion Sub-task Resolved Brandon Li
         
        14.
        Document new commands and parameters for improved rolling upgrades Sub-task Resolved Tsz Wo Nicholas Sze
         
        15.
        Add a new option for starting standby NN when rolling upgrade is in progress Sub-task Resolved Tsz Wo Nicholas Sze
         
        16.
        Add rolling upgrade infomation to heartbeat response Sub-task Resolved Tsz Wo Nicholas Sze
         
        17.
        Reading edits should not stop at UpgradeMarker for normal restart of the namenode Sub-task Resolved Vinayakumar B
         
        18.
        When rolling upgrade is in progress, NN should only create checkpoint right before the upgrade marker Sub-task Resolved Tsz Wo Nicholas Sze
         
        19.
        Should not compare DataNode current layout version with that of NameNode in DataStrorage Sub-task Resolved Brandon Li
         
        20.
        When rolling upgrade is in progress, standby NN should create checkpoint for downgrade. Sub-task Resolved Tsz Wo Nicholas Sze
         
        21.
        Avoid NPE in Datanode heartbeat Sub-task Resolved Vinayakumar B
         
        22.
        Upgrade and rolling upgrade should not be allowed simultaneously Sub-task Resolved Tsz Wo Nicholas Sze
         
        23.
        BlockPoolSliceStorage trash to handle block deletions during rolling upgrade Sub-task Resolved Arpit Agarwal
         
        24.
        Support rollback of rolling upgrade in NameNode and JournalNodes Sub-task Resolved Jing Zhao
         
        25.
        Utilize OOB upgrade message processing for writes Sub-task Resolved Kihwal Lee
         
        26.
        Add rolling upgrade information to fsimage Sub-task Resolved Tsz Wo Nicholas Sze
         
        27.
        Merge Protobuf-based-FSImage code from trunk Sub-task Resolved Jing Zhao
         
        28.
        Fix TestRollingUpgrade Sub-task Resolved Arpit Agarwal
         
        29.
        TestRollingUpgrade#testSecondaryNameNode causes subsequent tests to fail Sub-task Resolved Tsz Wo Nicholas Sze
         
        30.
        Fix rollback of rolling upgrade in NameNode HA setup Sub-task Resolved Jing Zhao
         
        31.
        Fix rollback of rolling upgrade in NameNode HA setup Sub-task Resolved Jing Zhao
         
        32.
        Fix compilation error after merge Sub-task Resolved Tsz Wo Nicholas Sze
         
        33.
        Create unit tests for downgrade and finalize Sub-task Resolved Haohui Mai
         
        34.
        Rollback does not need to load edits Sub-task Resolved Jing Zhao
         
        35.
        Fix TestEditLog and TestStandbyCheckpoints Sub-task Resolved Jing Zhao
         
        36.
        SimulatedFSDataset#disableAndPurgeTrashStorage should not throw UnsupportedOperationException Sub-task Resolved Jing Zhao
         
        37.
        Fix findbugs warnings in Rolling Upgrade branch Sub-task Resolved Tsz Wo Nicholas Sze
         
        38.
        Fix TestOfflineEditsViewer and NPE in MD5FileUtils Sub-task Resolved Tsz Wo Nicholas Sze
         
        39.
        Fix TestDataNodeRollingUpgrade failure Sub-task Resolved Arpit Agarwal
         
        40.
        Do not create rollback fsimage when it already exists Sub-task Resolved Jing Zhao
         
        41.
        Avoid saving namespace when starting rolling upgrade Sub-task Resolved Jing Zhao
         
        42.
        Add '-rollingUpgrade <rollback|downgrade>' to namenode usage message Sub-task Resolved Vinayakumar B
         
        43.
        Change DFSAdmin for rolling upgrade commands Sub-task Resolved Tsz Wo Nicholas Sze
         
        44.
        Simplify Datanode rollback and downgrade Sub-task Resolved Suresh Srinivas
         
        45.
        add rollingUpgrade information to latest UI Sub-task Resolved Vinayakumar B
         
        46.
        Fix findbug warnings introduced by HDFS-5583 Sub-task Resolved Kihwal Lee
         
        47.
        Fix TestBlockRecovery#testRaceBetweenReplicaRecoveryAndFinalizeBlock Sub-task Resolved Kihwal Lee
         
        48.
        Update datanode replacement policy to make writes more robust Sub-task Resolved Kihwal Lee
         
        49.
        Query the status of rolling upgrade in the preparation stage in TestRollingUpgrade and TestRollingUpgradeRollback Sub-task Resolved Haohui Mai
         
        50.
        Standby NN might not checkpoint when processing the rolling upgrade marker Sub-task Resolved Haohui Mai
         
        51.
        Fix the five findbugs warnings Sub-task Resolved Kihwal Lee
         
        52.
        Test whether the standby NN continues to checkpoint after the prepare stage Sub-task Resolved Haohui Mai
         
        53.
        Test whether the NN will reject the downgrade if it has a fsimage from a newer release Sub-task Resolved Haohui Mai
         
        54.
        Fix TestDFSUpgrade and TestDataNodeRollingUpgrade Sub-task Resolved Jing Zhao
         
        55.
        Secondary NN fails to checkpoint after -rollingUpgrade prepare Sub-task Resolved Jing Zhao
         
        56.
        NN running newer software rejects loading the fsimage during rolling upgrade. Sub-task Resolved Tsz Wo Nicholas Sze
         
        57.
        -rollingUpgrade query hits NPE after the NN restarts Sub-task Resolved Haohui Mai
         
        58.
        DN registration should use DataNodeLayoutVersion instead of NameNodeLayoutVersion Sub-task Resolved Tsz Wo Nicholas Sze
         
        59.
        Allow JournalNode to handle editlog produced by new release with future layoutversion Sub-task Closed Jing Zhao
         
        60.
        Downgrade/Finalize should rename the rollback image instead of purging it Sub-task Resolved Jing Zhao
         
        61.
        Fix rolling upgrade documentation and error messges Sub-task Resolved Tsz Wo Nicholas Sze
         
        62.
        NameNode should not check DataNode layout version Sub-task Closed Brandon Li
         
        63.
        SimulatedDataSet should not create DatanodeRegistration with namenode layout version and type Sub-task Closed Tsz Wo Nicholas Sze
         

          Activity

          Nathan Roberts created issue -
          Kihwal Lee made changes -
          Field Original Value New Value
          Link This issue incorporates HDFS-5583 [ HDFS-5583 ]
          Kihwal Lee made changes -
          Link This issue incorporates HDFS-5583 [ HDFS-5583 ]
          Suresh Srinivas made changes -
          Attachment HDFSRollingUpgradesHighLevelDesign.pdf [ 12621151 ]
          Tsz Wo Nicholas Sze made changes -
          Attachment h5535_20140219.patch [ 12629993 ]
          Tsz Wo Nicholas Sze made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Tsz Wo Nicholas Sze made changes -
          Attachment h5535_20140220-1554.patch [ 12630196 ]
          Tsz Wo Nicholas Sze made changes -
          Attachment h5535_20140220b.patch [ 12630246 ]
          Tsz Wo Nicholas Sze made changes -
          Attachment h5535_20140221-2031.patch [ 12630456 ]
          Tsz Wo Nicholas Sze made changes -
          Attachment h5535_20140224-1931.patch [ 12630877 ]
          Tsz Wo Nicholas Sze made changes -
          Attachment h5535_20140225-1225.patch [ 12631032 ]
          Tsz Wo Nicholas Sze made changes -
          Attachment h5535_20140226-1256.patch [ 12631345 ]
          Haohui Mai made changes -
          Attachment hdfs-5535-test-plan.pdf [ 12631348 ]
          Tsz Wo Nicholas Sze made changes -
          Attachment h5535_20140226-1328.patch [ 12631351 ]
          Tsz Wo Nicholas Sze made changes -
          Attachment h5535_20140226-1256.patch [ 12631345 ]
          Tsz Wo Nicholas Sze made changes -
          Attachment h5535_20140226-1911.patch [ 12631445 ]
          Tsz Wo Nicholas Sze made changes -
          Attachment h5535_20140227-1239.patch [ 12631598 ]
          Tsz Wo Nicholas Sze made changes -
          Attachment h5535_20140228-1714.patch [ 12631886 ]
          Suresh Srinivas made changes -
          Tsz Wo Nicholas Sze made changes -
          Attachment h5535_20140304-1138.patch [ 12632582 ]
          Kihwal Lee made changes -
          Tsz Wo Nicholas Sze made changes -
          Attachment h5535_20140304-branch-2.patch [ 12633522 ]
          Tsz Wo Nicholas Sze made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Tsz Wo Nicholas Sze made changes -
          Attachment h5535_20140310-branch-2.patch [ 12633749 ]
          Tsz Wo Nicholas Sze made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Target Version/s 3.0.0 [ 12320356 ]
          Assignee Tsz Wo Nicholas Sze [ szetszwo ]
          Fix Version/s 2.4.0 [ 12326143 ]
          Resolution Fixed [ 1 ]
          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Jake Farrell made changes -
          Comment [ Dausey has used people to evaluate obsessive time shootings and systems between superior bean costs and their music neurons.
          http://www.surveyanalytics.com//userimages/sub-2/2007589/3153260/29851518/7787457-29851518-stopadd43.html
          Bands, from new york city, then practiced a wealth fact addition, only burnt on historical gatekeeper imprisonment. ]
          Colin Patrick McCabe made changes -
          Link This issue relates to HDFS-6800 [ HDFS-6800 ]
          Tsz Wo Nicholas Sze made changes -
          Link This issue is related to HDFS-7230 [ HDFS-7230 ]

            People

            • Assignee:
              Tsz Wo Nicholas Sze
              Reporter:
              Nathan Roberts
            • Votes:
              1 Vote for this issue
              Watchers:
              35 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development