Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.13.0
    • Fix Version/s: 0.14.0
    • Component/s: None
    • Labels:
      None

      Description

      Some data layout changes in HDFS require more than just a version upgrade introduced in HADOOP-702,
      because the cluster can function properly only when all components have upgraded, and the components
      need to communicate to each other and exchange data before they can perform the upgrade.
      The CRC upgrade discussed in HADOOP-1134 is one of such examples. Future enhancements like
      implementation of appends can change block meta-data and may require distributed upgrades.

      Distributed upgrade (DU) starts with a version upgrade (VU) so that at any time one could rollback
      all changes and start over.
      When VU is finished the name-node enters safe mode and persistently records that the DU have been started.
      It will also need to write a record when DU is finished. This is necessary to report unfinished upgrades in case
      of failure or for monitoring.

      The actual upgrade code from version vO to vN should be implemented in a separate UpgradeObject class,
      which implements interface Upgradeable.
      We create a new UpgradeObject for each pair of versions vO to vN that require a DU.
      We keep a (hard coded) table that determines which UpgradeObject(s) are applicable for the version pairs.
      Something like:

      Old version New version class names
      vO1 vN1 NameUpgradeObject1, DataUpgradeObject1
      vO2 vN2 NameUpgradeObject2, DataUpgradeObject2

      where vO1 < vN1 < vO2 < vN2 ...
      Now, if we need to upgrade from version version vX to version vY, we look for all pairs <vOi, vNi>
      in the table such that vX < vOi < vNi < vY and perform corresponding DUs one after another as they appear in the table.

      Each DU can and most probably should contain multiple UpgradeObjects.
      I'd define one object for the name-node and one for the data-nodes.
      The upgrade objects (in the same row) can communicate to each other either via existing protocols or using
      temporary protocols defined exclusively for this particular upgrade objects.
      I envision that some DUs will need to use old (vO) protocols to exchange the pre-upgrade data,
      and new (vN) protocols to reoport the upgraded data.

      UpgradeObjects should be able to bypass safe mode restrictions, be able to modify name-node data.

      1. Upgradeable.java
        2 kB
        Konstantin Shvachko
      2. DistUpgradeFramework6.patch
        66 kB
        Konstantin Shvachko

        Issue Links

          Activity

          Hide
          Konstantin Shvachko added a comment - - edited

          Some more ideas on the distributed upgrade.

          • Upgradeable interface
            All upgrade objects should implement this interface. It includes the following methods (see attached file for details):
            getName() returns component name that is to be upgraded by this object
            getDescription() description of the object for displaying
            versionFrom() which version does the object upgrade
            versionTo() which version does the object upgrade to
            static newInstance() creates new instance of this object
            getUpgradeStatus() - returns a value in [0,100]
            UpgradeCommand processUpgradeCommand(UpgradeCommand comm)
          • Upgrade object is a class that implements Upgradeable interface
            Naming conventions for the upgrade objects
            Upgrade_<version>_<component name>
            E.g. Upgrade_5_NameNode
          • UpgradeObjectCollection
            A class that statically lists versions and their upgrade objects.
            One can get a sequence of upgrade objects given the component name and two versions.
            <componentName, verStart, verEnd> ==> <UpgradeObject(verStart), ...., UpgradeObject(verEnd)>
          • UpgradeManager class should detect which upgrade objects are required to perform the upgrade,
            and applies the upgrades in the correct sequence.
          • UpgradeCommand
            ClientProtocol and DatanodeProtocol should be extended with one extra call
            UpgradeCommand processUpgradeCommand(UpgradeCommand comm)
            This command calls a respective procedure on current upgrade object.
            The object itself should recognize the actions to be performed and perform them.

          The attached file contains the Upgradeable interface and the UpgradeCommand base class,
          which in real life should of course be in separate java files.

          Show
          Konstantin Shvachko added a comment - - edited Some more ideas on the distributed upgrade. Upgradeable interface All upgrade objects should implement this interface. It includes the following methods (see attached file for details): getName() returns component name that is to be upgraded by this object getDescription() description of the object for displaying versionFrom() which version does the object upgrade versionTo() which version does the object upgrade to static newInstance() creates new instance of this object getUpgradeStatus() - returns a value in [0,100] UpgradeCommand processUpgradeCommand(UpgradeCommand comm) Upgrade object is a class that implements Upgradeable interface Naming conventions for the upgrade objects Upgrade_<version>_<component name> E.g. Upgrade_5_NameNode UpgradeObjectCollection A class that statically lists versions and their upgrade objects. One can get a sequence of upgrade objects given the component name and two versions. <componentName, verStart, verEnd> ==> <UpgradeObject(verStart), ...., UpgradeObject(verEnd)> UpgradeManager class should detect which upgrade objects are required to perform the upgrade, and applies the upgrades in the correct sequence. UpgradeCommand ClientProtocol and DatanodeProtocol should be extended with one extra call UpgradeCommand processUpgradeCommand(UpgradeCommand comm) This command calls a respective procedure on current upgrade object. The object itself should recognize the actions to be performed and perform them. The attached file contains the Upgradeable interface and the UpgradeCommand base class, which in real life should of course be in separate java files.
          Hide
          Raghu Angadi added a comment -

          Thanks Konstantin. I will use this interface for Block CRC upgrade as part of HADOOP-1134.

          Show
          Raghu Angadi added a comment - Thanks Konstantin. I will use this interface for Block CRC upgrade as part of HADOOP-1134 .
          Hide
          Konstantin Shvachko added a comment - - edited
          • UpgradeManager manager determines which upgrade objects are required for the distributed upgrade.
            Each upgrade object performs necessary upgrade until done and returns control to the upgrade manager.
          • ClientProtocol needs to have a method that returns current status of the distributed upgrade, which should be
            reported by DFSAdmin.report().
          • Only the DatanodeProtocol should be extended with processUpgradeCommand().
            ClientProtocol protocol should not have it. It should always obey the safe mode restrictions.
          • Each UpgradeObject should correspond to a specific version rather than to a new-old version pair as stated before.
            So the UpgradeObjectCollection table rather looks as
            Version class names
            v1 NameUpgradeObject1, DataUpgradeObject1
            v2 NameUpgradeObject2, DataUpgradeObject2

            where v1 < v2 < ...
            If we need to upgrade from vX to vY we find all versions vi such that vX < vi < vY and perform corresponding
            upgrades for each of them in that order.
            This particularly means that Upgradeable interface should have just one method getVersion() instead of the two
            previously declared versionFrom() and versionTo().

          Show
          Konstantin Shvachko added a comment - - edited UpgradeManager manager determines which upgrade objects are required for the distributed upgrade. Each upgrade object performs necessary upgrade until done and returns control to the upgrade manager. ClientProtocol needs to have a method that returns current status of the distributed upgrade, which should be reported by DFSAdmin.report(). Only the DatanodeProtocol should be extended with processUpgradeCommand(). ClientProtocol protocol should not have it. It should always obey the safe mode restrictions. Each UpgradeObject should correspond to a specific version rather than to a new-old version pair as stated before. So the UpgradeObjectCollection table rather looks as Version class names v1 NameUpgradeObject1, DataUpgradeObject1 v2 NameUpgradeObject2, DataUpgradeObject2 where v1 < v2 < ... If we need to upgrade from vX to vY we find all versions vi such that vX < vi < vY and perform corresponding upgrades for each of them in that order. This particularly means that Upgradeable interface should have just one method getVersion() instead of the two previously declared versionFrom() and versionTo().
          Hide
          Konstantin Shvachko added a comment -

          This is the distributed upgrade framework implementation.

          In order to define a dsitributed upgrade you need

          1. extend respective UgradeObject[Datanode | Namenode] classes
          2. register these objects in UpgradeObjectCollection
          3. define UpgradeCommand-s for the communication related to the upgrade
          4. start cluster in upgrade mode

          An example can be found in TestDistributedUpgrade.
          Please review

          Show
          Konstantin Shvachko added a comment - This is the distributed upgrade framework implementation. In order to define a dsitributed upgrade you need extend respective UgradeObject [Datanode | Namenode] classes register these objects in UpgradeObjectCollection define UpgradeCommand-s for the communication related to the upgrade start cluster in upgrade mode An example can be found in TestDistributedUpgrade. Please review
          Hide
          Raghu Angadi added a comment -

          Konstantin, could you define what getVersion() should return? In the case of Block Level CRCs, upgrade happens while going from version '-5' to '-6'. Should it return -5 or -6?

          Also 'vX < vi < vY' should be either 'vX <= vi < vY' or 'vX < vi <= vY' depeding on contract of getVersion().

          Show
          Raghu Angadi added a comment - Konstantin, could you define what getVersion() should return? In the case of Block Level CRCs, upgrade happens while going from version '-5' to '-6'. Should it return -5 or -6? Also 'vX < vi < vY' should be either 'vX <= vi < vY' or 'vX < vi <= vY' depeding on contract of getVersion().
          Hide
          Konstantin Shvachko added a comment -

          This patch contains more javaDoc comments and the test covers more cases.

          Show
          Konstantin Shvachko added a comment - This patch contains more javaDoc comments and the test covers more cases.
          Hide
          Hadoop QA added a comment -
          Show
          Hadoop QA added a comment - +0, new Findbugs warnings http://issues.apache.org/jira/secure/attachment/12360956/DistUpgradeFramework.patch applied and successfully tested against trunk revision r552548, but there appear to be new Findbugs warnings introduced by this patch. New Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/353/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/353/testReport/ Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/353/console
          Hide
          Konstantin Shvachko added a comment -

          Fixed Findbugs warning. Implemented equals() as required for Comparable interface.

          Show
          Konstantin Shvachko added a comment - Fixed Findbugs warning. Implemented equals() as required for Comparable interface.
          Hide
          Hadoop QA added a comment -
          Show
          Hadoop QA added a comment - +0, new Findbugs warnings http://issues.apache.org/jira/secure/attachment/12360979/DistUpgradeFramework3.patch applied and successfully tested against trunk revision r552548, but there appear to be new Findbugs warnings introduced by this patch. New Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/355/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/355/testReport/ Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/355/console
          Hide
          Konstantin Shvachko added a comment -

          FindBugs warnings fixed.

          Show
          Konstantin Shvachko added a comment - FindBugs warnings fixed.
          Hide
          Raghu Angadi added a comment -

          UpgradeManager seems to invoke upgradeobject.processUpgradeCommand() inside synchronized. This will serialize all upgrade related RPCs. Its probably ok now but the framework should not enforce this serialization, objects can have some commands that can be excecuted in parallel. For Block Level CRCs, command that reports status is independent of command that requests crc info for a block.

          Show
          Raghu Angadi added a comment - UpgradeManager seems to invoke upgradeobject.processUpgradeCommand() inside synchronized. This will serialize all upgrade related RPCs. Its probably ok now but the framework should not enforce this serialization, objects can have some commands that can be excecuted in parallel. For Block Level CRCs, command that reports status is independent of command that requests crc info for a block.
          Hide
          Hadoop QA added a comment -
          Show
          Hadoop QA added a comment - +0, new Findbugs warnings http://issues.apache.org/jira/secure/attachment/12360984/DistUpgradeFramework4.patch applied and successfully tested against trunk revision r552548, but there appear to be new Findbugs warnings introduced by this patch. New Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/357/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/357/testReport/ Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/357/console
          Hide
          Konstantin Shvachko added a comment -

          Another findBugs warning fixed.
          Plus this patch is consistent with HADOOP-1134.
          Some functionality was added to support cases like when data-nodes complete upgrade
          and need to report to the name-node again that it is done while other upgrades are in progress.
          Raghu please confirm that it is in synch with your patch.

          Show
          Konstantin Shvachko added a comment - Another findBugs warning fixed. Plus this patch is consistent with HADOOP-1134 . Some functionality was added to support cases like when data-nodes complete upgrade and need to report to the name-node again that it is done while other upgrades are in progress. Raghu please confirm that it is in synch with your patch.
          Hide
          Raghu Angadi added a comment -

          > UpgradeManager seems to invoke upgradeobject.processUpgradeCommand() inside synchronized.

          This can affect crc upgrade noticeably in one case. I am adding a dfsadmin command that lets admin to force update of block level stats (as an option), which involves iterating over all the blocks. If there are a few million blocks, cluster wide upgrade can stall for a few minutes.

          Show
          Raghu Angadi added a comment - > UpgradeManager seems to invoke upgradeobject.processUpgradeCommand() inside synchronized. This can affect crc upgrade noticeably in one case. I am adding a dfsadmin command that lets admin to force update of block level stats (as an option), which involves iterating over all the blocks. If there are a few million blocks, cluster wide upgrade can stall for a few minutes.
          Hide
          Konstantin Shvachko added a comment -

          A new patch
          1. We also need to reject data-nodes that missed a distributed upgrade.
          The data-node now detects that send an error message to the name-node and shuts down.
          2. Fixed some more FindBugs.
          3. Fixed HADOOP-1567 since build version consistency is important for upgrades.

          Show
          Konstantin Shvachko added a comment - A new patch 1. We also need to reject data-nodes that missed a distributed upgrade. The data-node now detects that send an error message to the name-node and shuts down. 2. Fixed some more FindBugs. 3. Fixed HADOOP-1567 since build version consistency is important for upgrades.
          Hide
          Hadoop QA added a comment -

          -1, new javadoc warnings

          The javadoc tool appears to have generated warning messages when testing the latest attachment http://issues.apache.org/jira/secure/attachment/12361244/DistUpgradeFramework6.patch against trunk revision r553623.

          Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/367/testReport/
          Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/367/console

          Please note that this message is automatically generated and may represent a problem with the automation system and not the patch.

          Show
          Hadoop QA added a comment - -1, new javadoc warnings The javadoc tool appears to have generated warning messages when testing the latest attachment http://issues.apache.org/jira/secure/attachment/12361244/DistUpgradeFramework6.patch against trunk revision r553623. Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/367/testReport/ Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/367/console Please note that this message is automatically generated and may represent a problem with the automation system and not the patch.
          Hide
          Konstantin Shvachko added a comment -

          JavaDoc warning fixed

          Show
          Konstantin Shvachko added a comment - JavaDoc warning fixed
          Show
          Hadoop QA added a comment - +1 http://issues.apache.org/jira/secure/attachment/12361252/DistUpgradeFramework6.patch applied and successfully tested against trunk revision r553623. Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/368/testReport/ Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/368/console
          Hide
          Doug Cutting added a comment -

          I just committed this. Thanks, Konstantin!

          Show
          Doug Cutting added a comment - I just committed this. Thanks, Konstantin!
          Hide
          Hudson added a comment -
          Show
          Hudson added a comment - Integrated in Hadoop-Nightly #152 (See http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Nightly/152/ )

            People

            • Assignee:
              Konstantin Shvachko
              Reporter:
              Konstantin Shvachko
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development