Hadoop Common
  1. Hadoop Common
  2. HADOOP-2656

Support for upgrading existing cluster to facilitate appends to HDFS files

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.18.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Associated a generation stamp with each block. On data nodes, the generation stamp is stored as part of the file name of the block's meta-data file.

      Description

      HADOOP-1700 describes the design for supporting appends to HDFS files. This design requires a distributed-upgrade to existing cluster installations. The design specifies that the DataNode persist the 8-byte BlockGenerationStamp in the block metadata file. The upgrade code will introduce this new field in the block metadata file and initialize this value to 0.

      1. upgradeGenStamp12.patch
        107 kB
        dhruba borthakur
      2. upgradeGenStamp11.patch
        108 kB
        dhruba borthakur
      3. upgradeGenStamp10.patch
        108 kB
        dhruba borthakur
      4. upgradeGenStamp9.patch
        111 kB
        dhruba borthakur
      5. upgradeGenStamp9.patch
        110 kB
        dhruba borthakur
      6. upgradeGenStamp8.patch
        110 kB
        dhruba borthakur
      7. upgradeGenStamp7.patch
        110 kB
        dhruba borthakur
      8. upgradeGenStamp6.patch
        109 kB
        dhruba borthakur
      9. upgradeGenStamp.patch
        44 kB
        dhruba borthakur
      10. upgradeGenStamp5.patch
        109 kB
        dhruba borthakur
      11. upgradeGenStamp4.patch
        98 kB
        dhruba borthakur

        Issue Links

          Activity

          Hide
          dhruba borthakur added a comment -

          The Datanode needs to store a block generation stamp for each block. The original idea was to store the block generation stamp inside the meta file of each block. One major disadvantage of this approach is that generation of a block report requires that each meta file be opened and the generation stamp read from it. With about 50K blocks per datanode and a seek time of 10 ms, this might require 300 seconds. This means that the time to restart a cluster cannot be lesser than this 300 seconds.

          A few other alternatives:
          1. Encode the generation stamp into the name of the metafile. Each metafile will look like blkxxxxxx.genstamp.meta. The block file will remain the same.

          2. Encode the generation stamp into the name of the block file. Each block file will be of the form blkxxxxxx.genstamp. The metafile will remain the same.

          3. Encode the generation stamp into the name of a new zero-size file named blkxxxxx.genstamp. The block file and the metadata file will remain the same.

          4. A completely separate file (one per datanode) that records the metadata of all blocks in the datanode.

          I propose that we implement option 1.

          Show
          dhruba borthakur added a comment - The Datanode needs to store a block generation stamp for each block. The original idea was to store the block generation stamp inside the meta file of each block. One major disadvantage of this approach is that generation of a block report requires that each meta file be opened and the generation stamp read from it. With about 50K blocks per datanode and a seek time of 10 ms, this might require 300 seconds. This means that the time to restart a cluster cannot be lesser than this 300 seconds. A few other alternatives: 1. Encode the generation stamp into the name of the metafile. Each metafile will look like blkxxxxxx.genstamp.meta. The block file will remain the same. 2. Encode the generation stamp into the name of the block file. Each block file will be of the form blkxxxxxx.genstamp. The metafile will remain the same. 3. Encode the generation stamp into the name of a new zero-size file named blkxxxxx.genstamp. The block file and the metadata file will remain the same. 4. A completely separate file (one per datanode) that records the metadata of all blocks in the datanode. I propose that we implement option 1.
          Hide
          dhruba borthakur added a comment -

          This patch does the following:

          1. Upgrades existing clusters to the new disk format. Each Block has a generation stamp associated with it. Existing blocks get a generation stamp of 0. The generation stamp is used to create the name of the block metafile on the datanode.

          2. The Block object has a new field called "generationStamp" of type "long". The BlocksMap on the namenode is keyed on the blockid and the generation stamp.

          3. The datanode sends the generation stamp, block id and size of each block in a block report.

          3. All log statements that print the blockid now prints the blockid and generationstamp.

          4. The client receives the blockid and generation stamp as part of an RPC that receives a block.

          5. The DataTransferProtocol sends the blockid and generation stamp with every connection request that the client makes to the datanode(s).

          Show
          dhruba borthakur added a comment - This patch does the following: 1. Upgrades existing clusters to the new disk format. Each Block has a generation stamp associated with it. Existing blocks get a generation stamp of 0. The generation stamp is used to create the name of the block metafile on the datanode. 2. The Block object has a new field called "generationStamp" of type "long". The BlocksMap on the namenode is keyed on the blockid and the generation stamp. 3. The datanode sends the generation stamp, block id and size of each block in a block report. 3. All log statements that print the blockid now prints the blockid and generationstamp. 4. The client receives the blockid and generation stamp as part of an RPC that receives a block. 5. The DataTransferProtocol sends the blockid and generation stamp with every connection request that the client makes to the datanode(s).
          Hide
          Tsz Wo Nicholas Sze added a comment -

          upgradeGenStamp4.patch :

          • In Block.isBlockFilename(File f), it disallows file name containing "." but "." is used in Block.toString() in the new codes.
          • A constant BLOCK_FILE_PREFIX="blk_" is defined in DataStorage but not is used in other part of the codes.
          • Another METADATA_EXTENSION=".meta" is defined in FSDataset, you should not define a new one in BlockMetadata

          We have to be careful about handling the filenames for blocks and meta data. It would be better if we put all the codes in one place.

          Show
          Tsz Wo Nicholas Sze added a comment - upgradeGenStamp4.patch : In Block.isBlockFilename(File f), it disallows file name containing "." but "." is used in Block.toString() in the new codes. A constant BLOCK_FILE_PREFIX="blk_" is defined in DataStorage but not is used in other part of the codes. Another METADATA_EXTENSION=".meta" is defined in FSDataset, you should not define a new one in BlockMetadata We have to be careful about handling the filenames for blocks and meta data. It would be better if we put all the codes in one place.
          Hide
          dhruba borthakur added a comment -

          This patch incorporates the feedback given by Nicholas.

          Show
          dhruba borthakur added a comment - This patch incorporates the feedback given by Nicholas.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12380249/upgradeGenStamp5.patch
          against trunk revision 645773.

          @author +1. The patch does not contain any @author tags.

          tests included +1. The patch appears to include 27 new or modified tests.

          javadoc +1. The javadoc tool did not generate any warning messages.

          javac +1. The applied patch does not generate any new javac compiler warnings.

          release audit +1. The applied patch does not generate any new release audit warnings.

          findbugs -1. The patch appears to cause Findbugs to fail.

          core tests -1. The patch failed core unit tests.

          contrib tests -1. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2252/testReport/
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2252/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2252/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12380249/upgradeGenStamp5.patch against trunk revision 645773. @author +1. The patch does not contain any @author tags. tests included +1. The patch appears to include 27 new or modified tests. javadoc +1. The javadoc tool did not generate any warning messages. javac +1. The applied patch does not generate any new javac compiler warnings. release audit +1. The applied patch does not generate any new release audit warnings. findbugs -1. The patch appears to cause Findbugs to fail. core tests -1. The patch failed core unit tests. contrib tests -1. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2252/testReport/ Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2252/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2252/console This message is automatically generated.
          Hide
          Konstantin Shvachko added a comment -

          Am I correct that only the last block in a file need to have generation stamp?
          In that case generationStamp should be a member of INodeFile rather than Block.
          It should also be a part LocatedBlock so that clients had the same api to work with blocks.

          Show
          Konstantin Shvachko added a comment - Am I correct that only the last block in a file need to have generation stamp? In that case generationStamp should be a member of INodeFile rather than Block. It should also be a part LocatedBlock so that clients had the same api to work with blocks.
          Hide
          Konstantin Shvachko added a comment - - edited

          BlockMetadata.

          1. 3 redundant imports
          2. HEADER_BUF_SIZE is never read locally
          3. upgradeToCurVersion() should be a part of GenerationStampUpgrade
            also there is problem with indentations.
          4. It is better to place all upgrade-related code into GenerationStampUpgrade.
            Say, Header.needsUpgrade() should be static needsUpgrade(Header).
          5. not sure all 3 variants of readHeader() are needed. At least one is not called anywhere.
          6. I propose to restructure this class in the following way:
            make BlockMetadata.Header class the top level class rather than the inner, rename it to BlockMetadataHeader.
            In other words this means getting rid of the inner class Header, moving all its contents up to BlockMetadata,
            and then renaming the latter.
          7. line 160: should not use deprecated getMetaFile() method. You will need old meta data file names
            as long as the upgrade exists, so it is better to have a getPreGenerationMetaFile() or smth. in the
            corresponding upgrade object.

          Datanode

          1. New comment about the return value in startDistributedUpgradeIfNeedes() is wrong, pls remove.
          2. Starting block scanner in offerService() you should check that the scaner has already been started,
            otherwise you will be starting it every hour after the upgrade is finished.

          DatanodeBlockInfo

          • In detachBlock() can you use static getMetaFile(File, Block) rather than passing FSDataset into it.

          DataBlockScanner

          • assignInitialVerificationTimes() should use tmpBlock.set() rather then assigning fields directly.

          FSDataset

          • Deprecated getMetaFile() the one that is "used only by Upgrade code" is not public and could be
            moved directly to the upgrade code rather than deprecated.

          FSEditLog

          1. In loadFSEdit() line "if (logVersion < -13)" should read as
            if (logVersion <= -13)
            
          2. Introduction of BlockTwo for reading old versions of block layout is kinda confusing.
            I'd rather explicitly read the array length and then each of the two fields in a loop,
            which you have anyway since the prehistoric generation stamp needs to be assigned.

          FSImage

          • line "if (-14 < imgVersion)" should read
            if (imgVersion <= -13)
            

            swap the if and else bodies.

          FSNamesystem
          NameNode
          NamenodeFsck
          UnderReplicatedBlocks

          • This is all mostly about replacing block.getBlockName() with block.toString().

          UpgradeManager

          boolean isUpgradeCompleted() {
            return currentUpgrade == null;
          }
          

          GenerationStampUpgrade

          1. same as in BlockMetadata(7) in line 260.
          2. ";;" remove one
          3. There is a lot of common code in BlockCrcUpgrade and GenerationStampUpgrade.
            Many classes and types are literally (past&copy) the same:
            • UpgradeStatus (with the STARTED field never used)
            • DNInfo
            • BlockLevelStats
            • GenerationStampUpgradeStatusReport = BlockCrcUpgradeStatusReport
            • DatanodeStatsCommand
            • I probably missed some.
          4. We should make them all static classes and move either to the base classes
            if the actions are common and useful for other generations of upgrades or create
            UpgradeUtils class and keep them there.
          5. The important question is how do we test the generation stamp upgrade?
            With crcs we had a test plan and were testing different scenarios of failure of the upgrade at different stages.
            We should either do the same for genStamps or have a unit test, that tests the scenarios automatically.

          ClientProtocol
          The ClientProtocol version should be set to 30 (not just the comment).

          General

          1. Please look for empty line changes in the final patch.
          2. Please replace /* */ comments with /** */ for classes, class methods, members, and constants.
          Show
          Konstantin Shvachko added a comment - - edited BlockMetadata. 3 redundant imports HEADER_BUF_SIZE is never read locally upgradeToCurVersion() should be a part of GenerationStampUpgrade also there is problem with indentations. It is better to place all upgrade-related code into GenerationStampUpgrade. Say, Header.needsUpgrade() should be static needsUpgrade(Header). not sure all 3 variants of readHeader() are needed. At least one is not called anywhere. I propose to restructure this class in the following way: make BlockMetadata.Header class the top level class rather than the inner, rename it to BlockMetadataHeader. In other words this means getting rid of the inner class Header, moving all its contents up to BlockMetadata, and then renaming the latter. line 160: should not use deprecated getMetaFile() method. You will need old meta data file names as long as the upgrade exists, so it is better to have a getPreGenerationMetaFile() or smth. in the corresponding upgrade object. Datanode New comment about the return value in startDistributedUpgradeIfNeedes() is wrong, pls remove. Starting block scanner in offerService() you should check that the scaner has already been started, otherwise you will be starting it every hour after the upgrade is finished. DatanodeBlockInfo In detachBlock() can you use static getMetaFile(File, Block) rather than passing FSDataset into it. DataBlockScanner assignInitialVerificationTimes() should use tmpBlock.set() rather then assigning fields directly. FSDataset Deprecated getMetaFile() the one that is "used only by Upgrade code" is not public and could be moved directly to the upgrade code rather than deprecated. FSEditLog In loadFSEdit() line "if (logVersion < -13)" should read as if (logVersion <= -13) Introduction of BlockTwo for reading old versions of block layout is kinda confusing. I'd rather explicitly read the array length and then each of the two fields in a loop, which you have anyway since the prehistoric generation stamp needs to be assigned. FSImage line "if (-14 < imgVersion)" should read if (imgVersion <= -13) swap the if and else bodies. FSNamesystem NameNode NamenodeFsck UnderReplicatedBlocks This is all mostly about replacing block.getBlockName() with block.toString(). UpgradeManager boolean isUpgradeCompleted() { return currentUpgrade == null ; } GenerationStampUpgrade same as in BlockMetadata(7) in line 260. ";;" remove one There is a lot of common code in BlockCrcUpgrade and GenerationStampUpgrade. Many classes and types are literally (past&copy) the same: UpgradeStatus (with the STARTED field never used) DNInfo BlockLevelStats GenerationStampUpgradeStatusReport = BlockCrcUpgradeStatusReport DatanodeStatsCommand I probably missed some. We should make them all static classes and move either to the base classes if the actions are common and useful for other generations of upgrades or create UpgradeUtils class and keep them there. The important question is how do we test the generation stamp upgrade? With crcs we had a test plan and were testing different scenarios of failure of the upgrade at different stages. We should either do the same for genStamps or have a unit test, that tests the scenarios automatically. ClientProtocol The ClientProtocol version should be set to 30 (not just the comment). General Please look for empty line changes in the final patch. Please replace /* */ comments with /** */ for classes, class methods, members, and constants.
          Hide
          Konstantin Shvachko added a comment -

          May be this is a good time to return back to the question of renaming hdfs blocks, stop generating block ids randomly, and replace it with sequentially generated ids.
          This is related to my previous question whether (in the name-node) we need to store block generation stamp for each block or only for the last block of each file. The only problem here is with prehistoric (according to Dhruba, HADOOP-1700, HADOOP-146, HADOOP-158) blocks.

          Reminder: a block is prehistoric if it is reported to the system after its id was reassigned to another physical block.
          This happens when a data-node is down for a rather long period of time during which 2 things happen:

          1. a block that it owns is removed and then
          2. a new block is created with the same block id.

          This leads to data corruption, but this can be avoided if block ids are generated sequentially rather than randomly.
          AFAIR, the only reason for not converting to sequential ids was that we were afraid of getting into a rather massive (distributed) upgrade that renames all blocks in the system. This patch leads exactly to such an upgrade, and I think it is the right time to adopt the right id generation practice.
          The advantage here is that we will be able to store a (smaller) generation stamp (if any) per file rather than per block.

          Show
          Konstantin Shvachko added a comment - May be this is a good time to return back to the question of renaming hdfs blocks, stop generating block ids randomly, and replace it with sequentially generated ids. This is related to my previous question whether (in the name-node) we need to store block generation stamp for each block or only for the last block of each file. The only problem here is with prehistoric (according to Dhruba, HADOOP-1700 , HADOOP-146 , HADOOP-158 ) blocks. Reminder: a block is prehistoric if it is reported to the system after its id was reassigned to another physical block. This happens when a data-node is down for a rather long period of time during which 2 things happen: a block that it owns is removed and then a new block is created with the same block id. This leads to data corruption, but this can be avoided if block ids are generated sequentially rather than randomly. AFAIR, the only reason for not converting to sequential ids was that we were afraid of getting into a rather massive (distributed) upgrade that renames all blocks in the system. This patch leads exactly to such an upgrade, and I think it is the right time to adopt the right id generation practice. The advantage here is that we will be able to store a (smaller) generation stamp (if any) per file rather than per block.
          Hide
          dhruba borthakur added a comment -

          Incorporates most of Konstantin's comments. I left the definition of BlockTwo just because I think it is cleaner this way. Also, the reason some code is duplicated between BlockCrcUpgrade and BlockGenerationStampUpgrade is because the BlockCrcUpgrade code might probably go away in the next release or so.

          Show
          dhruba borthakur added a comment - Incorporates most of Konstantin's comments. I left the definition of BlockTwo just because I think it is cleaner this way. Also, the reason some code is duplicated between BlockCrcUpgrade and BlockGenerationStampUpgrade is because the BlockCrcUpgrade code might probably go away in the next release or so.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12380642/upgradeGenStamp6.patch
          against trunk revision 645773.

          @author +1. The patch does not contain any @author tags.

          tests included +1. The patch appears to include 27 new or modified tests.

          javadoc +1. The javadoc tool did not generate any warning messages.

          javac +1. The applied patch does not generate any new javac compiler warnings.

          release audit +1. The applied patch does not generate any new release audit warnings.

          findbugs -1. The patch appears to introduce 10 new Findbugs warnings.

          core tests +1. The patch passed core unit tests.

          contrib tests +1. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2296/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2296/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2296/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2296/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12380642/upgradeGenStamp6.patch against trunk revision 645773. @author +1. The patch does not contain any @author tags. tests included +1. The patch appears to include 27 new or modified tests. javadoc +1. The javadoc tool did not generate any warning messages. javac +1. The applied patch does not generate any new javac compiler warnings. release audit +1. The applied patch does not generate any new release audit warnings. findbugs -1. The patch appears to introduce 10 new Findbugs warnings. core tests +1. The patch passed core unit tests. contrib tests +1. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2296/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2296/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2296/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2296/console This message is automatically generated.
          Hide
          dhruba borthakur added a comment -

          Fix findbugs warnings.

          Show
          dhruba borthakur added a comment - Fix findbugs warnings.
          Hide
          dhruba borthakur added a comment -

          Fix findbugs warnings.

          Show
          dhruba borthakur added a comment - Fix findbugs warnings.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12380784/upgradeGenStamp7.patch
          against trunk revision 645773.

          @author +1. The patch does not contain any @author tags.

          tests included +1. The patch appears to include 27 new or modified tests.

          javadoc +1. The javadoc tool did not generate any warning messages.

          javac +1. The applied patch does not generate any new javac compiler warnings.

          release audit +1. The applied patch does not generate any new release audit warnings.

          findbugs -1. The patch appears to introduce 1 new Findbugs warnings.

          core tests +1. The patch passed core unit tests.

          contrib tests +1. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2310/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2310/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2310/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2310/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12380784/upgradeGenStamp7.patch against trunk revision 645773. @author +1. The patch does not contain any @author tags. tests included +1. The patch appears to include 27 new or modified tests. javadoc +1. The javadoc tool did not generate any warning messages. javac +1. The applied patch does not generate any new javac compiler warnings. release audit +1. The applied patch does not generate any new release audit warnings. findbugs -1. The patch appears to introduce 1 new Findbugs warnings. core tests +1. The patch passed core unit tests. contrib tests +1. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2310/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2310/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2310/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2310/console This message is automatically generated.
          Hide
          dhruba borthakur added a comment -

          Fixed the last (hopefully) findbugs warnings.

          Show
          dhruba borthakur added a comment - Fixed the last (hopefully) findbugs warnings.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12380799/upgradeGenStamp8.patch
          against trunk revision 645773.

          @author +1. The patch does not contain any @author tags.

          tests included +1. The patch appears to include 27 new or modified tests.

          javadoc +1. The javadoc tool did not generate any warning messages.

          javac +1. The applied patch does not generate any new javac compiler warnings.

          release audit +1. The applied patch does not generate any new release audit warnings.

          findbugs +1. The patch does not introduce any new Findbugs warnings.

          core tests +1. The patch passed core unit tests.

          contrib tests +1. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2313/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2313/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2313/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2313/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12380799/upgradeGenStamp8.patch against trunk revision 645773. @author +1. The patch does not contain any @author tags. tests included +1. The patch appears to include 27 new or modified tests. javadoc +1. The javadoc tool did not generate any warning messages. javac +1. The applied patch does not generate any new javac compiler warnings. release audit +1. The applied patch does not generate any new release audit warnings. findbugs +1. The patch does not introduce any new Findbugs warnings. core tests +1. The patch passed core unit tests. contrib tests +1. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2313/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2313/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2313/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2313/console This message is automatically generated.
          Hide
          dhruba borthakur added a comment -

          Merged patch with latest trunk.

          Show
          dhruba borthakur added a comment - Merged patch with latest trunk.
          Hide
          dhruba borthakur added a comment -

          merged path with latest trunk.

          Show
          dhruba borthakur added a comment - merged path with latest trunk.
          Hide
          dhruba borthakur added a comment -

          Here is a Test Plan for testing Distributed Upgrades for Block Generation Stamp

          Test Cases

          T1 Install build version 0.17 and create a new HDFS cluster of size 500 nodes.. Run random writer, sort, and then sort validation. Then shutdown cluster. Install version 0.18 build on the cluster. Start cluster with the -upgrade option.
          Expected behavior: The Namenode should trigger an upgrade on all Datanodes. Then the Namenode should exit safe mode automatically. Run sort validation again to verify that data is intact.

          T2 Start a DFS Upgrade and restart Namenode during the upgrade.
          Expected Behaviour: Upgrade should proceed normally with appropriate messages in Namenode log

          T3 Start a DFS upgrade and restart some of the Datanodes while the upgrade is occurring.
          Expected behaviour: Upgrade should proceed normally with appropriate messages in Datenode logs.

          T4 Start an upgrade and stop two Datanodes.
          Expected Behavior: Upgrade should complete successfully because at least one replica of every block should still be available.

          T4 Start an upgrade and stop half of the Datanodes.
          Expected Behavior: Upgrade should stall because at least one replica of every block would not be available. Verify that useful message is displayed in the Namenode logs.

          T5 Start a DFS upgrade and let it complete successfully. Then issue the admin command to finalize the upgrade.
          Excepted Behavior: The Namenode should be able to finalize the cluster.

          T5 Same as T4. Once the upgrade completes successfully, bring up the two Datanodes that were shut down earlier.
          Expected behavior: These two Datanodes should start their own upgrades. Note that the Namenode could have already replicated these blocks.

          T6 Same at T1 but with more aggressive periodic block scanner by setting dfs.datanode.scan.period.hours to 1.
          Expected Behavior: Upgrade should succeed. Look at the Datanode log file to verify that no bad blocks were found by the periodic block scanner.

          T7 Same as T1. Before starting the upgrade, log into a Datanode and delete a particular blk_xxx.meta file. Then start the DFS upgrade.
          Expected Behavior: Look at the Namenode log to detect that the upgrade failed on that Datanode.

          Show
          dhruba borthakur added a comment - Here is a Test Plan for testing Distributed Upgrades for Block Generation Stamp Test Cases T1 Install build version 0.17 and create a new HDFS cluster of size 500 nodes.. Run random writer, sort, and then sort validation. Then shutdown cluster. Install version 0.18 build on the cluster. Start cluster with the -upgrade option. Expected behavior: The Namenode should trigger an upgrade on all Datanodes. Then the Namenode should exit safe mode automatically. Run sort validation again to verify that data is intact. T2 Start a DFS Upgrade and restart Namenode during the upgrade. Expected Behaviour: Upgrade should proceed normally with appropriate messages in Namenode log T3 Start a DFS upgrade and restart some of the Datanodes while the upgrade is occurring. Expected behaviour: Upgrade should proceed normally with appropriate messages in Datenode logs. T4 Start an upgrade and stop two Datanodes. Expected Behavior: Upgrade should complete successfully because at least one replica of every block should still be available. T4 Start an upgrade and stop half of the Datanodes. Expected Behavior: Upgrade should stall because at least one replica of every block would not be available. Verify that useful message is displayed in the Namenode logs. T5 Start a DFS upgrade and let it complete successfully. Then issue the admin command to finalize the upgrade. Excepted Behavior: The Namenode should be able to finalize the cluster. T5 Same as T4. Once the upgrade completes successfully, bring up the two Datanodes that were shut down earlier. Expected behavior: These two Datanodes should start their own upgrades. Note that the Namenode could have already replicated these blocks. T6 Same at T1 but with more aggressive periodic block scanner by setting dfs.datanode.scan.period.hours to 1. Expected Behavior: Upgrade should succeed. Look at the Datanode log file to verify that no bad blocks were found by the periodic block scanner. T7 Same as T1. Before starting the upgrade, log into a Datanode and delete a particular blk_xxx.meta file. Then start the DFS upgrade. Expected Behavior: Look at the Namenode log to detect that the upgrade failed on that Datanode.
          Hide
          dhruba borthakur added a comment -

          Merged patch with latest trunk. The latest trunk does not have BlockCrcUpgrade any more.

          Show
          dhruba borthakur added a comment - Merged patch with latest trunk. The latest trunk does not have BlockCrcUpgrade any more.
          Hide
          Mukund Madhugiri added a comment -

          I ran testcase T1 on 100 nodes and it passed using the patch upgradeGenStamp9.patch

          Show
          Mukund Madhugiri added a comment - I ran testcase T1 on 100 nodes and it passed using the patch upgradeGenStamp9.patch
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12381224/upgradeGenStamp10.patch
          against trunk revision 645773.

          @author +1. The patch does not contain any @author tags.

          tests included +1. The patch appears to include 27 new or modified tests.

          javadoc +1. The javadoc tool did not generate any warning messages.

          javac +1. The applied patch does not generate any new javac compiler warnings.

          release audit +1. The applied patch does not generate any new release audit warnings.

          findbugs +1. The patch does not introduce any new Findbugs warnings.

          core tests -1. The patch failed core unit tests.

          contrib tests +1. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2370/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2370/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2370/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2370/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12381224/upgradeGenStamp10.patch against trunk revision 645773. @author +1. The patch does not contain any @author tags. tests included +1. The patch appears to include 27 new or modified tests. javadoc +1. The javadoc tool did not generate any warning messages. javac +1. The applied patch does not generate any new javac compiler warnings. release audit +1. The applied patch does not generate any new release audit warnings. findbugs +1. The patch does not introduce any new Findbugs warnings. core tests -1. The patch failed core unit tests. contrib tests +1. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2370/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2370/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2370/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2370/console This message is automatically generated.
          Hide
          dhruba borthakur added a comment -

          The timeout failure in the TestBalancer test does not seem to be related with this patch at all. I have run this patch multiple times now without seeing this failure again.

          Show
          dhruba borthakur added a comment - The timeout failure in the TestBalancer test does not seem to be related with this patch at all. I have run this patch multiple times now without seeing this failure again.
          Hide
          dhruba borthakur added a comment -

          I would like to commit this patch earlier rather than later, especially because it is a disk format change needed for supporting "appends" to files. Please let me know (before EOB May 6th) if anybody thinks that I should hold off checking in this patch.

          Show
          dhruba borthakur added a comment - I would like to commit this patch earlier rather than later, especially because it is a disk format change needed for supporting "appends" to files. Please let me know (before EOB May 6th) if anybody thinks that I should hold off checking in this patch.
          Hide
          dhruba borthakur added a comment -

          Merged patch with latest trunk.

          Show
          dhruba borthakur added a comment - Merged patch with latest trunk.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12381480/upgradeGenStamp11.patch
          against trunk revision 653638.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 27 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2408/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2408/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2408/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2408/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12381480/upgradeGenStamp11.patch against trunk revision 653638. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 27 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2408/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2408/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2408/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2408/console This message is automatically generated.
          Hide
          Konstantin Shvachko added a comment -

          Resubmitting. TestEditLog failure does not seem to be related to the patch.

          Show
          Konstantin Shvachko added a comment - Resubmitting. TestEditLog failure does not seem to be related to the patch.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12381480/upgradeGenStamp11.patch
          against trunk revision 654128.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 27 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2416/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2416/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2416/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2416/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12381480/upgradeGenStamp11.patch against trunk revision 654128. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 27 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2416/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2416/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2416/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2416/console This message is automatically generated.
          Hide
          dhruba borthakur added a comment -

          Merged patch with latest trunk.

          Show
          dhruba borthakur added a comment - Merged patch with latest trunk.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12381844/upgradeGenStamp12.patch
          against trunk revision 655337.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 27 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2447/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2447/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2447/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2447/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12381844/upgradeGenStamp12.patch against trunk revision 655337. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 27 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2447/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2447/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2447/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2447/console This message is automatically generated.
          Hide
          dhruba borthakur added a comment -

          I plan on committing this patch tonight May 13th. This means that all existing clusters will have to do an "-upgrade".

          Show
          dhruba borthakur added a comment - I plan on committing this patch tonight May 13th. This means that all existing clusters will have to do an "-upgrade".
          Hide
          dhruba borthakur added a comment -

          I just committed this.

          Show
          dhruba borthakur added a comment - I just committed this.
          Hide
          Hudson added a comment -
          Show
          Hudson added a comment - Integrated in Hadoop-trunk #491 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/491/ )

            People

            • Assignee:
              dhruba borthakur
              Reporter:
              dhruba borthakur
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development