Support for upgrading existing cluster to facilitate appends to HDFS files

• Key: HADOOP-2656
• Type: Bug
• Status: Closed
• Resolution: Fixed
• Priority: Major
• Assignee: dhruba borthakur
• Reporter: dhruba borthakur
• Fix Version/s: 0.18.0

Description

HADOOP-1700 describes the design for supporting appends to HDFS files. This design requires a distributed-upgrade to existing cluster installations. The design specifies that the DataNode persist the 8-byte BlockGenerationStamp in the block metadata file. The upgrade code will introduce this new field in the block metadata file and initialize this value to 0.

Comments

[dhruba borthakur]

The Datanode needs to store a block generation stamp for each block. The original idea was to store the block generation stamp inside the meta file of each block. One major disadvantage of this approach is that generation of a block report requires that each meta file be opened and the generation stamp read from it. With about 50K blocks per datanode and a seek time of 10 ms, this might require 300 seconds. This means that the time to restart a cluster cannot be lesser than this 300 seconds.

A few other alternatives:
1. Encode the generation stamp into the name of the metafile. Each metafile will look like blkxxxxxx.genstamp.meta. The block file will remain the same.

2. Encode the generation stamp into the name of the block file. Each block file will be of the form blkxxxxxx.genstamp. The metafile will remain the same.

3. Encode the generation stamp into the name of a new zero-size file named blkxxxxx.genstamp. The block file and the metadata file will remain the same.

4. A completely separate file (one per datanode) that records the metadata of all blocks in the datanode.

I propose that we implement option 1.

[dhruba borthakur]

This patch does the following:

1. Upgrades existing clusters to the new disk format. Each Block has a generation stamp associated with it. Existing blocks get a generation stamp of 0. The generation stamp is used to create the name of the block metafile on the datanode.

2. The Block object has a new field called "generationStamp" of type "long". The BlocksMap on the namenode is keyed on the blockid and the generation stamp.

3. The datanode sends the generation stamp, block id and size of each block in a block report.

3. All log statements that print the blockid now prints the blockid and generationstamp.

4. The client receives the blockid and generation stamp as part of an RPC that receives a block.

5. The DataTransferProtocol sends the blockid and generation stamp with every connection request that the client makes to the datanode(s).

[Tsz Wo (Nicholas), SZE]

upgradeGenStamp4.patch :

• In Block.isBlockFilename(File f), it disallows file name containing "." but "." is used in Block.toString() in the new codes.

• A constant BLOCK_FILE_PREFIX="blk_" is defined in DataStorage but not is used in other part of the codes.

• Another METADATA_EXTENSION=".meta" is defined in FSDataset, you should not define a new one in BlockMetadata

We have to be careful about handling the filenames for blocks and meta data. It would be better if we put all the codes in one place.

[dhruba borthakur]

This patch incorporates the feedback given by Nicholas.

[Hadoop QA]

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12380249/upgradeGenStamp5.patch
against trunk revision 645773.

@author +1. The patch does not contain any @author tags.

tests included +1. The patch appears to include 27 new or modified tests.

javadoc +1. The javadoc tool did not generate any warning messages.

javac +1. The applied patch does not generate any new javac compiler warnings.

release audit +1. The applied patch does not generate any new release audit warnings.

findbugs -1. The patch appears to cause Findbugs to fail.

core tests -1. The patch failed core unit tests.

contrib tests -1. The patch failed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2252/testReport/
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2252/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2252/console

This message is automatically generated.

[Konstantin Shvachko]

Am I correct that only the last block in a file need to have generation stamp?
In that case generationStamp should be a member of INodeFile rather than Block.
It should also be a part LocatedBlock so that clients had the same api to work with blocks.

[Konstantin Shvachko]

BlockMetadata.

1. 3 redundant imports
2. HEADER_BUF_SIZE is never read locally
3. upgradeToCurVersion() should be a part of GenerationStampUpgrade
   also there is problem with indentations.
4. It is better to place all upgrade-related code into GenerationStampUpgrade.
   Say, Header.needsUpgrade() should be static needsUpgrade(Header).
5. not sure all 3 variants of readHeader() are needed. At least one is not called anywhere.
6. I propose to restructure this class in the following way:
   make BlockMetadata.Header class the top level class rather than the inner, rename it to BlockMetadataHeader.
   In other words this means getting rid of the inner class Header, moving all its contents up to BlockMetadata,
   and then renaming the latter.
7. line 160: should not use deprecated getMetaFile() method. You will need old meta data file names
   as long as the upgrade exists, so it is better to have a getPreGenerationMetaFile() or smth. in the
   corresponding upgrade object.

Datanode

1. New comment about the return value in startDistributedUpgradeIfNeedes() is wrong, pls remove.
2. Starting block scanner in offerService() you should check that the scaner has already been started,
   otherwise you will be starting it every hour after the upgrade is finished.

DatanodeBlockInfo

• In detachBlock() can you use static getMetaFile(File, Block) rather than passing FSDataset into it.

DataBlockScanner

• assignInitialVerificationTimes() should use tmpBlock.set() rather then assigning fields directly.

FSDataset

• Deprecated getMetaFile() the one that is "used only by Upgrade code" is not public and could be
  moved directly to the upgrade code rather than deprecated.

FSEditLog

1. In loadFSEdit() line "if (logVersion < -13)" should read as

   if (logVersion <= -13)

2. Introduction of BlockTwo for reading old versions of block layout is kinda confusing.
   I'd rather explicitly read the array length and then each of the two fields in a loop,
   which you have anyway since the prehistoric generation stamp needs to be assigned.

FSImage

• line "if (-14 < imgVersion)" should read

  if (imgVersion <= -13)

  swap the if and else bodies.

FSNamesystem
NameNode
NamenodeFsck
UnderReplicatedBlocks

• This is all mostly about replacing block.getBlockName() with block.toString().

UpgradeManager

boolean isUpgradeCompleted() {
  return currentUpgrade == null;
}

GenerationStampUpgrade

1. same as in BlockMetadata(7) in line 260.
2. ";;" remove one
3. There is a lot of common code in BlockCrcUpgrade and GenerationStampUpgrade.
   Many classes and types are literally (past&copy) the same:
   • UpgradeStatus (with the STARTED field never used)
   • DNInfo
   • BlockLevelStats
   • GenerationStampUpgradeStatusReport = BlockCrcUpgradeStatusReport
   • DatanodeStatsCommand
   • I probably missed some.
4. We should make them all static classes and move either to the base classes
   if the actions are common and useful for other generations of upgrades or create
   UpgradeUtils class and keep them there.
5. The important question is how do we test the generation stamp upgrade?
   With crcs we had a test plan and were testing different scenarios of failure of the upgrade at different stages.
   We should either do the same for genStamps or have a unit test, that tests the scenarios automatically.

ClientProtocol
The ClientProtocol version should be set to 30 (not just the comment).

General

1. Please look for empty line changes in the final patch.
2. Please replace /* */ comments with /** */ for classes, class methods, members, and constants.

[Konstantin Shvachko]

May be this is a good time to return back to the question of renaming hdfs blocks, stop generating block ids randomly, and replace it with sequentially generated ids.
This is related to my previous question whether (in the name-node) we need to store block generation stamp for each block or only for the last block of each file. The only problem here is with prehistoric (according to Dhruba, HADOOP-1700, HADOOP-146, HADOOP-158) blocks.

Reminder: a block is prehistoric if it is reported to the system after its id was reassigned to another physical block.
This happens when a data-node is down for a rather long period of time during which 2 things happen:

1. a block that it owns is removed and then
2. a new block is created with the same block id.

This leads to data corruption, but this can be avoided if block ids are generated sequentially rather than randomly.
AFAIR, the only reason for not converting to sequential ids was that we were afraid of getting into a rather massive (distributed) upgrade that renames all blocks in the system. This patch leads exactly to such an upgrade, and I think it is the right time to adopt the right id generation practice.
The advantage here is that we will be able to store a (smaller) generation stamp (if any) per file rather than per block.

[dhruba borthakur]

Incorporates most of Konstantin's comments. I left the definition of BlockTwo just because I think it is cleaner this way. Also, the reason some code is duplicated between BlockCrcUpgrade and BlockGenerationStampUpgrade is because the BlockCrcUpgrade code might probably go away in the next release or so.

[Hadoop QA]

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12380642/upgradeGenStamp6.patch
against trunk revision 645773.

@author +1. The patch does not contain any @author tags.

tests included +1. The patch appears to include 27 new or modified tests.

javadoc +1. The javadoc tool did not generate any warning messages.

javac +1. The applied patch does not generate any new javac compiler warnings.

release audit +1. The applied patch does not generate any new release audit warnings.

findbugs -1. The patch appears to introduce 10 new Findbugs warnings.

core tests +1. The patch passed core unit tests.

contrib tests +1. The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2296/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2296/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2296/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2296/console

This message is automatically generated.

[dhruba borthakur]

Fix findbugs warnings.

[dhruba borthakur]

Fix findbugs warnings.

[Hadoop QA]

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12380784/upgradeGenStamp7.patch
against trunk revision 645773.

@author +1. The patch does not contain any @author tags.

tests included +1. The patch appears to include 27 new or modified tests.

javadoc +1. The javadoc tool did not generate any warning messages.

javac +1. The applied patch does not generate any new javac compiler warnings.

release audit +1. The applied patch does not generate any new release audit warnings.

findbugs -1. The patch appears to introduce 1 new Findbugs warnings.

core tests +1. The patch passed core unit tests.

contrib tests +1. The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2310/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2310/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2310/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2310/console

This message is automatically generated.

[dhruba borthakur]

Fixed the last (hopefully) findbugs warnings.

[Hadoop QA]

+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12380799/upgradeGenStamp8.patch
against trunk revision 645773.

@author +1. The patch does not contain any @author tags.

tests included +1. The patch appears to include 27 new or modified tests.

javadoc +1. The javadoc tool did not generate any warning messages.

javac +1. The applied patch does not generate any new javac compiler warnings.

release audit +1. The applied patch does not generate any new release audit warnings.

findbugs +1. The patch does not introduce any new Findbugs warnings.

core tests +1. The patch passed core unit tests.

contrib tests +1. The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2313/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2313/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2313/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2313/console

This message is automatically generated.

[dhruba borthakur]

Merged patch with latest trunk.

[dhruba borthakur]

merged path with latest trunk.

[dhruba borthakur]

Here is a Test Plan for testing Distributed Upgrades for Block Generation Stamp

Test Cases

T1 Install build version 0.17 and create a new HDFS cluster of size 500 nodes.. Run random writer, sort, and then sort validation. Then shutdown cluster. Install version 0.18 build on the cluster. Start cluster with the -upgrade option.
Expected behavior: The Namenode should trigger an upgrade on all Datanodes. Then the Namenode should exit safe mode automatically. Run sort validation again to verify that data is intact.

T2 Start a DFS Upgrade and restart Namenode during the upgrade.
Expected Behaviour: Upgrade should proceed normally with appropriate messages in Namenode log

T3 Start a DFS upgrade and restart some of the Datanodes while the upgrade is occurring.
Expected behaviour: Upgrade should proceed normally with appropriate messages in Datenode logs.

T4 Start an upgrade and stop two Datanodes.
Expected Behavior: Upgrade should complete successfully because at least one replica of every block should still be available.

T4 Start an upgrade and stop half of the Datanodes.
Expected Behavior: Upgrade should stall because at least one replica of every block would not be available. Verify that useful message is displayed in the Namenode logs.

T5 Start a DFS upgrade and let it complete successfully. Then issue the admin command to finalize the upgrade.
Excepted Behavior: The Namenode should be able to finalize the cluster.

T5 Same as T4. Once the upgrade completes successfully, bring up the two Datanodes that were shut down earlier.
Expected behavior: These two Datanodes should start their own upgrades. Note that the Namenode could have already replicated these blocks.

T6 Same at T1 but with more aggressive periodic block scanner by setting dfs.datanode.scan.period.hours to 1.
Expected Behavior: Upgrade should succeed. Look at the Datanode log file to verify that no bad blocks were found by the periodic block scanner.

T7 Same as T1. Before starting the upgrade, log into a Datanode and delete a particular blk_xxx.meta file. Then start the DFS upgrade.
Expected Behavior: Look at the Namenode log to detect that the upgrade failed on that Datanode.

[dhruba borthakur]

Merged patch with latest trunk. The latest trunk does not have BlockCrcUpgrade any more.

[Mukund Madhugiri]

I ran testcase T1 on 100 nodes and it passed using the patch upgradeGenStamp9.patch

[Hadoop QA]

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12381224/upgradeGenStamp10.patch
against trunk revision 645773.

@author +1. The patch does not contain any @author tags.

tests included +1. The patch appears to include 27 new or modified tests.

javadoc +1. The javadoc tool did not generate any warning messages.

javac +1. The applied patch does not generate any new javac compiler warnings.

release audit +1. The applied patch does not generate any new release audit warnings.

findbugs +1. The patch does not introduce any new Findbugs warnings.

core tests -1. The patch failed core unit tests.

contrib tests +1. The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2370/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2370/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2370/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2370/console

This message is automatically generated.

[dhruba borthakur]

The timeout failure in the TestBalancer test does not seem to be related with this patch at all. I have run this patch multiple times now without seeing this failure again.

[dhruba borthakur]

I would like to commit this patch earlier rather than later, especially because it is a disk format change needed for supporting "appends" to files. Please let me know (before EOB May 6th) if anybody thinks that I should hold off checking in this patch.

[dhruba borthakur]

Merged patch with latest trunk.

[Hadoop QA]

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12381480/upgradeGenStamp11.patch
against trunk revision 653638.

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 27 new or modified tests.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 findbugs. The patch does not introduce any new Findbugs warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

-1 core tests. The patch failed core unit tests.

+1 contrib tests. The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2408/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2408/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2408/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2408/console

This message is automatically generated.

[Konstantin Shvachko]

Resubmitting. TestEditLog failure does not seem to be related to the patch.

[Hadoop QA]

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12381480/upgradeGenStamp11.patch
against trunk revision 654128.

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 27 new or modified tests.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 findbugs. The patch does not introduce any new Findbugs warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

-1 core tests. The patch failed core unit tests.

+1 contrib tests. The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2416/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2416/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2416/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2416/console

This message is automatically generated.

[dhruba borthakur]

Merged patch with latest trunk.

[Hadoop QA]

+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12381844/upgradeGenStamp12.patch
against trunk revision 655337.

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 27 new or modified tests.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 findbugs. The patch does not introduce any new Findbugs warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

+1 core tests. The patch passed core unit tests.

+1 contrib tests. The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2447/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2447/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2447/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2447/console

This message is automatically generated.

[dhruba borthakur]

I plan on committing this patch tonight May 13th. This means that all existing clusters will have to do an "-upgrade".

[dhruba borthakur]

I just committed this.

[Hudson]

Integrated in Hadoop-trunk #491 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/491/)