|
> If this workaround sounds feasible, then we can remove this issue from the 0.18 Blocker list.
And document this procedure in release notes? I think it is better to fix the problem than asking users to go through different procedure for upgrade. There are users with vastly different expertise and caution. What happens if they don't follow the procedure by mistake? Also this special work around needs to be followed in even in future when some one is upgrading from 0.17 to 0.20. May be a good solution would be to covert the distributed upgrade into local data-node upgrade.
It will solve both of the problems above, plus eliminate the warning message reported in The only disadvantage of this approach I can see is that data-nodes will take rather long time to startup, around 5 minutes each on a large cluster. But this can be solved by including reasonable messages about the upgrade progress. I disagree with Raghu to a certain extent. From my experience, when an administrator wants to upgrade the cluster, he/she restarts the namenode (before installing new software) so that that transaction log is consumed. This ensures that restarting with a new version of the software does not cause any unwanted interactions with edits-log-processing.
The workaround listed in this issue is just an extension of the above procedure. Konstantin: can you pl explain what is a local data-node upgrade? and why it solves this bug? Thanks. If the procedure is optional, then its ok. I was mainly commenting about "required" extra steps. This this was a blocker I thought workaround was required.
I haven't looked at the code around the upgrade or generation stamps yet. I think the basic fix (or work around) that Konstantin suggesting is to rename all the metadata files with default generation stamp when Datanode starts before reporting them to Namenode.
Main advantage I see is that it will avoid extra warnings for every block. Changing the distributed -upgrade to a local upgrade sounds ok (though not very essential) to me. It seems to be another hack: the real problem being that the distributed framework do not yet have a mechanism to specify that "do not send block reports before distributed upgrade is complete".
Another alternative would be to keep the namenode in safemode till distributed upgrade is complete.... isn't this already true today? > ... specify that "do not send block reports before distributed upgrade is complete".
Yes, we can fix it with more features like this. Still we will be left with thousands of warning messages. Question is what do we do for this jira. Whether a local upgrade is a hack I think is debatable. It makes logical sense to me : Datanode metada file name format has changed between 0.17 and 0.18, so datanode converts these names to new format when it is upgraded. In any case, a hack only the core developers need to know might be more desirable than a hack in upgrade procedure that all admins need to be aware of. If there consensus to convert metadata file name when datanode starts up, then I will submit a patch. Hi Raghu, I am perfectly ok with the fix you are proposing. Thanks for taking this up.
The advantages of the local upgrade (or as we also call it "version upgrade") as opposed to a distributed upgrade are that it avoids both problems stated in this jira, and also avoids thousands of warnings during data-node startup.
In my opinion, local/version upgrade is logically correct and is not a "hack" at all, because each data-node can complete the upgrade on its own without interacting with other data-nodes or the name-node. The distributed upgrade should be used when such intercommunications are required. E.g. during the crc-upgrade this was unavoidable. The startup time will be the only disadvantage so the upgrade progress should be logged every 20-30 seconds. Suggested patch for trunk. This has the minimum changes for upgrade to work and disables the distributed upgrade. It does not take any longer than normal version upgrade and thus does not need more notifications than normal upgrade has.
Once this fix is ok, I will submit a patch that removes code related to distributed upgrade.. or it could be done in a seperate jira. I like this approach because it combines hard-linking with renaming and therefore does the task with no overhead.
Thanks for the review Konstantin.
Attached patches for trunk and 0.18 has the suggested changes and removes GenStamp distributed upgrade. The patch is large mainly because it removes around 1000 lines. 3 files are deleted in trunk and one in 0.18. -1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12386129/HADOOP-3677-trunk.patch  against trunk revision 677470. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2885/testReport/ This message is automatically generated. Integrated in Hadoop-trunk #581 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/581/
) |
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
 |
|
Bug
Closed
Blocker
Problems with generation stamp upgrade
1. Shut down namenode and then restart namenode (with existing release). This will cause datanodes to send block reports and delete blocks that are not in the namespace.
2. Shutdown cluster. Install new software on all nodes. Restart with -upgrade option. This will not have to delete blocks becuase orphaned blocks were already deleted in Step-1.
If this workaround sounds feasible, then we can remove this issue from the 0.18 Blocker list.