Add some more hints of the problem when datanode and namenode don't match

• Key: HADOOP-3448
• Type: Improvement
• Status: Closed
• Resolution: Fixed
• Priority: Minor
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Affects Version/s: 0.18.0
• Fix Version/s: 0.18.0

Description

When there is a mismatch between name and data mode, and you are running with -ea set, then Datanode.handshake() bails out with an assertion "Data-node and name-node layout versions must be the same.";

However, this message doesnt actually say which version numbers are at fault. A better error message would include the version information, so pointing the finger of blame would be easier.

Comments

[Steve Loughran]

patch to make the assertion more detailed
"Data-node and name-node layout versions must be the same."
+ "Expected: "+ FSConstants.LAYOUT_VERSION + " actual "+ nsInfo.getLayoutVersion();

[Steve Loughran]

The patch I've submitted adds the version information, so the the stack trace can be more useful than :

java.lang.AssertionError: Data-node and name-node layout versions must be the same.,

However, one thing to consider is whether this should be checked every time the datanode starts up, rather than skipping it if -ea is disabled. Otherwise there is a risk that the problem will not be picked up in production.

[Steve Loughran]

patch available.

[Konstantin Shvachko]

This should actually never happen. Something is wrong with your build. Here is why.
DataNode.handshake() receives version information from the name-node and verifies it against its own versions.
And the first thing that is verified is the build version. BV is reset every time you re-build hadoop, while LAYOUT_VERSION
changes only if its is actually changed in the code.
So, if the LAYOUT_VERSIONs are different then the build versions must be different too.
The assert you mentioned is really an assert and should not be used for diagnostic purposes.
May be the problem is somewhere else. Could you please post here the (different) layout versions and (presumably equal) build versions.

[Steve Loughran]

The problem has 'gone away'; I think forcing clean builds through fixed it. Given I was running the same version everywhere, I was somewhat surprised too.

two possible causes
-classloader/dual version on path (ivy is in charge of CP setup; it should have caught this)
-cached versions of constants not being picked up
Although Java is fairly smart with dependencies, constant primitive types get copied around at compile time, so can be out of date if something changes.

anyway, the build is fixed. Should the patch be needed? Well, maybe the text could be clearer or just point to a wiki page discussing this issue.

[Konstantin Shvachko]

I also noticed that if you don't do a clean build the new constants will not be picked up everywhere.
This probably has something to do with dependencies in our build.xml.

[Steve Loughran]

Its probably because the .class files dont even track where constants came from. Change a constant: do a clean build.

Perhaps we should change the assert to say this.

[Robert Chansler]

Assigning to whomever submitted a patch so as to better manage committing things that are ready for prime time.

[Owen O'Malley]

I just committed this. Thanks, Steve!