Always calculate mis-replicated blocks when safe-mode is turned off.

• Key: HADOOP-4610
• Type: Bug
• Status: Closed
• Resolution: Fixed
• Priority: Major
• Assignee: Konstantin Shvachko
• Reporter: Konstantin Shvachko
• Affects Version/s: 0.18.0
• Fix Version/s: 0.18.3

Description

After HADOOP-4597 there is one last case when mis-replicated blocks are not processed. It is when the safe-mode extension = 0.
We should regenerate neededReplications and excessReplicateMap whenever the safe mode is turned off.

Comments

[Konstantin Shvachko]

This patch removes the boolean parameter which would be always set to true otherwise.

[Hadoop QA]

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12393482/NeededReplAlways.patch
against trunk revision 712033.

+1 @author. The patch does not contain any @author tags.

-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 findbugs. The patch does not introduce any new Findbugs warnings.

+1 Eclipse classpath. The patch retains Eclipse classpath integrity.

+1 core tests. The patch passed core unit tests.

+1 contrib tests. The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3548/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3548/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3548/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3548/console

This message is automatically generated.

[Raghu Angadi]

I think this should go into 0.18.3. This is a serious regression if 'extension' is set to 0 in some config. It will be many month before 0.20. becomes the stable branch. Note that we would have thought threshold being set to 1.1 would be rare.. but it happens.

+1 for the patch.

[Konstantin Shvachko]

The same manual test as in HADOOP-4597 is applicable here.

[Konstantin Shvachko]

> I think this should go into 0.18.3.

Setting the 'extension' to 0 does not have a real use case at the moment.
I do not think we should apply it to 0.18 until a need has been developed.
I will commit it to 0.19 because this does not require a separate patch and more testing.

[Konstantin Shvachko]

I just committed this.

[Koji Noguchi]

Setting the 'extension' to 0 does not have a real use case at the moment.

If it's straightforward, it would be nice to have this fixed in 0.18.3.
We don't know how other users configure their clusters.

Less blocks go missing, less angry users on the mailing list.

[Konstantin Shvachko]

I just committed this to 0.18 branch.

[Hudson]

Integrated in Hadoop-trunk #655 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/655/)
. Reflect commit to 0.18.3 in trunk.
. Always calculate mis-replicated blocks when safe-mode is turned off. Contributed by Konstantin Shvachko.