Could not get the exact reason for failures because they are always passing in my local box.

For me always they are passing. Anybody has some clue on this tests failures?

I got the problem here. These test-cases are order sensitive [Using static maps].
But JUnit runner will not guarantee the order of execution.
Corrected these tests. Will upload a patch soon.

Thanks Lakshman,
By looking at the testcode, sounds reasonable. That will be the cause.
By putting the last test to first, it is reliably reproducing in my eclipse
Please post the patch, It help to get the trunk green soon. They are failing from long time.

Thanks
Uma

Corrected the testcases.

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12496088/HDFS-2346.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 6 new or modified tests.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hdfs.server.namenode.TestStartup
org.apache.hadoop.hdfs.TestDfsOverAvroRpc

+1 contrib tests. The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1280//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1280//console

This message is automatically generated.

Hi Lakshman,

Thanks a lot for the patch.
I just gone through the patch.
Change looks good. But one mistake is there are other tests to add @Test annotation.

Thanks
Uma

It have just checked, it will be applicable for 20.205 and 22 versions also. Trunk patch can be applied directly on 23 branch.

Updated the patches on behalf of Lakshman.

Thanks
Uma

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12496140/HDFS-2346.Trunk.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 6 new or modified tests.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hdfs.TestDfsOverAvroRpc
org.apache.hadoop.hdfs.TestDFSUpgrade

+1 contrib tests. The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1282//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1282//console

This message is automatically generated.

Thanks for your review and correction Uma.

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12496465/HDFS-2346.20-security.205.Patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 3 new or modified tests.

-1 patch. The patch command could not apply the patch.

Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1287//console

This message is automatically generated.

Current patch is for 20security branch.

+1 I looked at 0.22 ant trunk patches. Both look good to me.

To reduce the inter-test dependency, might it make sense to reinitialize the ReplicasMap object in the setup() method each time? Even though ReplicasMap is no longer static, it's still shared between the test methods in the class. The original code would only add the block to the map once, since it was annotated with @BeforeClass, but now the setup() method will be executed for each test method.

The previous comment applies specifically to TestReplicasMap, but roughly the same issue applies to TestHost2NodesMap as well. The setup() method was previously executed once at class-initialization time, and now it will be executed once for each test method, but the Host2NodesMap object will not be reinitialized.

This is beacuse some tests are removing the elements completely. when other tests are assrting on that eleents they are failing. I feel for this tests resetting the elements in setup would be the better option. Every test should be able to run indepandantly rite. Tests performance wise also it should be very less impact.

Thanks
Uma

To reduce the inter-test dependency, might it make sense to reinitialize the ReplicasMap object in the setup() method each time?

For each testcase (testMethod), TestClass will be instantiated once. i.e, each test method will be invoked on a new instance of the test class. For every new instance, this map [instance variables] will be re-instantiated. So, it doesn't make any difference from resetting in setup method.

I just verified above behavior from eclipse by executing the test methods of these class in different order. With this patch tests are always passing irrelevant to the execution order.

@atm, I hope this clarifies your concern about execution order dependency.

Thanks a lot Lakshman for more clarifications.

I just checked this by printing the map in setUp method:
org.apache.hadoop.hdfs.server.namenode.Host2NodesMap@ab50cd
org.apache.hadoop.hdfs.server.namenode.Host2NodesMap@758fc9
org.apache.hadoop.hdfs.server.namenode.Host2NodesMap@32fb4f
org.apache.hadoop.hdfs.server.namenode.Host2NodesMap@1113708
Here map objects are different for each test.

Thanks
Uma

Yup, my mistake. I didn't realize the test class would get instantiated for each test method. Thanks for checking on that.

+1 - the patch looks good to me.

Aaron do you mind committing this into all those versions or at least some of them.

These tests are passing now in 0.20-security and 0.20.205, so removed those release numbers from the "Affected Versions" list.

Integrated in Hadoop-Mapreduce-trunk-Commit #1000 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1000/)
HDFS-2346. TestHost2NodesMap & TestReplicasMap will fail depending upon execution order of test methods (Laxman via atm)

atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1176994
Files :

• /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
• /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestHost2NodesMap.java
• /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestReplicasMap.java

Integrated in Hadoop-Common-trunk-Commit #979 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/979/)
HDFS-2346. TestHost2NodesMap & TestReplicasMap will fail depending upon execution order of test methods (Laxman via atm)

atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1176994
Files :

• /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
• /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestHost2NodesMap.java
• /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestReplicasMap.java

Integrated in Hadoop-Hdfs-trunk-Commit #1057 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1057/)
HDFS-2346. TestHost2NodesMap & TestReplicasMap will fail depending upon execution order of test methods (Laxman via atm)

atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1176994
Files :

• /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
• /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestHost2NodesMap.java
• /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestReplicasMap.java

Hi Matt,
By looking at the code this patch is applicable for 205 as well. Since test failures because of order, it may not fail always.
So, it is better to commit in 205 also.

Thanks
Uma

I've just committed this to 0.23 and trunk. Uma, the 0.22 patch appears to no longer apply.

I didn't apply this to branch-0.20-security pending a resolution of Matt and Uma's discussion.

Note that currently the entry for this JIRA is in CHANGES.txt under version 0.23, and so will need to be changes if/when this patch is committed to 0.20-* or 0.22.

I committed this to 0.22. It applied fine. Thanks Laxman and Uma.

Aaron, thanks for the commits. One nit for you. CHANGES.txt should reflect the oldest version of Hadoop the patch has been applied to. Since the security branch essentially tracks its changes independently of other branches, the latest version in this case would be 0.22.
It would be good to move the changes under 0.22.0 section in both of your commits.

Thanks Konst. I deliberately didn't put it under the 0.22 section in those commits since I hadn't committed it to that branch. I originally put it under 0.24, since I committed it to trunk first, but then put it under 0.23 and removed the entry from 0.24 when I committed it to 0.23.

Integrated in Hadoop-Hdfs-22-branch #94 (See https://builds.apache.org/job/Hadoop-Hdfs-22-branch/94/)
HDFS-2346. TestHost2NodesMap & TestReplicasMap will fail depending upon execution order of test methods. Contributed by Laxman and Uma Maheswara Rao G.

shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1177034
Files :

• /hadoop/common/branches/branch-0.22/hdfs/CHANGES.txt
• /hadoop/common/branches/branch-0.22/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/datanode/TestReplicasMap.java
• /hadoop/common/branches/branch-0.22/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestHost2NodesMap.java

I didn't apply this to branch-0.20-security pending a resolution of Matt and Uma's discussion.

@Matt, if you agree with my previous comment, can you please apply the patch on 205.So that we can move it to closure.

To Konstantin and Aaron, Thanks a lot for taking a look!

Integrated in Hadoop-Hdfs-trunk #815 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/815/)
HDFS-2346. TestHost2NodesMap & TestReplicasMap will fail depending upon execution order of test methods (Laxman via atm)

atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1176994
Files :

• /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
• /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestHost2NodesMap.java
• /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestReplicasMap.java

Integrated in Hadoop-Hdfs-0.23-Build #24 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/24/)
svn merge -c 1176994 from trunk for HDFS-2346.

atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1177019
Files :

• /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
• /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestHost2NodesMap.java
• /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestReplicasMap.java

Integrated in Hadoop-Mapreduce-0.23-Build #31 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/31/)
svn merge -c 1176994 from trunk for HDFS-2346.

atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1177019
Files :

• /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
• /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestHost2NodesMap.java
• /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestReplicasMap.java

Integrated in Hadoop-Mapreduce-trunk #845 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/845/)
HDFS-2346. TestHost2NodesMap & TestReplicasMap will fail depending upon execution order of test methods (Laxman via atm)

atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1176994
Files :

• /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
• /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestHost2NodesMap.java
• /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestReplicasMap.java

Hi Matt,
Since the code changes applicable for 205 also, i updated the Affect versions to 205.

Thanks
Uma

Committed to branch-0.20-security and branch-0.20-security-205. Observed fixed in all affected versions, so marking Resolved.

Thanks, Uma and Laxman!

Thanks a lot Matt, for taking a look!

Integrated in Hadoop-Common-trunk-Commit #1335 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1335/)
Fix CHANGES.txt for HDFS-2346 in trunk.

shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1207700
Files :

• /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt

Integrated in Hadoop-Hdfs-trunk-Commit #1409 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1409/)
Fix CHANGES.txt for HDFS-2346 in trunk.

shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1207700
Files :

• /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt

Integrated in Hadoop-Mapreduce-trunk-Commit #1359 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1359/)
Fix CHANGES.txt for HDFS-2346 in trunk.

shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1207700
Files :

• /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt

Integrated in Hadoop-Hdfs-trunk #879 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/879/)
Fix CHANGES.txt for HDFS-2346 in trunk.

shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1207700
Files :

• /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt

Integrated in Hadoop-Mapreduce-trunk #912 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/912/)
Fix CHANGES.txt for HDFS-2346 in trunk.

shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1207700
Files :

• /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt