Details
-
Type:
Bug
-
Status: Resolved
-
Priority:
Major
-
Resolution: Fixed
-
Affects Version/s: None
-
Fix Version/s: 3.0.0-alpha1
-
Component/s: test
-
Labels:None
-
Hadoop Flags:Reviewed
Description
TestSeveralNameNodes has been failing in precommit builds. It usually times out on waiting for the last thread to finish writing.
Activity
- All
- Comments
- Work Log
- History
- Activity
- Transitions
The failover thread in HAStressTestHarness will invoke failover periodically with fixed sleep time. The msBetweenFailovers is set to 1000 ms for TestSeveralNameNodes.
for (int i = 0; i < nns; i++) {
int next = (i + 1) % nns;
...
cluster.transitionToStandby(i);
cluster.transitionToActive(next);
...
Thread.sleep(msBetweenFailovers);
Retry proxy of client have sleep time exponential to number of retries on failover. The client is possible to sleep up to around 15 seconds if it repeatedly fails on the operation. The client may not get enough effective run time due to this.
2015-12-24 12:22:00,784 [Thread-250] INFO retry.RetryInvocationHandler (RetryInvocationHandler.java:invoke(147)) - Exception while invoking create of class ClientNamenodeProtocolTranslatorPB over localhost/127.0.0.1:42201 after 4 fail over attempts. Trying to fail over after sleeping for 10161ms. org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category WRITE is not supported in state standby. Visit https://s.apache.org/sbnn-error
If we set time between failover to greater value, we might not get enough failovers kicking in while test threads are doing operations. I think it would be better to set smaller maximum sleep time for clients by setting dfs.client.failover.sleep.max.millis to make test stable and shorter.
String SLEEPTIME_BASE_KEY = PREFIX + "sleep.base.millis";
int SLEEPTIME_BASE_DEFAULT = 500;
String SLEEPTIME_MAX_KEY = PREFIX + "sleep.max.millis";
int SLEEPTIME_MAX_DEFAULT = 15000;
In addition, test should exit immediately after all test threads finish the work.
 -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | reexec | 0m 0s | Docker mode activated. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
| +1 | mvninstall | 7m 37s | trunk passed |
| +1 | compile | 0m 41s | trunk passed with JDK v1.8.0_66 |
| +1 | compile | 0m 43s | trunk passed with JDK v1.7.0_91 |
| +1 | checkstyle | 0m 16s | trunk passed |
| +1 | mvnsite | 0m 53s | trunk passed |
| +1 | mvneclipse | 0m 13s | trunk passed |
| +1 | findbugs | 1m 55s | trunk passed |
| +1 | javadoc | 1m 7s | trunk passed with JDK v1.8.0_66 |
| +1 | javadoc | 1m 47s | trunk passed with JDK v1.7.0_91 |
| +1 | mvninstall | 0m 49s | the patch passed |
| +1 | compile | 0m 42s | the patch passed with JDK v1.8.0_66 |
| +1 | javac | 0m 42s | the patch passed |
| +1 | compile | 0m 43s | the patch passed with JDK v1.7.0_91 |
| +1 | javac | 0m 43s | the patch passed |
| +1 | checkstyle | 0m 16s | the patch passed |
| +1 | mvnsite | 0m 53s | the patch passed |
| +1 | mvneclipse | 0m 14s | the patch passed |
| +1 | whitespace | 0m 0s | Patch has no whitespace issues. |
| +1 | findbugs | 2m 3s | the patch passed |
| +1 | javadoc | 1m 8s | the patch passed with JDK v1.8.0_66 |
| +1 | javadoc | 1m 49s | the patch passed with JDK v1.7.0_91 |
| -1 | unit | 73m 26s | hadoop-hdfs in the patch failed with JDK v1.8.0_66. |
| -1 | unit | 72m 38s | hadoop-hdfs in the patch failed with JDK v1.7.0_91. |
| +1 | asflicense | 0m 28s | Patch does not generate ASF License warnings. |
| 173m 10s |
| Reason | Tests |
|---|---|
| JDK v1.8.0_66 Failed junit tests | hadoop.hdfs.server.blockmanagement.TestReplicationPolicyConsiderLoad |
| JDK v1.7.0_91 Failed junit tests | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure |
| hadoop.hdfs.server.blockmanagement.TestReplicationPolicyConsiderLoad | |
| hadoop.hdfs.server.datanode.TestFsDatasetCache | |
| hadoop.hdfs.web.TestWebHDFS |
This message was automatically generated.
I updated the patch. Though the possibility is low, clients could exhaust retries in test run time. The max retries is increased in 002.
I could reproduce the test failure in 10 runs without the patch. No failure in 100 runs with the patch applied.
The failure of TestReplicationPolicyConsiderLoad was already fixed by HDFS-9597. Other tests are flaky regardless of the patch and succeeded on my local environment.
| -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | reexec | 0m 0s | Docker mode activated. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
| +1 | mvninstall | 7m 31s | trunk passed |
| +1 | compile | 0m 39s | trunk passed with JDK v1.8.0_66 |
| +1 | compile | 0m 43s | trunk passed with JDK v1.7.0_91 |
| +1 | checkstyle | 0m 15s | trunk passed |
| +1 | mvnsite | 0m 52s | trunk passed |
| +1 | mvneclipse | 0m 14s | trunk passed |
| +1 | findbugs | 1m 54s | trunk passed |
| +1 | javadoc | 1m 8s | trunk passed with JDK v1.8.0_66 |
| +1 | javadoc | 1m 49s | trunk passed with JDK v1.7.0_91 |
| +1 | mvninstall | 0m 47s | the patch passed |
| +1 | compile | 0m 46s | the patch passed with JDK v1.8.0_66 |
| +1 | javac | 0m 46s | the patch passed |
| +1 | compile | 0m 45s | the patch passed with JDK v1.7.0_91 |
| +1 | javac | 0m 45s | the patch passed |
| +1 | checkstyle | 0m 16s | the patch passed |
| +1 | mvnsite | 0m 51s | the patch passed |
| +1 | mvneclipse | 0m 12s | the patch passed |
| +1 | whitespace | 0m 0s | Patch has no whitespace issues. |
| +1 | findbugs | 2m 1s | the patch passed |
| +1 | javadoc | 1m 1s | the patch passed with JDK v1.8.0_66 |
| +1 | javadoc | 1m 45s | the patch passed with JDK v1.7.0_91 |
| -1 | unit | 67m 56s | hadoop-hdfs in the patch failed with JDK v1.8.0_66. |
| -1 | unit | 65m 44s | hadoop-hdfs in the patch failed with JDK v1.7.0_91. |
| +1 | asflicense | 0m 21s | Patch does not generate ASF License warnings. |
| 160m 17s |
| Reason | Tests |
|---|---|
| JDK v1.8.0_66 Failed junit tests | hadoop.hdfs.server.datanode.TestBlockScanner |
| hadoop.hdfs.server.datanode.TestBlockReplacement | |
| hadoop.hdfs.server.datanode.TestTriggerBlockReport | |
| JDK v1.7.0_91 Failed junit tests | hadoop.hdfs.TestBlockStoragePolicy |
| hadoop.hdfs.TestRollingUpgrade |
This message was automatically generated.
+1 for the patch. I had an environment that was able to repro this test failure consistently. After applying the patch, I could no longer repro the failure. I have committed this to trunk. Masatake Iwasaki, thank you!
FAILURE: Integrated in Hadoop-trunk-Commit #9035 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9035/)
HDFS-9376. TestSeveralNameNodes fails occasionally. Contributed by (cnauroth: rev 84a81477912644290173518d566b586305b85bf7)
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestSeveralNameNodes.java
- hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
Thanks, Chris Nauroth.
This is what we get.
Running org.apache.hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 128.809 sec <<< FAILURE! - in org.apache.hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes
testCircularLinkedListWrites(org.apache.hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes) Time elapsed: 128.646 sec <<< FAILURE!
java.lang.AssertionError: Some writers didn't complete in expected runtime! Current writer state:[Circular Writer:
directory: /test-1
target length: 50
current item: 49
done: false
] expected:<0> but was:<1>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
When I tried following, it passed more reliably, but there might be a better fix or a more fundamental issue in the core code.