The stack is typically:
It can be reproduced without parallelization or without executing the other tests in the class. It seems to fail about 5% of the time.
This comes from the naming policy for the directories in MiniDFSCluster#startDataNode. It depends on the number of nodes currently in the cluster, and does not take into account previous starts/stops:
This means that it if we want to stop/start a datanode, we should always stop the last one, if not the names will conflict. This test exhibits the behavior:
And then this behavior is randomly triggered in testLogRollOnDatanodeDeath because when we do
and then kill the datanodes in the pipeline, we will have:
- most of the time: pipeline = 1 & 2, so after killing 1&2 we can start a new datanode that will reuse the available 2's directory.
- sometimes: pipeline = 1 & 3. In this case,when we try to launch the new datanode, it fails because it wants to use the same directory as the still alive '2'.
There are two ways of fixing the test:
1) Fix the naming rule in MiniDFSCluster#startDataNode, for example to ensure that the directory names will not be reused. But I wonder if there is not a testCase somewhere (may be not in HBase) depending on this behavior.
2) Kill explicitly the first and second datanode without using the pipeline to be sure that the names won't conflict.
Feedback welcome and the choice to make here...
|Status||Resolved [ 5 ]||Closed [ 6 ]|
|Status||Patch Available [ 10002 ]||Resolved [ 5 ]|
|Resolution||Fixed [ 1 ]|
|Hadoop Flags||Reviewed [ 10343 ]|
|Fix Version/s||0.92.0 [ 12314223 ]|
|Fix Version/s||0.94.0 [ 12316419 ]|
|Affects Version/s||0.92.0 [ 12314223 ]|
|Affects Version/s||0.94.0 [ 12316419 ]|
|Summary||TestLogRolling#testLogRollOnDatanodeDeath fails sometimes on central build or hadoop QA on trunk ("The directory is already locked.")||TestLogRolling#testLogRollOnDatanodeDeath fails sometimes on Jenkins or hadoop QA ("The directory is already locked.")|
|Status||Open [ 1 ]||Patch Available [ 10002 ]|