[HDFS-13580] FailOnTimeout error in TestDataNodeVolumeFailure$testVolumeFailure

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Unassigned
• Reporter: Ewan Higgs
• Created: 17/May/18 09:09
• Updated: 17/May/18 09:09

Description

testVolumeFailure is flaky. If we run it 50 times, it will fail about twice with the following backtrace:

 

java.lang.Exception: test timed out after 120000 milliseconds

    at java.lang.Object.wait(Native Method)
    at java.lang.Thread.join(Thread.java:1253)
    at org.junit.internal.runners.statements.FailOnTimeout.evaluateStatement(FailOnTimeout.java:26)
    at org.junit.internal.runners.statements.FailOnTimeout.evaluate(FailOnTimeout.java:17)
    at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
    at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
    at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)

The second error (immediately after) is probably due to an issue with cleaning up a timed out test:

java.io.IOException: Cannot remove data directory: /Users/ehiggs/src/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/datapath '/Users/ehiggs/src/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data': 
   absolute:/Users/ehiggs/src/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data
   permissions: drwx
path '/Users/ehiggs/src/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs': 
   absolute:/Users/ehiggs/src/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs
   permissions: drwx
path '/Users/ehiggs/src/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data': 
   absolute:/Users/ehiggs/src/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data
   permissions: drwx
path '/Users/ehiggs/src/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test': 
   absolute:/Users/ehiggs/src/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test
   permissions: drwx
path '/Users/ehiggs/src/hadoop/hadoop-hdfs-project/hadoop-hdfs/target': 
   absolute:/Users/ehiggs/src/hadoop/hadoop-hdfs-project/hadoop-hdfs/target
   permissions: drwx
path '/Users/ehiggs/src/hadoop/hadoop-hdfs-project/hadoop-hdfs': 
   absolute:/Users/ehiggs/src/hadoop/hadoop-hdfs-project/hadoop-hdfs
   permissions: drwx
path '/Users/ehiggs/src/hadoop/hadoop-hdfs-project': 
   absolute:/Users/ehiggs/src/hadoop/hadoop-hdfs-project
   permissions: drwx
path '/Users/ehiggs/src/hadoop': 
   absolute:/Users/ehiggs/src/hadoop
   permissions: drwx
path '/Users/ehiggs/src': 
   absolute:/Users/ehiggs/src
   permissions: drwx
path '/Users/ehiggs': 
   absolute:/Users/ehiggs
   permissions: drwx
path '/Users': 
   absolute:/Users
   permissions: dr-x
path '/': 
   absolute:/
   permissions: dr-x


   at org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:896)
   at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:517)
   at org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:476)
   at org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure.setUp(TestDataNodeVolumeFailure.java:125)
   at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   at java.lang.reflect.Method.invoke(Method.java:498)
   at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
   at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
   at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
   at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
   at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
   at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)