Details
Critical Description
Actively-used test helper function `testVolumeConfig` in `org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration` chmod a directory with invalid perm 000 for testing purposes but later failed to chmod back this directory with a valid perm if the assertion inside this function failed. Any subsequent `mvn test` command would fail to run if this test had failed before. It is because Maven failed to build itself as it did not have permission to clean the temporarily-generated directory that has perm 000. See below for the code snippet that is buggy.
try { for (int i = 0; i < volumesFailed; i++) { prepareDirToFail(dirs[i]); // this will chmod dirs[i] to perm 000 } restartDatanodes(volumesTolerated, manageDfsDirs); } catch (DiskErrorException e) { ... } finally { ... } assertEquals(expectedBPServiceState, bpServiceState); for (File dir : dirs) { FileUtil.chmod(dir.toString(), "755"); } }
The failure of the statement `assertEquals(expectedBPServiceState, bpServiceState)` caused function to terminate without executing `FileUtil.chmod(dir.toString(), "755")` for each temporary directory with invalid perm 000 the test has created.
Consequence
Any subsequent `mvn test` command would fail to run if this test had failed before. It is because Maven failed to build itself since it does not have permission to clean this temporarily-generated directory. For details of the failure, see below:
[INFO] --- maven-antrun-plugin:1.7:run (create-log-dir) @ hadoop-hdfs --- [INFO] Executing tasks main: [delete] Deleting directory /home/ctest/app/Ctest-Hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 8.349 s [INFO] Finished at: 2019-12-27T03:53:04-06:00 [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goalorg.apache.maven.plugins:maven-antrun-plugin:1.7:run (create-log-dir) on project hadoop-hdfs: An Ant BuildException has occured: Unable to delete directory /home/ctest/app/Ctest-Hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data1/current [ERROR] around Ant part ...<delete dir="/home/ctest/app/Ctest-Hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data"/>... @ 4:105 in /home/ctest/app/Ctest-Hadoop/hadoop-hdfs-project/hadoop-hdfs/target/antrun/build-main.xml [ERROR] -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
Root Cause
The test helper function `org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration#testVolumeConfig` purposely set the directory `/home/ctest/app/Ctest-Hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data1/current` to have perm 000. And at the end of this function, it changed the perm of this directory to 755. However, there is an assertion in this function before the perm was able to changed to 755. Once this assertion fails, the function terminates before the directory’s perm can be changed to 755. Hence, this directory was later unable to be removed by Maven for when executing `mvn test`.
Fix
In `org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration#testVolumeConfig`, move the assertion `assertEquals(expectedBPServiceState, bpServiceState)` to the last line of this function. This fix will fix the bug and will not change the test outcome.