Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-15128

Unit test failing to clean testing data and crashed future Maven test run due to failure in TestDataNodeVolumeFailureToleration



    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 3.2.1
    • Fix Version/s: 3.3.0
    • Component/s: hdfs, test
    • Labels:
    • Hadoop Flags:
    • Flags:


      Actively-used test helper function `testVolumeConfig` in `org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration` chmod a directory with invalid perm 000 for testing purposes but later failed to chmod back this directory with a valid perm if the assertion inside this function failed. Any subsequent `mvn test` command would fail to run if this test had failed before. It is because Maven failed to build itself as it did not have permission to clean the temporarily-generated directory that has perm 000. See below for the code snippet that is buggy.

      try {
            for (int i = 0; i < volumesFailed; i++) {
              prepareDirToFail(dirs[i]); // this will chmod dirs[i] to perm 000
            restartDatanodes(volumesTolerated, manageDfsDirs);
          } catch (DiskErrorException e) {
          } finally {
            assertEquals(expectedBPServiceState, bpServiceState);
            for (File dir : dirs) {
              FileUtil.chmod(dir.toString(), "755");

      The failure of the statement `assertEquals(expectedBPServiceState, bpServiceState)` caused function to terminate without executing `FileUtil.chmod(dir.toString(), "755")` for each temporary directory with invalid perm 000 the test has created. 



      Any subsequent `mvn test` command would fail to run if this test had failed before. It is because Maven failed to build itself since it does not have permission to clean this temporarily-generated directory. For details of the failure, see below:

      [INFO] --- maven-antrun-plugin:1.7:run (create-log-dir) @ hadoop-hdfs ---
      [INFO] Executing tasks
      [delete] Deleting directory /home/ctest/app/Ctest-Hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data
      [INFO] ------------------------------------------------------------------------
      [INFO] ------------------------------------------------------------------------
      [INFO] Total time:  8.349 s
      [INFO] Finished at: 2019-12-27T03:53:04-06:00
      [INFO] ------------------------------------------------------------------------
      [ERROR] Failed to execute goalorg.apache.maven.plugins:maven-antrun-plugin:1.7:run (create-log-dir) on project hadoop-hdfs: An Ant BuildException has occured: Unable to delete directory /home/ctest/app/Ctest-Hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data1/current
      [ERROR] around Ant part ...<delete dir="/home/ctest/app/Ctest-Hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data"/>... @ 4:105 in /home/ctest/app/Ctest-Hadoop/hadoop-hdfs-project/hadoop-hdfs/target/antrun/build-main.xml
      [ERROR] -> [Help 1]
      [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
      [ERROR] Re-run Maven using the -X switch to enable full debug logging.
      [ERROR] For more information about the errors and possible solutions, please read the following articles:
      [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException


      Root Cause

      The test helper function `org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration#testVolumeConfig` purposely set the directory `/home/ctest/app/Ctest-Hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data1/current` to have perm 000. And at the end of this function, it changed the perm of this directory to 755. However, there is an assertion in this function before the perm was able to changed to 755. Once this assertion fails, the function terminates before the directory’s perm can be changed to 755. Hence, this directory was later unable to be removed by Maven for when executing `mvn test`. 



      In `org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration#testVolumeConfig`, move the assertion `assertEquals(expectedBPServiceState, bpServiceState)`  to the last line of this function. This fix will fix the bug and will not change the test outcome. 


        1. HDFS-15128-000.patch
          1.0 kB
        2. HDFS-15128-001.patch
          1 kB



            • Assignee:
              ctest.team Ctest
              ctest.team Ctest
            • Votes:
              0 Vote for this issue
              4 Start watching this issue


              • Created: