Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-7947 Fix HDFS unit test failures on Windows
  3. HDFS-8309

Skip unit test using DataNodeTestUtils#injectDataDirFailure() on Windows

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.7.0
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: test
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      As Chris Nauroth noted in HDFS-7917 below, DataNodeTestUtils.injectDataDirFailure() won't work for Windows as rename will fail due to open handles on data node dir. This ticket is opened to skip these tests for Windows.

      Unfortunately, I just remembered that the rename isn't going to work on Windows. It typically doesn't allow you to rename a directory where there are open file handles anywhere in the sub-tree. We'd have to shutdown the DataNode before doing the rename and then start it up. By doing that, we'd be changing the meaning of the test from covering an online failure to covering a failure at DataNode startup, so I don't think we can make that change.

      Below are the two test cases that need to be fixed:

      1. TestDataNodeVolumeFailure#testFailedVolumeBeingRemovedFromDataNode
      2. TestDataNodeHotSwapVolumes.testDirectlyReloadAfterCheckDiskError

        Issue Links

          Activity

          Hide
          xyao Xiaoyu Yao added a comment -

          Attach a patch that ensure all tests that use DataNodeTestUtils.injectDataDirFailure() are skipped for Windows and update comments.

          Show
          xyao Xiaoyu Yao added a comment - Attach a patch that ensure all tests that use DataNodeTestUtils.injectDataDirFailure() are skipped for Windows and update comments.
          Hide
          hadoopqa Hadoop QA added a comment -



          +1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 5m 5s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 3 new or modified test files.
          +1 javac 7m 22s There were no new javac warning messages.
          +1 release audit 0m 20s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 2m 12s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 31s mvn install still works.
          +1 eclipse:eclipse 0m 31s The patch built with eclipse:eclipse.
          +1 findbugs 3m 1s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
          +1 native 1m 18s Pre-build of native portion
          +1 hdfs tests 165m 10s Tests passed in hadoop-hdfs.
              186m 35s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12729894/HDFS-8309.00.patch
          Optional Tests javac unit findbugs checkstyle
          git revision trunk / f1a152c
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/10517/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/10517/testReport/
          Java 1.7.0_55
          uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10517/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 5m 5s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 3 new or modified test files. +1 javac 7m 22s There were no new javac warning messages. +1 release audit 0m 20s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 2m 12s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 31s mvn install still works. +1 eclipse:eclipse 0m 31s The patch built with eclipse:eclipse. +1 findbugs 3m 1s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 native 1m 18s Pre-build of native portion +1 hdfs tests 165m 10s Tests passed in hadoop-hdfs.     186m 35s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12729894/HDFS-8309.00.patch Optional Tests javac unit findbugs checkstyle git revision trunk / f1a152c hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/10517/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/10517/testReport/ Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/10517/console This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          LGTM: +1

          Show
          stevel@apache.org Steve Loughran added a comment - LGTM: +1
          Hide
          xyao Xiaoyu Yao added a comment -

          Thanks Steve Loughran for the review. I will commit this shortly.

          Show
          xyao Xiaoyu Yao added a comment - Thanks Steve Loughran for the review. I will commit this shortly.
          Hide
          xyao Xiaoyu Yao added a comment -

          I've committed the patch to trunk and branch-2. Thanks Steve for the review.

          Show
          xyao Xiaoyu Yao added a comment - I've committed the patch to trunk and branch-2. Thanks Steve for the review.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #7718 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7718/)
          HDFS-8309. Skip unit test using DataNodeTestUtils#injectDataDirFailure() on Windows. (xyao) (xyao: rev a319771d1d9eebaf8e4165dba73383a229cb1525)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeHotSwapVolumes.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailure.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailureReporting.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7718 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7718/ ) HDFS-8309 . Skip unit test using DataNodeTestUtils#injectDataDirFailure() on Windows. (xyao) (xyao: rev a319771d1d9eebaf8e4165dba73383a229cb1525) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeHotSwapVolumes.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailure.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailureReporting.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #916 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/916/)
          HDFS-8309. Skip unit test using DataNodeTestUtils#injectDataDirFailure() on Windows. (xyao) (xyao: rev a319771d1d9eebaf8e4165dba73383a229cb1525)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailureReporting.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailure.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeHotSwapVolumes.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #916 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/916/ ) HDFS-8309 . Skip unit test using DataNodeTestUtils#injectDataDirFailure() on Windows. (xyao) (xyao: rev a319771d1d9eebaf8e4165dba73383a229cb1525) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailureReporting.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailure.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeHotSwapVolumes.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #182 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/182/)
          HDFS-8309. Skip unit test using DataNodeTestUtils#injectDataDirFailure() on Windows. (xyao) (xyao: rev a319771d1d9eebaf8e4165dba73383a229cb1525)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailureReporting.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailure.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeHotSwapVolumes.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #182 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/182/ ) HDFS-8309 . Skip unit test using DataNodeTestUtils#injectDataDirFailure() on Windows. (xyao) (xyao: rev a319771d1d9eebaf8e4165dba73383a229cb1525) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailureReporting.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailure.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeHotSwapVolumes.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #183 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/183/)
          HDFS-8309. Skip unit test using DataNodeTestUtils#injectDataDirFailure() on Windows. (xyao) (xyao: rev a319771d1d9eebaf8e4165dba73383a229cb1525)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailureReporting.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailure.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeHotSwapVolumes.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #183 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/183/ ) HDFS-8309 . Skip unit test using DataNodeTestUtils#injectDataDirFailure() on Windows. (xyao) (xyao: rev a319771d1d9eebaf8e4165dba73383a229cb1525) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailureReporting.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailure.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeHotSwapVolumes.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Hdfs-trunk #2114 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2114/)
          HDFS-8309. Skip unit test using DataNodeTestUtils#injectDataDirFailure() on Windows. (xyao) (xyao: rev a319771d1d9eebaf8e4165dba73383a229cb1525)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailureReporting.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailure.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeHotSwapVolumes.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk #2114 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2114/ ) HDFS-8309 . Skip unit test using DataNodeTestUtils#injectDataDirFailure() on Windows. (xyao) (xyao: rev a319771d1d9eebaf8e4165dba73383a229cb1525) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailureReporting.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailure.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeHotSwapVolumes.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2132 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2132/)
          HDFS-8309. Skip unit test using DataNodeTestUtils#injectDataDirFailure() on Windows. (xyao) (xyao: rev a319771d1d9eebaf8e4165dba73383a229cb1525)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailureReporting.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailure.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeHotSwapVolumes.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2132 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2132/ ) HDFS-8309 . Skip unit test using DataNodeTestUtils#injectDataDirFailure() on Windows. (xyao) (xyao: rev a319771d1d9eebaf8e4165dba73383a229cb1525) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailureReporting.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailure.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeHotSwapVolumes.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Hdfs-trunk-Java8 #173 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/173/)
          HDFS-8309. Skip unit test using DataNodeTestUtils#injectDataDirFailure() on Windows. (xyao) (xyao: rev a319771d1d9eebaf8e4165dba73383a229cb1525)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailure.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailureReporting.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeHotSwapVolumes.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk-Java8 #173 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/173/ ) HDFS-8309 . Skip unit test using DataNodeTestUtils#injectDataDirFailure() on Windows. (xyao) (xyao: rev a319771d1d9eebaf8e4165dba73383a229cb1525) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailure.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailureReporting.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeHotSwapVolumes.java

            People

            • Assignee:
              xyao Xiaoyu Yao
              Reporter:
              xyao Xiaoyu Yao
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development