Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-12856

FileUtil.checkDest() and RawLocalFileSystem.mkdirs() to throw stricter IOEs; RawLocalFS contract tests to verify

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.8.0
    • Fix Version/s: 2.9.0, 3.0.0-alpha4
    • Component/s: fs
    • Labels:
      None

      Description

      FileUtil.checkDest() throws IOEs when its conditions are not met, with meaningful text.

      The code could be improved if it were to throw PathExistsException and PathIsDirectoryException when appropriate

      1. HADOOP-12856-001.patch
        4 kB
        Steve Loughran
      2. HADOOP-12856-002.patch
        4 kB
        Steve Loughran
      3. HADOOP-12856-003.patch
        4 kB
        Steve Loughran

        Activity

        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11630 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11630/)
        HADOOP-12856. FileUtil.checkDest() and RawLocalFileSystem.mkdirs() to (liuml07: rev d11d280b8b527d572ca0b29ef57434e9ff533ffe)

        • (edit) hadoop-common-project/hadoop-common/src/test/resources/contract/rawlocal.xml
        • (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/rawlocal/RawlocalFSContract.java
        • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileUtil.java
        • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/RawLocalFileSystem.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11630 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11630/ ) HADOOP-12856 . FileUtil.checkDest() and RawLocalFileSystem.mkdirs() to (liuml07: rev d11d280b8b527d572ca0b29ef57434e9ff533ffe) (edit) hadoop-common-project/hadoop-common/src/test/resources/contract/rawlocal.xml (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/rawlocal/RawlocalFSContract.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileUtil.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/RawLocalFileSystem.java
        Hide
        liuml07 Mingliang Liu added a comment -

        Committed to branch-2 and trunk. The findbugs warnings are not related and are being fixed in separate JIRA. Thanks for your contribution Steve Loughran.

        Show
        liuml07 Mingliang Liu added a comment - Committed to branch-2 and trunk . The findbugs warnings are not related and are being fixed in separate JIRA. Thanks for your contribution Steve Loughran .
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 17s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
        +1 mvninstall 15m 9s trunk passed
        +1 compile 17m 2s trunk passed
        +1 checkstyle 0m 39s trunk passed
        +1 mvnsite 1m 15s trunk passed
        +1 mvneclipse 0m 20s trunk passed
        -1 findbugs 1m 32s hadoop-common-project/hadoop-common in trunk has 17 extant Findbugs warnings.
        +1 javadoc 0m 48s trunk passed
        +1 mvninstall 0m 40s the patch passed
        +1 compile 15m 35s the patch passed
        +1 javac 15m 35s the patch passed
        +1 checkstyle 0m 38s the patch passed
        +1 mvnsite 1m 15s the patch passed
        +1 mvneclipse 0m 21s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 2s The patch has no ill-formed XML file.
        +1 findbugs 1m 53s the patch passed
        +1 javadoc 0m 49s the patch passed
        -1 unit 7m 25s hadoop-common in the patch failed.
        +1 asflicense 0m 37s The patch does not generate ASF License warnings.
        68m 18s



        Reason Tests
        Failed junit tests hadoop.ha.TestZKFailoverController



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ac17dc
        JIRA Issue HADOOP-12856
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12864985/HADOOP-12856-003.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
        uname Linux 646d8732d888 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 475f933
        Default Java 1.8.0_121
        findbugs v3.1.0-RC1
        findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/12183/artifact/patchprocess/branch-findbugs-hadoop-common-project_hadoop-common-warnings.html
        unit https://builds.apache.org/job/PreCommit-HADOOP-Build/12183/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12183/testReport/
        modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12183/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 15m 9s trunk passed +1 compile 17m 2s trunk passed +1 checkstyle 0m 39s trunk passed +1 mvnsite 1m 15s trunk passed +1 mvneclipse 0m 20s trunk passed -1 findbugs 1m 32s hadoop-common-project/hadoop-common in trunk has 17 extant Findbugs warnings. +1 javadoc 0m 48s trunk passed +1 mvninstall 0m 40s the patch passed +1 compile 15m 35s the patch passed +1 javac 15m 35s the patch passed +1 checkstyle 0m 38s the patch passed +1 mvnsite 1m 15s the patch passed +1 mvneclipse 0m 21s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. +1 findbugs 1m 53s the patch passed +1 javadoc 0m 49s the patch passed -1 unit 7m 25s hadoop-common in the patch failed. +1 asflicense 0m 37s The patch does not generate ASF License warnings. 68m 18s Reason Tests Failed junit tests hadoop.ha.TestZKFailoverController Subsystem Report/Notes Docker Image:yetus/hadoop:0ac17dc JIRA Issue HADOOP-12856 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12864985/HADOOP-12856-003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 646d8732d888 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 475f933 Default Java 1.8.0_121 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/12183/artifact/patchprocess/branch-findbugs-hadoop-common-project_hadoop-common-warnings.html unit https://builds.apache.org/job/PreCommit-HADOOP-Build/12183/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12183/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12183/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        liuml07 Mingliang Liu added a comment -

        +1

        Show
        liuml07 Mingliang Liu added a comment - +1
        Hide
        stevel@apache.org Steve Loughran added a comment -

        rebuilt against trunk (no changes), retested, and changed exception message to == the previous one for the FileAlreadyExistsException, so it's only the subclass of IOE which changes

        Show
        stevel@apache.org Steve Loughran added a comment - rebuilt against trunk (no changes), retested, and changed exception message to == the previous one for the FileAlreadyExistsException, so it's only the subclass of IOE which changes
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 10s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
        +1 mvninstall 6m 42s trunk passed
        +1 compile 6m 2s trunk passed with JDK v1.8.0_72
        +1 compile 6m 44s trunk passed with JDK v1.7.0_95
        +1 checkstyle 0m 21s trunk passed
        +1 mvnsite 1m 4s trunk passed
        +1 mvneclipse 0m 14s trunk passed
        +1 findbugs 1m 35s trunk passed
        +1 javadoc 0m 51s trunk passed with JDK v1.8.0_72
        +1 javadoc 1m 2s trunk passed with JDK v1.7.0_95
        +1 mvninstall 0m 41s the patch passed
        +1 compile 5m 56s the patch passed with JDK v1.8.0_72
        +1 javac 5m 56s the patch passed
        +1 compile 6m 37s the patch passed with JDK v1.7.0_95
        +1 javac 6m 37s the patch passed
        +1 checkstyle 0m 21s the patch passed
        +1 mvnsite 1m 1s the patch passed
        +1 mvneclipse 0m 13s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        +1 findbugs 1m 49s the patch passed
        +1 javadoc 0m 52s the patch passed with JDK v1.8.0_72
        +1 javadoc 1m 3s the patch passed with JDK v1.7.0_95
        +1 unit 6m 53s hadoop-common in the patch passed with JDK v1.8.0_72.
        +1 unit 7m 11s hadoop-common in the patch passed with JDK v1.7.0_95.
        +1 asflicense 0m 22s Patch does not generate ASF License warnings.
        58m 54s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12790711/HADOOP-12856-002.patch
        JIRA Issue HADOOP-12856
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
        uname Linux dc4a2dbb5274 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / d251e55
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8755/testReport/
        modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8755/console
        Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 10s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 6m 42s trunk passed +1 compile 6m 2s trunk passed with JDK v1.8.0_72 +1 compile 6m 44s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 21s trunk passed +1 mvnsite 1m 4s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 1m 35s trunk passed +1 javadoc 0m 51s trunk passed with JDK v1.8.0_72 +1 javadoc 1m 2s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 41s the patch passed +1 compile 5m 56s the patch passed with JDK v1.8.0_72 +1 javac 5m 56s the patch passed +1 compile 6m 37s the patch passed with JDK v1.7.0_95 +1 javac 6m 37s the patch passed +1 checkstyle 0m 21s the patch passed +1 mvnsite 1m 1s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 1m 49s the patch passed +1 javadoc 0m 52s the patch passed with JDK v1.8.0_72 +1 javadoc 1m 3s the patch passed with JDK v1.7.0_95 +1 unit 6m 53s hadoop-common in the patch passed with JDK v1.8.0_72. +1 unit 7m 11s hadoop-common in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 22s Patch does not generate ASF License warnings. 58m 54s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12790711/HADOOP-12856-002.patch JIRA Issue HADOOP-12856 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux dc4a2dbb5274 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / d251e55 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8755/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8755/console Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        stevel@apache.org Steve Loughran added a comment -

        HADOOP-12856 patch 002. address style and in contract tests strict->false

        Show
        stevel@apache.org Steve Loughran added a comment - HADOOP-12856 patch 002. address style and in contract tests strict->false
        Hide
        stevel@apache.org Steve Loughran added a comment -

        tests failed as this patch actually had strict=true in the tests; the RawLocal throws FileNotFoundException on attempts to overwrite a directory, which is not the exception expected.

        Show
        stevel@apache.org Steve Loughran added a comment - tests failed as this patch actually had strict=true in the tests; the RawLocal throws FileNotFoundException on attempts to overwrite a directory, which is not the exception expected.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 14m 56s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
        +1 mvninstall 6m 41s trunk passed
        +1 compile 6m 33s trunk passed with JDK v1.8.0_72
        +1 compile 6m 56s trunk passed with JDK v1.7.0_95
        +1 checkstyle 0m 21s trunk passed
        +1 mvnsite 1m 5s trunk passed
        +1 mvneclipse 0m 14s trunk passed
        +1 findbugs 1m 40s trunk passed
        +1 javadoc 0m 58s trunk passed with JDK v1.8.0_72
        +1 javadoc 1m 7s trunk passed with JDK v1.7.0_95
        +1 mvninstall 0m 43s the patch passed
        +1 compile 6m 48s the patch passed with JDK v1.8.0_72
        +1 javac 6m 48s the patch passed
        +1 compile 7m 9s the patch passed with JDK v1.7.0_95
        +1 javac 7m 9s the patch passed
        -1 checkstyle 0m 21s hadoop-common-project/hadoop-common: patch generated 1 new + 85 unchanged - 0 fixed = 86 total (was 85)
        +1 mvnsite 1m 3s the patch passed
        +1 mvneclipse 0m 13s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 xml 0m 0s The patch has no ill-formed XML file.
        +1 findbugs 1m 55s the patch passed
        +1 javadoc 0m 59s the patch passed with JDK v1.8.0_72
        +1 javadoc 1m 6s the patch passed with JDK v1.7.0_95
        -1 unit 7m 55s hadoop-common in the patch failed with JDK v1.8.0_72.
        -1 unit 7m 50s hadoop-common in the patch failed with JDK v1.7.0_95.
        +1 asflicense 0m 23s Patch does not generate ASF License warnings.
        78m 4s



        Reason Tests
        JDK v1.8.0_72 Failed junit tests hadoop.fs.contract.rawlocal.TestRawlocalContractCreate
        JDK v1.7.0_95 Failed junit tests hadoop.fs.contract.rawlocal.TestRawlocalContractCreate



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12790576/HADOOP-12856-001.patch
        JIRA Issue HADOOP-12856
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
        uname Linux bb9b0751aa81 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 0fa54d4
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/8748/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
        unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8748/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_72.txt
        unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8748/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt
        unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8748/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/8748/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8748/testReport/
        modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8748/console
        Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 14m 56s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 6m 41s trunk passed +1 compile 6m 33s trunk passed with JDK v1.8.0_72 +1 compile 6m 56s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 21s trunk passed +1 mvnsite 1m 5s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 1m 40s trunk passed +1 javadoc 0m 58s trunk passed with JDK v1.8.0_72 +1 javadoc 1m 7s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 43s the patch passed +1 compile 6m 48s the patch passed with JDK v1.8.0_72 +1 javac 6m 48s the patch passed +1 compile 7m 9s the patch passed with JDK v1.7.0_95 +1 javac 7m 9s the patch passed -1 checkstyle 0m 21s hadoop-common-project/hadoop-common: patch generated 1 new + 85 unchanged - 0 fixed = 86 total (was 85) +1 mvnsite 1m 3s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 0s The patch has no ill-formed XML file. +1 findbugs 1m 55s the patch passed +1 javadoc 0m 59s the patch passed with JDK v1.8.0_72 +1 javadoc 1m 6s the patch passed with JDK v1.7.0_95 -1 unit 7m 55s hadoop-common in the patch failed with JDK v1.8.0_72. -1 unit 7m 50s hadoop-common in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 23s Patch does not generate ASF License warnings. 78m 4s Reason Tests JDK v1.8.0_72 Failed junit tests hadoop.fs.contract.rawlocal.TestRawlocalContractCreate JDK v1.7.0_95 Failed junit tests hadoop.fs.contract.rawlocal.TestRawlocalContractCreate Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12790576/HADOOP-12856-001.patch JIRA Issue HADOOP-12856 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux bb9b0751aa81 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 0fa54d4 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/8748/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8748/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_72.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8748/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8748/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/8748/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8748/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8748/console Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        stevel@apache.org Steve Loughran added a comment -

        patch

        1. tightens the exceptions (They are still IOEs)
        2. Raw local contract tests to load the rawlocal.xml file
        3. rawlocal.xml file synced up with the localfs.xml file

        we can't turn strict exceptions on in the contract tests, as the local filesystems don't do what we want them to do in some failure modes. I manually turned them on and verified that the rename tests worked; it was some tests related to opening files that fail in strict mode.

        Show
        stevel@apache.org Steve Loughran added a comment - patch tightens the exceptions (They are still IOEs) Raw local contract tests to load the rawlocal.xml file rawlocal.xml file synced up with the localfs.xml file we can't turn strict exceptions on in the contract tests, as the local filesystems don't do what we want them to do in some failure modes. I manually turned them on and verified that the rename tests worked; it was some tests related to opening files that fail in strict mode.
        Hide
        stevel@apache.org Steve Loughran added a comment -

        Note how the stack trace shows that this happens in RawLocalFS.rename(). The code has been able to get through the contract tests as the RawLocalContract tests were actually using the contract in localfs.xml, not rawlocal.xml, and it's contract options say fs.contract.supports-strict-exceptions=false,

        Show
        stevel@apache.org Steve Loughran added a comment - Note how the stack trace shows that this happens in RawLocalFS.rename() . The code has been able to get through the contract tests as the RawLocalContract tests were actually using the contract in localfs.xml, not rawlocal.xml, and it's contract options say fs.contract.supports-strict-exceptions=false ,
        Hide
        stevel@apache.org Steve Loughran added a comment -

        This is what the new stack trace would look like

        org.apache.hadoop.fs.PathIsDirectoryException: `file:/tmp/ats/done/1111/0000/000/application_1111_0000/application_1111_0000': Is a directory
        	at org.apache.hadoop.fs.FileUtil.checkDest(FileUtil.java:503)
        	at org.apache.hadoop.fs.FileUtil.checkDest(FileUtil.java:505)
        	at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:351)
        	at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:341)
        	at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:292)
        	at org.apache.hadoop.fs.RawLocalFileSystem.rename(RawLocalFileSystem.java:366)
        	at org.apache.hadoop.fs.ChecksumFileSystem.rename(ChecksumFileSystem.java:507)
        	at org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore$AppLogs.moveToDone(EntityGroupFSTimelineStore.java:694)
        	at org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore$ActiveLogParser.run(EntityGroupFSTimelineStore.java:751)
        	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
        	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
        	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        	at java.lang.Thread.run(Thread.java:745)
        
        Show
        stevel@apache.org Steve Loughran added a comment - This is what the new stack trace would look like org.apache.hadoop.fs.PathIsDirectoryException: `file:/tmp/ats/done/1111/0000/000/application_1111_0000/application_1111_0000': Is a directory at org.apache.hadoop.fs.FileUtil.checkDest(FileUtil.java:503) at org.apache.hadoop.fs.FileUtil.checkDest(FileUtil.java:505) at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:351) at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:341) at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:292) at org.apache.hadoop.fs.RawLocalFileSystem.rename(RawLocalFileSystem.java:366) at org.apache.hadoop.fs.ChecksumFileSystem.rename(ChecksumFileSystem.java:507) at org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore$AppLogs.moveToDone(EntityGroupFSTimelineStore.java:694) at org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore$ActiveLogParser.run(EntityGroupFSTimelineStore.java:751) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang. Thread .run( Thread .java:745)

          People

          • Assignee:
            stevel@apache.org Steve Loughran
            Reporter:
            stevel@apache.org Steve Loughran
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development