Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-14506

Add create() contract test that verifies ancestor dir creation

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 3.0.0-alpha4
    • Fix Version/s: 2.9.0, 3.0.0-alpha4
    • Component/s: fs
    • Labels:
      None

      Description

      The semantics of FileSystem#create() appear to include implicit creation of any parent or ancestor directories in the Path being created, unlike Posix filesystems, which expect the parent directory to already exist.

      S3AFileSystem does implicitly create ancestor directories. It does not currently enforce that any ancestors are directories (HADOOP-13221).

      Deliverable for this JIRA is just a test case added to AbstractContractCreateTest which verifies that missing ancestors are created by create().

      Pulling this test dev work out into separate jira from HADOOP-14457.

      1. HADOOP-14506.001.patch
        2 kB
        Sean Mackrory
      2. HADOOP-14506.002.patch
        2 kB
        Sean Mackrory

        Issue Links

          Activity

          Hide
          andrew.wang Andrew Wang added a comment -

          Please set the trunk fix version too, thanks!

          Show
          andrew.wang Andrew Wang added a comment - Please set the trunk fix version too, thanks!
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11867 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11867/)
          HADOOP-14506. Add create() contract test that verifies ancestor dir (stevel: rev d780a67864d2c687d23b69798c19ee865e38cc5a)

          • (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/AbstractContractCreateTest.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11867 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11867/ ) HADOOP-14506 . Add create() contract test that verifies ancestor dir (stevel: rev d780a67864d2c687d23b69798c19ee865e38cc5a) (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/AbstractContractCreateTest.java
          Hide
          stevel@apache.org Steve Loughran added a comment -

          +1
          committed, thanks

          Show
          stevel@apache.org Steve Loughran added a comment - +1 committed, thanks
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 12m 21s trunk passed
          +1 compile 18m 8s trunk passed
          +1 checkstyle 0m 36s trunk passed
          +1 mvnsite 1m 3s trunk passed
          -1 findbugs 1m 21s hadoop-common-project/hadoop-common in trunk has 19 extant Findbugs warnings.
          +1 javadoc 0m 49s trunk passed
          +1 mvninstall 0m 35s the patch passed
          +1 compile 9m 52s the patch passed
          +1 javac 9m 52s the patch passed
          +1 checkstyle 0m 35s the patch passed
          +1 mvnsite 0m 56s the patch passed
          +1 whitespace 0m 1s The patch has no whitespace issues.
          +1 findbugs 1m 27s the patch passed
          +1 javadoc 0m 45s the patch passed
          -1 unit 7m 2s hadoop-common in the patch failed.
          +1 asflicense 0m 35s The patch does not generate ASF License warnings.
          58m 15s



          Reason Tests
          Failed junit tests hadoop.fs.sftp.TestSFTPFileSystem



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:14b5c93
          JIRA Issue HADOOP-14506
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12872161/HADOOP-14506.002.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 934f0f214b73 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 7c9694c
          Default Java 1.8.0_131
          findbugs v3.1.0-RC1
          findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/12489/artifact/patchprocess/branch-findbugs-hadoop-common-project_hadoop-common-warnings.html
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/12489/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12489/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12489/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 12m 21s trunk passed +1 compile 18m 8s trunk passed +1 checkstyle 0m 36s trunk passed +1 mvnsite 1m 3s trunk passed -1 findbugs 1m 21s hadoop-common-project/hadoop-common in trunk has 19 extant Findbugs warnings. +1 javadoc 0m 49s trunk passed +1 mvninstall 0m 35s the patch passed +1 compile 9m 52s the patch passed +1 javac 9m 52s the patch passed +1 checkstyle 0m 35s the patch passed +1 mvnsite 0m 56s the patch passed +1 whitespace 0m 1s The patch has no whitespace issues. +1 findbugs 1m 27s the patch passed +1 javadoc 0m 45s the patch passed -1 unit 7m 2s hadoop-common in the patch failed. +1 asflicense 0m 35s The patch does not generate ASF License warnings. 58m 15s Reason Tests Failed junit tests hadoop.fs.sftp.TestSFTPFileSystem Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HADOOP-14506 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12872161/HADOOP-14506.002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 934f0f214b73 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 7c9694c Default Java 1.8.0_131 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/12489/artifact/patchprocess/branch-findbugs-hadoop-common-project_hadoop-common-warnings.html unit https://builds.apache.org/job/PreCommit-HADOOP-Build/12489/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12489/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12489/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          mackrorysd Sean Mackrory added a comment -

          Fixing the description and adding text to all asserts.

          Show
          mackrorysd Sean Mackrory added a comment - Fixing the description and adding text to all asserts.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Could you have the asserts include some text (ideally with the relevant status?) I know it makes the code more verbose, but I like to think of whoever will be trying to debug a test failure from a jenkins run and nothing else

          Show
          stevel@apache.org Steve Loughran added a comment - Could you have the asserts include some text (ideally with the relevant status?) I know it makes the code more verbose, but I like to think of whoever will be trying to debug a test failure from a jenkins run and nothing else
          Hide
          fabbri Aaron Fabbri added a comment -

          +1 (non-binding) on test code. If you re-roll patch, maybe improve the describe() text though.

          Show
          fabbri Aaron Fabbri added a comment - +1 (non-binding) on test code. If you re-roll patch, maybe improve the describe() text though.
          Hide
          mackrorysd Sean Mackrory added a comment -

          Adding the test from HADOOP-14457 (I'll put the other test in HADOOP-14457 as its S3Guard-specific). I expand this a bit to test 2 layers of parent directories. Will re-test but the previous iteration had been tested successfully on all filesystems except Aliyun, Swift, and FTP. It passes on S3A with the rest of HADOOP-14457 applied.

          Show
          mackrorysd Sean Mackrory added a comment - Adding the test from HADOOP-14457 (I'll put the other test in HADOOP-14457 as its S3Guard-specific). I expand this a bit to test 2 layers of parent directories. Will re-test but the previous iteration had been tested successfully on all filesystems except Aliyun, Swift, and FTP. It passes on S3A with the rest of HADOOP-14457 applied.

            People

            • Assignee:
              mackrorysd Sean Mackrory
              Reporter:
              fabbri Aaron Fabbri
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development