Details

    • Type: Sub-task
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 2.6.0
    • Fix Version/s: 2.7.0
    • Component/s: fs/s3
    • Labels:
      None

      Description

      The consequence is that mapreduce probably is not splitting s3a files in the expected way. This is similar to HADOOP-5861 (which was for s3n, though s3n was passing 5G rather than 0 for block size).

      FileInputFormat.getSplits() relies on the FileStatus block size being set:

              if (isSplitable(job, path)) {
                long blockSize = file.getBlockSize();
                long splitSize = computeSplitSize(blockSize, minSize, maxSize);
      

      However, S3AFileSystem does not set the FileStatus block size field. From S3AFileStatus.java:

        // Files
        public S3AFileStatus(long length, long modification_time, Path path) {
          super(length, false, 1, 0, modification_time, path);
          isEmptyDirectory = false;
        }
      

      I think it should use S3AFileSystem.getDefaultBlockSize() for each file's block size (where it's currently passing 0).

      1. HADOOP-111584.patch
        3 kB
        Brahma Reddy Battula
      2. HADOOP-11584-002.patch
        5 kB
        Brahma Reddy Battula
      3. HADOOP-10584-003.patch
        12 kB
        Steve Loughran

        Issue Links

          Activity

          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2062 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2062/)
          HADOOP-11584 s3a file block size set to 0 in getFileStatus. (Brahma Reddy Battula via stevel) (stevel: rev 709ff99cff4124823bde631e272af7be9a22f83b)

          • hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3ABlocksize.java
          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileStatus.java
          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-tools/hadoop-aws/src/test/resources/log4j.properties
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2062 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2062/ ) HADOOP-11584 s3a file block size set to 0 in getFileStatus. (Brahma Reddy Battula via stevel) (stevel: rev 709ff99cff4124823bde631e272af7be9a22f83b) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3ABlocksize.java hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileStatus.java hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java hadoop-common-project/hadoop-common/CHANGES.txt hadoop-tools/hadoop-aws/src/test/resources/log4j.properties
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #112 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/112/)
          HADOOP-11584 s3a file block size set to 0 in getFileStatus. (Brahma Reddy Battula via stevel) (stevel: rev 709ff99cff4124823bde631e272af7be9a22f83b)

          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileStatus.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
          • hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3ABlocksize.java
          • hadoop-tools/hadoop-aws/src/test/resources/log4j.properties
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #112 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/112/ ) HADOOP-11584 s3a file block size set to 0 in getFileStatus. (Brahma Reddy Battula via stevel) (stevel: rev 709ff99cff4124823bde631e272af7be9a22f83b) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileStatus.java hadoop-common-project/hadoop-common/CHANGES.txt hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3ABlocksize.java hadoop-tools/hadoop-aws/src/test/resources/log4j.properties
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #103 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/103/)
          HADOOP-11584 s3a file block size set to 0 in getFileStatus. (Brahma Reddy Battula via stevel) (stevel: rev 709ff99cff4124823bde631e272af7be9a22f83b)

          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
          • hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3ABlocksize.java
          • hadoop-tools/hadoop-aws/src/test/resources/log4j.properties
          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileStatus.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #103 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/103/ ) HADOOP-11584 s3a file block size set to 0 in getFileStatus. (Brahma Reddy Battula via stevel) (stevel: rev 709ff99cff4124823bde631e272af7be9a22f83b) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3ABlocksize.java hadoop-tools/hadoop-aws/src/test/resources/log4j.properties hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileStatus.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2044 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2044/)
          HADOOP-11584 s3a file block size set to 0 in getFileStatus. (Brahma Reddy Battula via stevel) (stevel: rev 709ff99cff4124823bde631e272af7be9a22f83b)

          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-tools/hadoop-aws/src/test/resources/log4j.properties
          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileStatus.java
          • hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3ABlocksize.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2044 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2044/ ) HADOOP-11584 s3a file block size set to 0 in getFileStatus. (Brahma Reddy Battula via stevel) (stevel: rev 709ff99cff4124823bde631e272af7be9a22f83b) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java hadoop-common-project/hadoop-common/CHANGES.txt hadoop-tools/hadoop-aws/src/test/resources/log4j.properties hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileStatus.java hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3ABlocksize.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #112 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/112/)
          HADOOP-11584 s3a file block size set to 0 in getFileStatus. (Brahma Reddy Battula via stevel) (stevel: rev 709ff99cff4124823bde631e272af7be9a22f83b)

          • hadoop-tools/hadoop-aws/src/test/resources/log4j.properties
          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileStatus.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
          • hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3ABlocksize.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #112 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/112/ ) HADOOP-11584 s3a file block size set to 0 in getFileStatus. (Brahma Reddy Battula via stevel) (stevel: rev 709ff99cff4124823bde631e272af7be9a22f83b) hadoop-tools/hadoop-aws/src/test/resources/log4j.properties hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileStatus.java hadoop-common-project/hadoop-common/CHANGES.txt hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3ABlocksize.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #846 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/846/)
          HADOOP-11584 s3a file block size set to 0 in getFileStatus. (Brahma Reddy Battula via stevel) (stevel: rev 709ff99cff4124823bde631e272af7be9a22f83b)

          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3ABlocksize.java
          • hadoop-tools/hadoop-aws/src/test/resources/log4j.properties
          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileStatus.java
          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #846 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/846/ ) HADOOP-11584 s3a file block size set to 0 in getFileStatus. (Brahma Reddy Battula via stevel) (stevel: rev 709ff99cff4124823bde631e272af7be9a22f83b) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3ABlocksize.java hadoop-tools/hadoop-aws/src/test/resources/log4j.properties hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileStatus.java hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Thanks Steve Loughran and others for reviews and update.

          Show
          brahmareddy Brahma Reddy Battula added a comment - Thanks Steve Loughran and others for reviews and update.
          Hide
          djannot Denis Jannot added a comment -

          Just build it using the latest source code and it works well.

          I had an issue with Spark creating one input split per byte because of this bug

          Show
          djannot Denis Jannot added a comment - Just build it using the latest source code and it works well. I had an issue with Spark creating one input split per byte because of this bug
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #7173 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7173/)
          HADOOP-11584 s3a file block size set to 0 in getFileStatus. (Brahma Reddy Battula via stevel) (stevel: rev 709ff99cff4124823bde631e272af7be9a22f83b)

          • hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3ABlocksize.java
          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileStatus.java
          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
          • hadoop-tools/hadoop-aws/src/test/resources/log4j.properties
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7173 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7173/ ) HADOOP-11584 s3a file block size set to 0 in getFileStatus. (Brahma Reddy Battula via stevel) (stevel: rev 709ff99cff4124823bde631e272af7be9a22f83b) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3ABlocksize.java hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileStatus.java hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java hadoop-tools/hadoop-aws/src/test/resources/log4j.properties hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Based on own review and tests of others: +1, committing.

          Some of the code in the patch is mine, so I feel I should justify why I am making that vote myself, i.e. am I tainted.

          1. the initial patch did actually fix the bug, so I could have +1'd it there and then
          2. except I wanted tests, which started in -002.patch
          3. The -003 patch, which does involve me, moved the tests into a new class, tested two new corner cases (consistency between stat and ls; blocksize of root).
          4. I was placed to do that as the author of the contract test code and being set up to run s3 tests
          5. it's been reviewed and tested by others, including S3 US

          If anyone feels I shouldn't be committing patch 3, they are free to revert it and take my +1 as a vote for the "no-tests-fix-bugs" patch 001.

          Show
          stevel@apache.org Steve Loughran added a comment - Based on own review and tests of others: +1, committing. Some of the code in the patch is mine, so I feel I should justify why I am making that vote myself, i.e. am I tainted. the initial patch did actually fix the bug, so I could have +1'd it there and then except I wanted tests, which started in -002.patch The -003 patch, which does involve me, moved the tests into a new class, tested two new corner cases (consistency between stat and ls; blocksize of root). I was placed to do that as the author of the contract test code and being set up to run s3 tests it's been reviewed and tested by others, including S3 US If anyone feels I shouldn't be committing patch 3, they are free to revert it and take my +1 as a vote for the "no-tests-fix-bugs" patch 001.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          S3A blocksize is a subclass of the Hadoop 2.5+ fs contract tests, where the pattern is fs.contract.test.fs.$

          {filesystemuri}

          == test FS URI. the other one is just a copy of the s3n naming policy.

          you can use variables in your XML test resources to share the values

          Show
          stevel@apache.org Steve Loughran added a comment - S3A blocksize is a subclass of the Hadoop 2.5+ fs contract tests, where the pattern is fs.contract.test.fs.$ {filesystemuri} == test FS URI. the other one is just a copy of the s3n naming policy. you can use variables in your XML test resources to share the values
          Hide
          eddyxu Lei (Eddy) Xu added a comment -

          Tests passed on AWS US standard region. Non-binding +1.

          Just one small question, why do TestS3ABlocksize and TestS3AFileSystemContract require different s3a fs name ?
          TestS3ABlocksize asks for fs.contract.test.fs.s3a while TestS3AFileSystemContract asks for test.fs.s3a.name?

          Show
          eddyxu Lei (Eddy) Xu added a comment - Tests passed on AWS US standard region. Non-binding +1. Just one small question, why do TestS3ABlocksize and TestS3AFileSystemContract require different s3a fs name ? TestS3ABlocksize asks for fs.contract.test.fs.s3a while TestS3AFileSystemContract asks for test.fs.s3a.name ?
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          That's Great!!! Thanks a lot Thomas Demoor for update....

          Show
          brahmareddy Brahma Reddy Battula added a comment - That's Great!!! Thanks a lot Thomas Demoor for update....
          Hide
          thodemoor Thomas Demoor added a comment -

          tests pass vs AWS EU. LGTM

          Show
          thodemoor Thomas Demoor added a comment - tests pass vs AWS EU. LGTM
          Hide
          stevel@apache.org Steve Loughran added a comment -

          other people will need to test + review this patch. I'm potentially too involved in the code to vote it in (arguable; it's only the tests that I've worked on)

          Show
          stevel@apache.org Steve Loughran added a comment - other people will need to test + review this patch. I'm potentially too involved in the code to vote it in (arguable; it's only the tests that I've worked on)
          Hide
          stevel@apache.org Steve Loughran added a comment -

          thomas: interesting; I saw it on a HADOOP-11570 test run too, which didn't go near this code. Let's open a JIRA just to note it's occurrence -interesting it's on the root dir. We could/should add more diags there, but FSContractTestBase is the Hadoop 1.x FS test, broadly subclassed and not well isolated. Reproducing in a ContractTest would be better. I'm not volunteering...

          Show
          stevel@apache.org Steve Loughran added a comment - thomas: interesting; I saw it on a HADOOP-11570 test run too, which didn't go near this code. Let's open a JIRA just to note it's occurrence -interesting it's on the root dir. We could/should add more diags there, but FSContractTestBase is the Hadoop 1.x FS test, broadly subclassed and not well isolated. Reproducing in a ContractTest would be better. I'm not volunteering...
          Hide
          thodemoor Thomas Demoor added a comment -

          I have seen transient failures on testRenameRootForbidden prior to this patch.

          Show
          thodemoor Thomas Demoor added a comment - I have seen transient failures on testRenameRootForbidden prior to this patch.
          Hide
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12699261/HADOOP-10584-003.patch
          against trunk revision 2f0f756.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-tools/hadoop-aws.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5723//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5723//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12699261/HADOOP-10584-003.patch against trunk revision 2f0f756. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-tools/hadoop-aws. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5723//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5723//console This message is automatically generated.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Hi Steve Loughran

          Thanks a lot for updating the testcase.

          Show
          brahmareddy Brahma Reddy Battula added a comment - Hi Steve Loughran Thanks a lot for updating the testcase.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Tested patch -003 & all hadoop-aws tests against S3 EU tests multiple times', passed.

          One test failed once'; assume a transient network glitch, as it never re-occurred.

          Running org.apache.hadoop.fs.s3a.TestS3AFileSystemContract
          Tests run: 43, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 356.841 sec <<< FAILURE! - in org.apache.hadoop.fs.s3a.TestS3AFileSystemContract
          testRenameRootDirForbidden(org.apache.hadoop.fs.s3a.TestS3AFileSystemContract)  Time elapsed: 2.492 sec  <<< FAILURE!
          junit.framework.AssertionFailedError: Source exists expected:<true> but was:<false>
          	at junit.framework.Assert.fail(Assert.java:57)
          	at junit.framework.Assert.failNotEquals(Assert.java:329)
          	at junit.framework.Assert.assertEquals(Assert.java:78)
          	at junit.framework.Assert.assertEquals(Assert.java:174)
          	at junit.framework.TestCase.assertEquals(TestCase.java:333)
          	at org.apache.hadoop.fs.FileSystemContractBaseTest.rename(FileSystemContractBaseTest.java:490)
          	at org.apache.hadoop.fs.FileSystemContractBaseTest.testRenameRootDirForbidden(FileSystemContractBaseTest.java:598)
          	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          	at java.lang.reflect.Method.invoke(Method.java:606)
          	at junit.framework.TestCase.runTest(TestCase.java:176)
          	at junit.framework.TestCase.runBare(TestCase.java:141)
          	at junit.framework.TestResult$1.protect(TestResult.java:122)
          	at junit.framework.TestResult.runProtected(TestResult.java:142)
          	at junit.framework.TestResult.run(TestResult.java:125)
          	at junit.framework.TestCase.run(TestCase.java:129)
          	at junit.framework.TestSuite.runTest(TestSuite.java:255)
          	at junit.framework.TestSuite.run(TestSuite.java:250)
          	at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:84)
          	at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264)
          	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
          	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
          	at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
          	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
          	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
          

          That assert is from a FileSystem.exists(path) call, which does call getFileStatus(path), so is following a codepath that has now changed. But the exists() call should only fail if the getFileStatus(path) raises a 404, and it never re-occurred. Hence the assumption: *transient*

          Show
          stevel@apache.org Steve Loughran added a comment - Tested patch -003 & all hadoop-aws tests against S3 EU tests multiple times', passed. One test failed once '; assume a transient network glitch, as it never re-occurred. Running org.apache.hadoop.fs.s3a.TestS3AFileSystemContract Tests run: 43, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 356.841 sec <<< FAILURE! - in org.apache.hadoop.fs.s3a.TestS3AFileSystemContract testRenameRootDirForbidden(org.apache.hadoop.fs.s3a.TestS3AFileSystemContract) Time elapsed: 2.492 sec <<< FAILURE! junit.framework.AssertionFailedError: Source exists expected:< true > but was:< false > at junit.framework.Assert.fail(Assert.java:57) at junit.framework.Assert.failNotEquals(Assert.java:329) at junit.framework.Assert.assertEquals(Assert.java:78) at junit.framework.Assert.assertEquals(Assert.java:174) at junit.framework.TestCase.assertEquals(TestCase.java:333) at org.apache.hadoop.fs.FileSystemContractBaseTest.rename(FileSystemContractBaseTest.java:490) at org.apache.hadoop.fs.FileSystemContractBaseTest.testRenameRootDirForbidden(FileSystemContractBaseTest.java:598) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at junit.framework.TestCase.runTest(TestCase.java:176) at junit.framework.TestCase.runBare(TestCase.java:141) at junit.framework.TestResult$1.protect(TestResult.java:122) at junit.framework.TestResult.runProtected(TestResult.java:142) at junit.framework.TestResult.run(TestResult.java:125) at junit.framework.TestCase.run(TestCase.java:129) at junit.framework.TestSuite.runTest(TestSuite.java:255) at junit.framework.TestSuite.run(TestSuite.java:250) at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:84) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124) at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103) That assert is from a FileSystem.exists(path) call, which does call getFileStatus(path) , so is following a codepath that has now changed. But the exists() call should only fail if the getFileStatus(path) raises a 404, and it never re-occurred. Hence the assumption: * transient *
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Patch -003;

          1. src code: cut down lines > 80 chars, some IDE java7 src tuning while at it. no other changes over patches 001 & 002
          2. moved test code to new test, TestS3ABlocksize, extends AbstractFSContractTestBase setup & teardown FS
          3. tests:
          • stat(/dir/file) (from patch -002)
          • ls(dir) has entry for "file", same value as stat(/dir/file)
          • default value non-zero
          • stat("/") has value >= 0
          1. added a log4j test resource...needed to get log output on a failure
          Show
          stevel@apache.org Steve Loughran added a comment - Patch -003; src code: cut down lines > 80 chars, some IDE java7 src tuning while at it. no other changes over patches 001 & 002 moved test code to new test, TestS3ABlocksize , extends AbstractFSContractTestBase setup & teardown FS tests: stat(/dir/file) (from patch -002) ls(dir) has entry for "file", same value as stat(/dir/file) default value non-zero stat("/") has value >= 0 added a log4j test resource...needed to get log output on a failure
          Hide
          stevel@apache.org Steve Loughran added a comment -

          -1
          test doesn't really work.

          will submit new patch

          Show
          stevel@apache.org Steve Loughran added a comment - -1 test doesn't really work. will submit new patch
          Hide
          stevel@apache.org Steve Loughran added a comment -

          jenkins doesn't test this; will run manual test against S3 instance

          Show
          stevel@apache.org Steve Loughran added a comment - jenkins doesn't test this; will run manual test against S3 instance
          Hide
          stevel@apache.org Steve Loughran added a comment -

          +link to HADOOP-11601 which declares that the blocksize of a file status must be >0 for non -empty files; may be 0 for empty ones.

          Show
          stevel@apache.org Steve Loughran added a comment - +link to HADOOP-11601 which declares that the blocksize of a file status must be >0 for non -empty files; may be 0 for empty ones.
          Hide
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12699066/HADOOP-11584-002.patch
          against trunk revision ab0b958.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-tools/hadoop-aws.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5709//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5709//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12699066/HADOOP-11584-002.patch against trunk revision ab0b958. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-tools/hadoop-aws. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5709//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5709//console This message is automatically generated.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Hi Steve Loughran

          Thanks for taking look into this issue..Please check the updated patch which include basic testcase...

          Show
          brahmareddy Brahma Reddy Battula added a comment - Hi Steve Loughran Thanks for taking look into this issue..Please check the updated patch which include basic testcase...
          Hide
          stevel@apache.org Steve Loughran added a comment -

          -1
          Needs a (trivial) test to verify that on a status check the block size != 0

          Show
          stevel@apache.org Steve Loughran added a comment - -1 Needs a (trivial) test to verify that on a status check the block size != 0
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12698681/HADOOP-111584.patch
          against trunk revision ba3c80a.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-tools/hadoop-aws.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5684//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5684//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12698681/HADOOP-111584.patch against trunk revision ba3c80a. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-tools/hadoop-aws. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5684//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5684//console This message is automatically generated.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Attached the patch..

          Show
          brahmareddy Brahma Reddy Battula added a comment - Attached the patch..
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Yes, this will kill the client as it assigns 1 byte/split.

          Show
          stevel@apache.org Steve Loughran added a comment - Yes, this will kill the client as it assigns 1 byte/split.

            People

            • Assignee:
              brahmareddy Brahma Reddy Battula
              Reporter:
              dhecht Dan Hecht
            • Votes:
              0 Vote for this issue
              Watchers:
              15 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development