Details

    • Type: Sub-task
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.6.0
    • Fix Version/s: 2.7.0
    • Component/s: fs/s3
    • Labels:
      None
    • Target Version/s:

      Description

      Currently, S3AInputStream.close() calls S3Object.close(). But, S3Object.close() will read the remaining bytes of the S3 object, potentially transferring a lot of bytes from S3 that are discarded. Instead, the wrapped stream should be aborted to avoid transferring discarded bytes (unless the preceding read() finished at contentLength). For example, reading only the first byte of a 1 GB object and then closing the stream will result in all 1 GB transferred from S3.

      1. HADOOP-11570-001.patch
        2 kB
        Dan Hecht
      2. HADOOP-11570-002.patch
        2 kB
        Dan Hecht

        Issue Links

          Activity

          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2059 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2059/)
          HADOOP-11570. S3AInputStream.close() downloads the remaining bytes of the object from S3. (Dan Hecht via stevel). (stevel: rev 826267f789df657c62f7f5909e5a0b1a7b102c34)

          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AInputStream.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2059 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2059/ ) HADOOP-11570 . S3AInputStream.close() downloads the remaining bytes of the object from S3. (Dan Hecht via stevel). (stevel: rev 826267f789df657c62f7f5909e5a0b1a7b102c34) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AInputStream.java hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #109 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/109/)
          HADOOP-11570. S3AInputStream.close() downloads the remaining bytes of the object from S3. (Dan Hecht via stevel). (stevel: rev 826267f789df657c62f7f5909e5a0b1a7b102c34)

          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AInputStream.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #109 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/109/ ) HADOOP-11570 . S3AInputStream.close() downloads the remaining bytes of the object from S3. (Dan Hecht via stevel). (stevel: rev 826267f789df657c62f7f5909e5a0b1a7b102c34) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AInputStream.java hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Hdfs-trunk #2040 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2040/)
          HADOOP-11570. S3AInputStream.close() downloads the remaining bytes of the object from S3. (Dan Hecht via stevel). (stevel: rev 826267f789df657c62f7f5909e5a0b1a7b102c34)

          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AInputStream.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk #2040 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2040/ ) HADOOP-11570 . S3AInputStream.close() downloads the remaining bytes of the object from S3. (Dan Hecht via stevel). (stevel: rev 826267f789df657c62f7f5909e5a0b1a7b102c34) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AInputStream.java hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #99 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/99/)
          HADOOP-11570. S3AInputStream.close() downloads the remaining bytes of the object from S3. (Dan Hecht via stevel). (stevel: rev 826267f789df657c62f7f5909e5a0b1a7b102c34)

          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AInputStream.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #99 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/99/ ) HADOOP-11570 . S3AInputStream.close() downloads the remaining bytes of the object from S3. (Dan Hecht via stevel). (stevel: rev 826267f789df657c62f7f5909e5a0b1a7b102c34) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AInputStream.java hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #108 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/108/)
          HADOOP-11570. S3AInputStream.close() downloads the remaining bytes of the object from S3. (Dan Hecht via stevel). (stevel: rev 826267f789df657c62f7f5909e5a0b1a7b102c34)

          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AInputStream.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #108 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/108/ ) HADOOP-11570 . S3AInputStream.close() downloads the remaining bytes of the object from S3. (Dan Hecht via stevel). (stevel: rev 826267f789df657c62f7f5909e5a0b1a7b102c34) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AInputStream.java hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #842 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/842/)
          HADOOP-11570. S3AInputStream.close() downloads the remaining bytes of the object from S3. (Dan Hecht via stevel). (stevel: rev 826267f789df657c62f7f5909e5a0b1a7b102c34)

          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AInputStream.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #842 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/842/ ) HADOOP-11570 . S3AInputStream.close() downloads the remaining bytes of the object from S3. (Dan Hecht via stevel). (stevel: rev 826267f789df657c62f7f5909e5a0b1a7b102c34) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AInputStream.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #7128 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7128/)
          HADOOP-11570. S3AInputStream.close() downloads the remaining bytes of the object from S3. (Dan Hecht via stevel). (stevel: rev 826267f789df657c62f7f5909e5a0b1a7b102c34)

          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AInputStream.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7128 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7128/ ) HADOOP-11570 . S3AInputStream.close() downloads the remaining bytes of the object from S3. (Dan Hecht via stevel). (stevel: rev 826267f789df657c62f7f5909e5a0b1a7b102c34) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AInputStream.java hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          stevel@apache.org Steve Loughran added a comment -

          test run successfully locally;

          +1, committing.

          thanks!

          Show
          stevel@apache.org Steve Loughran added a comment - test run successfully locally; +1, committing. thanks!
          Hide
          dhecht Dan Hecht added a comment -

          Hi Steve, I've added patch 002 that does the threshold optimization for close. I'm okay with this, but don't know whether this is important in practice or how much it helps. It'll be pretty workload dependent, but it is simple enough so maybe worthwhile. Which patch do you prefer?

          In any case, I reran the S3A tests using patch 002 as well:

          ------------------------------------------------------
           T E S T S
          -------------------------------------------------------
          Running org.apache.hadoop.fs.s3a.scale.TestS3ADeleteManyFiles
          Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 233.817 sec - in org.apache.hadoop.fs.s3a.scale.TestS3ADeleteManyFiles
          Running org.apache.hadoop.fs.s3a.TestS3AFileSystemContract
          Tests run: 43, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 40.897 sec - in org.apache.hadoop.fs.s3a.TestS3AFileSystemContract
          Running org.apache.hadoop.fs.s3a.TestS3AConfiguration
          Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.383 sec - in org.apache.hadoop.fs.s3a.TestS3AConfiguration
          Running org.apache.hadoop.fs.contract.s3a.TestS3AContractOpen
          Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.457 sec - in org.apache.hadoop.fs.contract.s3a.TestS3AContractOpen
          Running org.apache.hadoop.fs.contract.s3a.TestS3AContractCreate
          Tests run: 6, Failures: 0, Errors: 0, Skipped: 3, Time elapsed: 5.925 sec - in org.apache.hadoop.fs.contract.s3a.TestS3AContractCreate
          Running org.apache.hadoop.fs.contract.s3a.TestS3AContractDelete
          Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.815 sec - in org.apache.hadoop.fs.contract.s3a.TestS3AContractDelete
          Running org.apache.hadoop.fs.contract.s3a.TestS3AContractRename
          Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.911 sec - in org.apache.hadoop.fs.contract.s3a.TestS3AContractRename
          Running org.apache.hadoop.fs.contract.s3a.TestS3AContractSeek
          Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.859 sec - in org.apache.hadoop.fs.contract.s3a.TestS3AContractSeek
          Running org.apache.hadoop.fs.contract.s3a.TestS3AContractRootDir
          Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.593 sec - in org.apache.hadoop.fs.contract.s3a.TestS3AContractRootDir
          Running org.apache.hadoop.fs.contract.s3a.TestS3AContractMkdir
          Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.036 sec - in org.apache.hadoop.fs.contract.s3a.TestS3AContractMkdir
          
          Results :
          
          Tests run: 96, Failures: 0, Errors: 0, Skipped: 3
          
          Show
          dhecht Dan Hecht added a comment - Hi Steve, I've added patch 002 that does the threshold optimization for close. I'm okay with this, but don't know whether this is important in practice or how much it helps. It'll be pretty workload dependent, but it is simple enough so maybe worthwhile. Which patch do you prefer? In any case, I reran the S3A tests using patch 002 as well: ------------------------------------------------------ T E S T S ------------------------------------------------------- Running org.apache.hadoop.fs.s3a.scale.TestS3ADeleteManyFiles Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 233.817 sec - in org.apache.hadoop.fs.s3a.scale.TestS3ADeleteManyFiles Running org.apache.hadoop.fs.s3a.TestS3AFileSystemContract Tests run: 43, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 40.897 sec - in org.apache.hadoop.fs.s3a.TestS3AFileSystemContract Running org.apache.hadoop.fs.s3a.TestS3AConfiguration Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.383 sec - in org.apache.hadoop.fs.s3a.TestS3AConfiguration Running org.apache.hadoop.fs.contract.s3a.TestS3AContractOpen Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.457 sec - in org.apache.hadoop.fs.contract.s3a.TestS3AContractOpen Running org.apache.hadoop.fs.contract.s3a.TestS3AContractCreate Tests run: 6, Failures: 0, Errors: 0, Skipped: 3, Time elapsed: 5.925 sec - in org.apache.hadoop.fs.contract.s3a.TestS3AContractCreate Running org.apache.hadoop.fs.contract.s3a.TestS3AContractDelete Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.815 sec - in org.apache.hadoop.fs.contract.s3a.TestS3AContractDelete Running org.apache.hadoop.fs.contract.s3a.TestS3AContractRename Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.911 sec - in org.apache.hadoop.fs.contract.s3a.TestS3AContractRename Running org.apache.hadoop.fs.contract.s3a.TestS3AContractSeek Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.859 sec - in org.apache.hadoop.fs.contract.s3a.TestS3AContractSeek Running org.apache.hadoop.fs.contract.s3a.TestS3AContractRootDir Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.593 sec - in org.apache.hadoop.fs.contract.s3a.TestS3AContractRootDir Running org.apache.hadoop.fs.contract.s3a.TestS3AContractMkdir Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.036 sec - in org.apache.hadoop.fs.contract.s3a.TestS3AContractMkdir Results : Tests run: 96, Failures: 0, Errors: 0, Skipped: 3
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12698483/HADOOP-11570-002.patch
          against trunk revision 9b0ba59.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-tools/hadoop-aws.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5674//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5674//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12698483/HADOOP-11570-002.patch against trunk revision 9b0ba59. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-tools/hadoop-aws. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5674//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5674//console This message is automatically generated.
          Hide
          dhecht Dan Hecht added a comment -

          This patch adds the threshold so that we close() if we are nearly at the end of the object when it's affordable to discard the remaining bytes, rather than close only when exactly the end of the object.

          Show
          dhecht Dan Hecht added a comment - This patch adds the threshold so that we close() if we are nearly at the end of the object when it's affordable to discard the remaining bytes, rather than close only when exactly the end of the object.
          Hide
          dhecht Dan Hecht added a comment -

          I considered that as well, but didn't know how to choose a good threshold. And I agree that the seek case is more important. So, my preference would be to get this patch committed and then the threshold optimization for seek and/or close could be explored as a separate issue.

          Show
          dhecht Dan Hecht added a comment - I considered that as well, but didn't know how to choose a good threshold. And I agree that the seek case is more important. So, my preference would be to get this patch committed and then the threshold optimization for seek and/or close could be explored as a separate issue.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          OK. Would there be any benefit in having the choice of action move from pos == contentLength to ({{(contentLength -pos) <= threshold) with some small threshold like 1-4K. That way, it'd be cleaner to close files near the end of the stream.

          I don't have enough stats on single-JVM read operations to know if that has any benefit. Making forward seek() operations more efficient is more critical, as the general sequence of an analytics read of a column structured format (ORC) is:

          1. open blob
          2. seek to start of "block"/allocated subset of data
          3. read through with skipping of regions that don't contain columns or ranges of interest
          4. stop at the end of their allocated data subset.
          5. close the stream

          This patch will address the close stream operation.

          Show
          stevel@apache.org Steve Loughran added a comment - OK. Would there be any benefit in having the choice of action move from pos == contentLength to ({{(contentLength -pos) <= threshold) with some small threshold like 1-4K. That way, it'd be cleaner to close files near the end of the stream. I don't have enough stats on single-JVM read operations to know if that has any benefit. Making forward seek() operations more efficient is more critical, as the general sequence of an analytics read of a column structured format (ORC) is: open blob seek to start of "block"/allocated subset of data read through with skipping of regions that don't contain columns or ranges of interest stop at the end of their allocated data subset. close the stream This patch will address the close stream operation.
          Hide
          dhecht Dan Hecht added a comment -

          Correct, the seek case already uses abort(). Additionally, the S3ObjectInputStream.abort() documentation makes it clear that this is the expected tradeoff between abort() and close():

             /**
               * {@inheritDoc}
               *
               * Aborts the underlying http request without reading any more data and
               * closes the stream.
               * <p>
               * By default Apache {@link HttpClient} tries to reuse http connections by
               * reading to the end of an attached input stream on
               * {@link InputStream#close()}. This is efficient from a socket pool
               * management perspective, but for objects with large payloads can incur
               * significant overhead while bytes are read from s3 and discarded. It's up
               * to clients to decide when to take the performance hit implicit in not
               * reusing an http connection in order to not read unnecessary information
               * from S3.
               *
               * @see EofSensorInputStream
               */
          
          Show
          dhecht Dan Hecht added a comment - Correct, the seek case already uses abort(). Additionally, the S3ObjectInputStream.abort() documentation makes it clear that this is the expected tradeoff between abort() and close(): /** * {@inheritDoc} * * Aborts the underlying http request without reading any more data and * closes the stream. * <p> * By default Apache {@link HttpClient} tries to reuse http connections by * reading to the end of an attached input stream on * {@link InputStream#close()}. This is efficient from a socket pool * management perspective, but for objects with large payloads can incur * significant overhead while bytes are read from s3 and discarded. It's up * to clients to decide when to take the performance hit implicit in not * reusing an http connection in order to not read unnecessary information * from S3. * * @see EofSensorInputStream */
          Hide
          thodemoor Thomas Demoor added a comment -

          Seek already correctly aborted the stream rather than closing it (code is in the reopen() method).

          Show
          thodemoor Thomas Demoor added a comment - Seek already correctly aborted the stream rather than closing it (code is in the reopen() method).
          Hide
          stevel@apache.org Steve Loughran added a comment -

          IF this problem exists, won't it also exist if there's a seek() operation which closes the existing stream so it can read elsewhere in the file?

          Show
          stevel@apache.org Steve Loughran added a comment - IF this problem exists, won't it also exist if there's a seek() operation which closes the existing stream so it can read elsewhere in the file?
          Hide
          thodemoor Thomas Demoor added a comment -

          Thanks Dan Hecht for the extra iftop feedback. +1

          Steve Loughran is currently the only active committer around here. He made this issue a subtask of HADOOP-11571 so i'm sure it's on his list.

          Show
          thodemoor Thomas Demoor added a comment - Thanks Dan Hecht for the extra iftop feedback. +1 Steve Loughran is currently the only active committer around here. He made this issue a subtask of HADOOP-11571 so i'm sure it's on his list.
          Hide
          dhecht Dan Hecht added a comment -

          I've run the tests:

          -------------------------------------------------------
           T E S T S
          -------------------------------------------------------
          Running org.apache.hadoop.fs.s3a.scale.TestS3ADeleteManyFiles
          Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 805.836 sec - in org.apache.hadoop.fs.s3a.scale.TestS3ADeleteManyFiles
          Running org.apache.hadoop.fs.s3a.TestS3AFileSystemContract
          Tests run: 43, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 99.061 sec - in org.apache.hadoop.fs.s3a.TestS3AFileSystemContract
          Running org.apache.hadoop.fs.s3a.TestS3AConfiguration
          Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.339 sec - in org.apache.hadoop.fs.s3a.TestS3AConfiguration
          Running org.apache.hadoop.fs.contract.s3a.TestS3AContractOpen
          Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.7 sec - in org.apache.hadoop.fs.contract.s3a.TestS3AContractOpen
          Running org.apache.hadoop.fs.contract.s3a.TestS3AContractCreate
          Tests run: 6, Failures: 0, Errors: 0, Skipped: 3, Time elapsed: 11.965 sec - in org.apache.hadoop.fs.contract.s3a.TestS3AContractCreate
          Running org.apache.hadoop.fs.contract.s3a.TestS3AContractDelete
          Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 20.024 sec - in org.apache.hadoop.fs.contract.s3a.TestS3AContractDelete
          Running org.apache.hadoop.fs.contract.s3a.TestS3AContractRename
          Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 20.032 sec - in org.apache.hadoop.fs.contract.s3a.TestS3AContractRename
          Running org.apache.hadoop.fs.contract.s3a.TestS3AContractSeek
          Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 25.334 sec - in org.apache.hadoop.fs.contract.s3a.TestS3AContractSeek
          Running org.apache.hadoop.fs.contract.s3a.TestS3AContractRootDir
          Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.292 sec - in org.apache.hadoop.fs.contract.s3a.TestS3AContractRootDir
          Running org.apache.hadoop.fs.contract.s3a.TestS3AContractMkdir
          Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.697 sec - in org.apache.hadoop.fs.contract.s3a.TestS3AContractMkdir
          
          Results :
          
          Tests run: 96, Failures: 0, Errors: 0, Skipped: 3
          

          I had also verified that the patch does as expected by watching RX bytes using 'iftop' under various seek/read patterns.

          Show
          dhecht Dan Hecht added a comment - I've run the tests: ------------------------------------------------------- T E S T S ------------------------------------------------------- Running org.apache.hadoop.fs.s3a.scale.TestS3ADeleteManyFiles Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 805.836 sec - in org.apache.hadoop.fs.s3a.scale.TestS3ADeleteManyFiles Running org.apache.hadoop.fs.s3a.TestS3AFileSystemContract Tests run: 43, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 99.061 sec - in org.apache.hadoop.fs.s3a.TestS3AFileSystemContract Running org.apache.hadoop.fs.s3a.TestS3AConfiguration Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.339 sec - in org.apache.hadoop.fs.s3a.TestS3AConfiguration Running org.apache.hadoop.fs.contract.s3a.TestS3AContractOpen Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.7 sec - in org.apache.hadoop.fs.contract.s3a.TestS3AContractOpen Running org.apache.hadoop.fs.contract.s3a.TestS3AContractCreate Tests run: 6, Failures: 0, Errors: 0, Skipped: 3, Time elapsed: 11.965 sec - in org.apache.hadoop.fs.contract.s3a.TestS3AContractCreate Running org.apache.hadoop.fs.contract.s3a.TestS3AContractDelete Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 20.024 sec - in org.apache.hadoop.fs.contract.s3a.TestS3AContractDelete Running org.apache.hadoop.fs.contract.s3a.TestS3AContractRename Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 20.032 sec - in org.apache.hadoop.fs.contract.s3a.TestS3AContractRename Running org.apache.hadoop.fs.contract.s3a.TestS3AContractSeek Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 25.334 sec - in org.apache.hadoop.fs.contract.s3a.TestS3AContractSeek Running org.apache.hadoop.fs.contract.s3a.TestS3AContractRootDir Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.292 sec - in org.apache.hadoop.fs.contract.s3a.TestS3AContractRootDir Running org.apache.hadoop.fs.contract.s3a.TestS3AContractMkdir Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.697 sec - in org.apache.hadoop.fs.contract.s3a.TestS3AContractMkdir Results : Tests run: 96, Failures: 0, Errors: 0, Skipped: 3 I had also verified that the patch does as expected by watching RX bytes using 'iftop' under various seek/read patterns.
          Hide
          thodemoor Thomas Demoor added a comment -

          Source looks good. High impact patch!

          Have you run the tests (by setting auth-keys.xml etc.)? I can give them a try in a couple of days: I want to finish up HADOOP-11522 and HADOOP-9565 first.

          Show
          thodemoor Thomas Demoor added a comment - Source looks good. High impact patch! Have you run the tests (by setting auth-keys.xml etc.)? I can give them a try in a couple of days: I want to finish up HADOOP-11522 and HADOOP-9565 first.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12697670/HADOOP-11570-001.patch
          against trunk revision e0ec071.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-tools/hadoop-aws.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5642//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5642//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12697670/HADOOP-11570-001.patch against trunk revision e0ec071. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-tools/hadoop-aws. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5642//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5642//console This message is automatically generated.
          Hide
          yuzhihong@gmail.com Ted Yu added a comment -

          lgtm

          Show
          yuzhihong@gmail.com Ted Yu added a comment - lgtm

            People

            • Assignee:
              dhecht Dan Hecht
              Reporter:
              dhecht Dan Hecht
            • Votes:
              1 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development