Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.8.0
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: fs/s3
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      S3A has added support for configurable input policies. Similar to fadvise, this configuration provides applications with a way to specify their expected access pattern (sequential or random) while reading a file. S3A then performs optimizations tailored to that access pattern. See site documentation of the fs.s3a.experimental.input.fadvise configuration property for more details. Please be advised that this feature is experimental and subject to backward-incompatible changes in future releases.
      Show
      S3A has added support for configurable input policies. Similar to fadvise, this configuration provides applications with a way to specify their expected access pattern (sequential or random) while reading a file. S3A then performs optimizations tailored to that access pattern. See site documentation of the fs.s3a.experimental.input.fadvise configuration property for more details. Please be advised that this feature is experimental and subject to backward-incompatible changes in future releases.

      Description

      Currently file's "contentLength" is set as the "requestedStreamLen", when invoking S3AInputStream::reopen(). As a part of lazySeek(), sometimes the stream had to be closed and reopened. But lots of times the stream was closed with abort() causing the internal http connection to be unusable. This incurs lots of connection establishment cost in some jobs. It would be good to set the correct value for the stream length to avoid connection aborts.

      I will post the patch once aws tests passes in my machine.

      1. HADOOP-13203-branch-2-010.patch
        57 kB
        Steve Loughran
      2. HADOOP-13203-branch-2-009.patch
        57 kB
        Steve Loughran
      3. HADOOP-13203-branch-2-008.patch
        53 kB
        Steve Loughran
      4. HADOOP-13203-branch-2-007.patch
        42 kB
        Steve Loughran
      5. HADOOP-13203-branch-2-006.patch
        43 kB
        Steve Loughran
      6. HADOOP-13203-branch-2-005.patch
        28 kB
        Steve Loughran
      7. stream_stats.tar.gz
        716 kB
        Rajesh Balamohan
      8. HADOOP-13203-branch-2-004.patch
        6 kB
        Rajesh Balamohan
      9. HADOOP-13203-branch-2-003.patch
        6 kB
        Rajesh Balamohan
      10. HADOOP-13203-branch-2-002.patch
        6 kB
        Rajesh Balamohan
      11. HADOOP-13203-branch-2-001.patch
        3 kB
        Rajesh Balamohan

        Issue Links

          Activity

          Hide
          stevel@apache.org Steve Loughran added a comment -

          So you are proposing some shorter block size for reads, on the basis that it allows for followon GETs to use the same SSL connection?

          How do you know how much to ask for? Or: how do you handle the end of the connection and so start reading the next block? Presumably the cost of that will be lower (reused connection and all), but the stream reading will need to recognise premature EOFs and react

          Show
          stevel@apache.org Steve Loughran added a comment - So you are proposing some shorter block size for reads, on the basis that it allows for followon GETs to use the same SSL connection? How do you know how much to ask for? Or: how do you handle the end of the connection and so start reading the next block? Presumably the cost of that will be lower (reused connection and all), but the stream reading will need to recognise premature EOFs and react
          Hide
          rajesh.balamohan Rajesh Balamohan added a comment -

          Yes Steve Loughran. In workloads like hive, there are lots of random seeks and lots of times the internal connection had to be aborted. It was a lot cheaper to reuse the connection with this patch. Amount of data to be requested for in the request can be determined by "Math.max(targetPos + readahead, (targetPos + length))".

          From the unit tests perspective for aws, following issues were there

          Test timeout failures:

          • TestS3ADeleteManyFiles.testBulkRenameAndDelete
          • org.apache.hadoop.fs.contract.s3a.TestS3AContractDistCp.largeFilesToRemote, largeFilesFromRemote
          • org.apache.hadoop.fs.s3a.scale.TestS3ADeleteManyFiles.testBulkRenameAndDelete

          Other failures

          • org.apache.hadoop.fs.contract.s3a.TestS3AContractRootDir (Root directory operation rejected) - This is already tracked in another jira.
          • org.apache.hadoop.fs.s3a.scale.TestS3AInputStreamPerformance.testReadAheadDefault/testReadBigBlocksBigReadahead (earlier this expected 1 open, but now it can be multiple requestedStreamLen would no longer be the file's length. At the max, we would be able to save a single read ahead call. For rest, it has to open multiple times.
            But this is ok compared with the connection restablishments in real workloads where it can be completely random set of ranges being requested for. E.g hive.). I have not updated the patch to fix this failure. Based on inputs, I can revise the patch.
          Show
          rajesh.balamohan Rajesh Balamohan added a comment - Yes Steve Loughran . In workloads like hive, there are lots of random seeks and lots of times the internal connection had to be aborted. It was a lot cheaper to reuse the connection with this patch. Amount of data to be requested for in the request can be determined by "Math.max(targetPos + readahead, (targetPos + length))". From the unit tests perspective for aws, following issues were there Test timeout failures: TestS3ADeleteManyFiles.testBulkRenameAndDelete org.apache.hadoop.fs.contract.s3a.TestS3AContractDistCp.largeFilesToRemote, largeFilesFromRemote org.apache.hadoop.fs.s3a.scale.TestS3ADeleteManyFiles.testBulkRenameAndDelete Other failures org.apache.hadoop.fs.contract.s3a.TestS3AContractRootDir (Root directory operation rejected) - This is already tracked in another jira. org.apache.hadoop.fs.s3a.scale.TestS3AInputStreamPerformance.testReadAheadDefault/testReadBigBlocksBigReadahead (earlier this expected 1 open, but now it can be multiple requestedStreamLen would no longer be the file's length. At the max, we would be able to save a single read ahead call. For rest, it has to open multiple times. But this is ok compared with the connection restablishments in real workloads where it can be completely random set of ranges being requested for. E.g hive.). I have not updated the patch to fix this failure. Based on inputs, I can revise the patch.
          Hide
          rajesh.balamohan Rajesh Balamohan added a comment - - edited

          Attaching .2 patch with test coverage and also fixed TestS3AInputStreamPerformance.testReadAheadDefault/testReadBigBlocksBigReadAhead. Ran aws s3a tests locally (everything passes with timeout exception in TestS3ADeleteManyFiles, TestS3ADeleteFilesOneByOne, DistCp)

          Show
          rajesh.balamohan Rajesh Balamohan added a comment - - edited Attaching .2 patch with test coverage and also fixed TestS3AInputStreamPerformance.testReadAheadDefault/testReadBigBlocksBigReadAhead. Ran aws s3a tests locally (everything passes with timeout exception in TestS3ADeleteManyFiles, TestS3ADeleteFilesOneByOne, DistCp)
          Hide
          cnauroth Chris Nauroth added a comment -

          Rajesh, thank you for the patch. I have to apologize. I think this might be a regression that traces back to code review feedback I gave Steve on HADOOP-13028:

          https://issues.apache.org/jira/browse/HADOOP-13028?focusedCommentId=15267729&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15267729

          My thinking during the HADOOP-13028 patch was that we might want to keep on reading through the same stream, regardless of the limit of the "current" read call, so we might as well request the whole content in the HTTP request. I was attempting to optimize away extraneous additional HTTP calls. It appears there was an intended side effect.

          I want to make sure I understand the problem here fully. Right now, I don't think I understand why the aborts were happening. Is it because requesting the full content, in combination with Hive's random seek workloads, left the underlying HTTP connection untouched and idle for a long time? Then, after a while, the HTTP connection was deemed inactive/not fully consumed, it assumed there was some kind of client error, and then the whole TCP connection was shut down?

          It's nice to see a comment on the requestedStreamLen calculation. Thank you for adding that. I might ask for some further details to be added to that comment, after I feel like I have a full understanding of the issue.

          Show
          cnauroth Chris Nauroth added a comment - Rajesh, thank you for the patch. I have to apologize. I think this might be a regression that traces back to code review feedback I gave Steve on HADOOP-13028 : https://issues.apache.org/jira/browse/HADOOP-13028?focusedCommentId=15267729&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15267729 My thinking during the HADOOP-13028 patch was that we might want to keep on reading through the same stream, regardless of the limit of the "current" read call, so we might as well request the whole content in the HTTP request. I was attempting to optimize away extraneous additional HTTP calls. It appears there was an intended side effect. I want to make sure I understand the problem here fully. Right now, I don't think I understand why the aborts were happening. Is it because requesting the full content, in combination with Hive's random seek workloads, left the underlying HTTP connection untouched and idle for a long time? Then, after a while, the HTTP connection was deemed inactive/not fully consumed, it assumed there was some kind of client error, and then the whole TCP connection was shut down? It's nice to see a comment on the requestedStreamLen calculation. Thank you for adding that. I might ask for some further details to be added to that comment, after I feel like I have a full understanding of the issue.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 25s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 1m 45s Maven dependency ordering for branch
          +1 mvninstall 6m 40s branch-2 passed
          +1 compile 6m 14s branch-2 passed with JDK v1.8.0_91
          +1 compile 6m 40s branch-2 passed with JDK v1.7.0_101
          +1 checkstyle 1m 21s branch-2 passed
          +1 mvnsite 1m 19s branch-2 passed
          +1 mvneclipse 0m 30s branch-2 passed
          +1 findbugs 2m 12s branch-2 passed
          +1 javadoc 1m 8s branch-2 passed with JDK v1.8.0_91
          +1 javadoc 1m 25s branch-2 passed with JDK v1.7.0_101
          0 mvndep 0m 13s Maven dependency ordering for patch
          +1 mvninstall 0m 58s the patch passed
          +1 compile 6m 5s the patch passed with JDK v1.8.0_91
          +1 javac 6m 5s the patch passed
          +1 compile 6m 38s the patch passed with JDK v1.7.0_101
          +1 javac 6m 38s the patch passed
          -1 checkstyle 1m 26s root: The patch generated 3 new + 22 unchanged - 0 fixed = 25 total (was 22)
          +1 mvnsite 1m 17s the patch passed
          +1 mvneclipse 0m 25s the patch passed
          -1 whitespace 0m 0s The patch has 49 line(s) that end in whitespace. Use git apply --whitespace=fix.
          -1 findbugs 0m 46s hadoop-tools/hadoop-aws generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
          +1 javadoc 1m 7s the patch passed with JDK v1.8.0_91
          +1 javadoc 1m 21s the patch passed with JDK v1.7.0_101
          -1 unit 7m 54s hadoop-common in the patch failed with JDK v1.8.0_91.
          +1 unit 0m 11s hadoop-aws in the patch passed with JDK v1.8.0_91.
          -1 unit 7m 57s hadoop-common in the patch failed with JDK v1.7.0_101.
          +1 unit 0m 23s hadoop-aws in the patch passed with JDK v1.7.0_101.
          +1 asflicense 0m 21s The patch does not generate ASF License warnings.
          70m 1s



          Reason Tests
          FindBugs module:hadoop-tools/hadoop-aws
            Inconsistent synchronization of org.apache.hadoop.fs.s3a.S3AInputStream.readahead; locked 60% of time Unsynchronized access at S3AInputStream.java:60% of time Unsynchronized access at S3AInputStream.java:[line 567]
          JDK v1.8.0_91 Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics
          JDK v1.7.0_101 Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:babe025
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12807708/HADOOP-13203-branch-2-002.patch
          JIRA Issue HADOOP-13203
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 72b43f900f48 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2 / cddf6b4
          Default Java 1.7.0_101
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/9655/artifact/patchprocess/diff-checkstyle-root.txt
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/9655/artifact/patchprocess/whitespace-eol.txt
          findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/9655/artifact/patchprocess/new-findbugs-hadoop-tools_hadoop-aws.html
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9655/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_91.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9655/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_101.txt
          unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/9655/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_91.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/9655/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_101.txt
          JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9655/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9655/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 25s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 1m 45s Maven dependency ordering for branch +1 mvninstall 6m 40s branch-2 passed +1 compile 6m 14s branch-2 passed with JDK v1.8.0_91 +1 compile 6m 40s branch-2 passed with JDK v1.7.0_101 +1 checkstyle 1m 21s branch-2 passed +1 mvnsite 1m 19s branch-2 passed +1 mvneclipse 0m 30s branch-2 passed +1 findbugs 2m 12s branch-2 passed +1 javadoc 1m 8s branch-2 passed with JDK v1.8.0_91 +1 javadoc 1m 25s branch-2 passed with JDK v1.7.0_101 0 mvndep 0m 13s Maven dependency ordering for patch +1 mvninstall 0m 58s the patch passed +1 compile 6m 5s the patch passed with JDK v1.8.0_91 +1 javac 6m 5s the patch passed +1 compile 6m 38s the patch passed with JDK v1.7.0_101 +1 javac 6m 38s the patch passed -1 checkstyle 1m 26s root: The patch generated 3 new + 22 unchanged - 0 fixed = 25 total (was 22) +1 mvnsite 1m 17s the patch passed +1 mvneclipse 0m 25s the patch passed -1 whitespace 0m 0s The patch has 49 line(s) that end in whitespace. Use git apply --whitespace=fix. -1 findbugs 0m 46s hadoop-tools/hadoop-aws generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) +1 javadoc 1m 7s the patch passed with JDK v1.8.0_91 +1 javadoc 1m 21s the patch passed with JDK v1.7.0_101 -1 unit 7m 54s hadoop-common in the patch failed with JDK v1.8.0_91. +1 unit 0m 11s hadoop-aws in the patch passed with JDK v1.8.0_91. -1 unit 7m 57s hadoop-common in the patch failed with JDK v1.7.0_101. +1 unit 0m 23s hadoop-aws in the patch passed with JDK v1.7.0_101. +1 asflicense 0m 21s The patch does not generate ASF License warnings. 70m 1s Reason Tests FindBugs module:hadoop-tools/hadoop-aws   Inconsistent synchronization of org.apache.hadoop.fs.s3a.S3AInputStream.readahead; locked 60% of time Unsynchronized access at S3AInputStream.java:60% of time Unsynchronized access at S3AInputStream.java: [line 567] JDK v1.8.0_91 Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics JDK v1.7.0_101 Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics Subsystem Report/Notes Docker Image:yetus/hadoop:babe025 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12807708/HADOOP-13203-branch-2-002.patch JIRA Issue HADOOP-13203 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 72b43f900f48 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / cddf6b4 Default Java 1.7.0_101 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/9655/artifact/patchprocess/diff-checkstyle-root.txt whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/9655/artifact/patchprocess/whitespace-eol.txt findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/9655/artifact/patchprocess/new-findbugs-hadoop-tools_hadoop-aws.html unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9655/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_91.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9655/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_101.txt unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/9655/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_91.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/9655/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_101.txt JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9655/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9655/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          rajesh.balamohan Rajesh Balamohan added a comment - - edited

          Thanks for the comments Chris Nauroth. In Hive, there can be lots of random seeks. In cases of backwards seek, it had to close the stream and reopen it. As a part of closing the stream, it has to make a decision on
          whether the connection can be re-used or to abort the connection. If it aborts the connection, it becomes un-usable later and subsequent calls have to go through the expensive process of re-establishing the connection.

          For e.g, assume it is reading a 1 MB file and the current position is in 512 KB. If it has to seek back to 128-th KB, it would end up calling close stream. As a part of earlier logic, since file contentLength was set as the requestedStreamLen , it would end up computing as (length - pos > CLOSE_THRESHOLD) (length would be requestedStreamLen in this case). This ended up aborting the connection. So apparently for any backwards seek, it would end up aborting the connection. Time taken to
          establish the connection was far expensive than reading small amount of data being requested for.

          Show
          rajesh.balamohan Rajesh Balamohan added a comment - - edited Thanks for the comments Chris Nauroth . In Hive, there can be lots of random seeks. In cases of backwards seek, it had to close the stream and reopen it. As a part of closing the stream, it has to make a decision on whether the connection can be re-used or to abort the connection. If it aborts the connection, it becomes un-usable later and subsequent calls have to go through the expensive process of re-establishing the connection. For e.g, assume it is reading a 1 MB file and the current position is in 512 KB. If it has to seek back to 128-th KB, it would end up calling close stream. As a part of earlier logic, since file contentLength was set as the requestedStreamLen , it would end up computing as (length - pos > CLOSE_THRESHOLD) (length would be requestedStreamLen in this case). This ended up aborting the connection. So apparently for any backwards seek, it would end up aborting the connection. Time taken to establish the connection was far expensive than reading small amount of data being requested for.
          Hide
          cnauroth Chris Nauroth added a comment -

          Rajesh, thank you for the further explanation. Sorry for my earlier confusion. I was misinterpreting the word "abort" to mean something happening at the TCP layer, e.g. an RST packet sent from the S3 back-end. Now I understand that we're really talking about our own abort logic in S3AInputStream#closeStream.

          Now that I understand the goal of this change, I can code review it. I'll try to do that later today (PST).

          Show
          cnauroth Chris Nauroth added a comment - Rajesh, thank you for the further explanation. Sorry for my earlier confusion. I was misinterpreting the word "abort" to mean something happening at the TCP layer, e.g. an RST packet sent from the S3 back-end. Now I understand that we're really talking about our own abort logic in S3AInputStream#closeStream . Now that I understand the goal of this change, I can code review it. I'll try to do that later today (PST).
          Hide
          cnauroth Chris Nauroth added a comment -
          1. The comment "In case this is set to contentLength, expect lots of connection closes with abort..." is not entirely accurate. I see how this is true for usage that seeks backward, but it's not true for usage that seeks forward a lot, as demonstrated during the HADOOP-13028 review. (More on this topic below.)
          2. Would you please revert the change in S3AInputStream#setReadahead? This is a public API, and the contract of that API is defined in interface CanSetReadahead. It states that callers are allowed to pass null to reset the read-ahead to its default value. This matches the behavior implemented by HDFS. The logic currently in S3A implements it correctly, but with this patch applied, it would cause a NullPointerException if a caller passed null.
          3. In TestS3AInputStreamPerformance, I see why these changes were required to make the tests pass, but it highlights that this change partly reverts what was achieved in HADOOP-13028 to minimize reopens on forward seeks. Before this patch, testReadAheadDefault generated 1 open. After applying the patch, I see it generating 343 opens. It seems we can't fully optimize forward seek without harming backwards seek due to the unintended aborts. I suppose one option would be to introduce an optional advice API, similar to calling fadvise(FADV_SEQUENTIAL) that forward-seeking applications could call. That would be a much bigger change though. I don't see a way to achieve anything better right now, although it's probably good that you changed closeStream to consider read-ahead instead of the old CLOSE_THRESHOLD to determine whether or not to abort. Steve, do you have any further thoughts on this?
          Show
          cnauroth Chris Nauroth added a comment - The comment "In case this is set to contentLength, expect lots of connection closes with abort..." is not entirely accurate. I see how this is true for usage that seeks backward, but it's not true for usage that seeks forward a lot, as demonstrated during the HADOOP-13028 review. (More on this topic below.) Would you please revert the change in S3AInputStream#setReadahead ? This is a public API, and the contract of that API is defined in interface CanSetReadahead . It states that callers are allowed to pass null to reset the read-ahead to its default value. This matches the behavior implemented by HDFS. The logic currently in S3A implements it correctly, but with this patch applied, it would cause a NullPointerException if a caller passed null . In TestS3AInputStreamPerformance , I see why these changes were required to make the tests pass, but it highlights that this change partly reverts what was achieved in HADOOP-13028 to minimize reopens on forward seeks. Before this patch, testReadAheadDefault generated 1 open. After applying the patch, I see it generating 343 opens. It seems we can't fully optimize forward seek without harming backwards seek due to the unintended aborts. I suppose one option would be to introduce an optional advice API, similar to calling fadvise(FADV_SEQUENTIAL) that forward-seeking applications could call. That would be a much bigger change though. I don't see a way to achieve anything better right now, although it's probably good that you changed closeStream to consider read-ahead instead of the old CLOSE_THRESHOLD to determine whether or not to abort. Steve, do you have any further thoughts on this?
          Hide
          rajesh.balamohan Rajesh Balamohan added a comment -

          Thanks Chris Nauroth. Updated the patch.

          1. Removed changes related to setReadAhead.
          2. Minor update to the comments section in reopen
          3. I agree with your comments on forward/backward seeks. Without the patch, it would be difficult to reduce the number of open calls on backward-seeks. Additional option for reducing the number of open calls with forward-seeks can be to set a higher value for readAhead.

          Show
          rajesh.balamohan Rajesh Balamohan added a comment - Thanks Chris Nauroth . Updated the patch. 1. Removed changes related to setReadAhead. 2. Minor update to the comments section in reopen 3. I agree with your comments on forward/backward seeks. Without the patch, it would be difficult to reduce the number of open calls on backward-seeks. Additional option for reducing the number of open calls with forward-seeks can be to set a higher value for readAhead.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 11m 14s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 0m 38s Maven dependency ordering for branch
          +1 mvninstall 6m 24s branch-2 passed
          +1 compile 5m 42s branch-2 passed with JDK v1.8.0_91
          +1 compile 6m 27s branch-2 passed with JDK v1.7.0_101
          +1 checkstyle 1m 23s branch-2 passed
          +1 mvnsite 1m 14s branch-2 passed
          +1 mvneclipse 0m 27s branch-2 passed
          +1 findbugs 2m 8s branch-2 passed
          +1 javadoc 1m 5s branch-2 passed with JDK v1.8.0_91
          +1 javadoc 1m 20s branch-2 passed with JDK v1.7.0_101
          0 mvndep 0m 14s Maven dependency ordering for patch
          +1 mvninstall 0m 56s the patch passed
          +1 compile 5m 48s the patch passed with JDK v1.8.0_91
          +1 javac 5m 48s the patch passed
          +1 compile 6m 22s the patch passed with JDK v1.7.0_101
          +1 javac 6m 22s the patch passed
          -1 checkstyle 1m 21s root: The patch generated 3 new + 21 unchanged - 0 fixed = 24 total (was 21)
          +1 mvnsite 1m 14s the patch passed
          +1 mvneclipse 0m 32s the patch passed
          -1 whitespace 0m 0s The patch has 49 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 findbugs 2m 34s the patch passed
          +1 javadoc 1m 8s the patch passed with JDK v1.8.0_91
          +1 javadoc 1m 26s the patch passed with JDK v1.7.0_101
          +1 unit 8m 27s hadoop-common in the patch passed with JDK v1.7.0_101.
          +1 unit 0m 14s hadoop-aws in the patch passed with JDK v1.7.0_101.
          +1 asflicense 0m 21s The patch does not generate ASF License warnings.
          78m 35s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:babe025
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12808236/HADOOP-13203-branch-2-003.patch
          JIRA Issue HADOOP-13203
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux cd11cfa13ba6 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2 / c11c2ee
          Default Java 1.7.0_101
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/9665/artifact/patchprocess/diff-checkstyle-root.txt
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/9665/artifact/patchprocess/whitespace-eol.txt
          JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9665/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9665/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 11m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 0m 38s Maven dependency ordering for branch +1 mvninstall 6m 24s branch-2 passed +1 compile 5m 42s branch-2 passed with JDK v1.8.0_91 +1 compile 6m 27s branch-2 passed with JDK v1.7.0_101 +1 checkstyle 1m 23s branch-2 passed +1 mvnsite 1m 14s branch-2 passed +1 mvneclipse 0m 27s branch-2 passed +1 findbugs 2m 8s branch-2 passed +1 javadoc 1m 5s branch-2 passed with JDK v1.8.0_91 +1 javadoc 1m 20s branch-2 passed with JDK v1.7.0_101 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 0m 56s the patch passed +1 compile 5m 48s the patch passed with JDK v1.8.0_91 +1 javac 5m 48s the patch passed +1 compile 6m 22s the patch passed with JDK v1.7.0_101 +1 javac 6m 22s the patch passed -1 checkstyle 1m 21s root: The patch generated 3 new + 21 unchanged - 0 fixed = 24 total (was 21) +1 mvnsite 1m 14s the patch passed +1 mvneclipse 0m 32s the patch passed -1 whitespace 0m 0s The patch has 49 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 findbugs 2m 34s the patch passed +1 javadoc 1m 8s the patch passed with JDK v1.8.0_91 +1 javadoc 1m 26s the patch passed with JDK v1.7.0_101 +1 unit 8m 27s hadoop-common in the patch passed with JDK v1.7.0_101. +1 unit 0m 14s hadoop-aws in the patch passed with JDK v1.7.0_101. +1 asflicense 0m 21s The patch does not generate ASF License warnings. 78m 35s Subsystem Report/Notes Docker Image:yetus/hadoop:babe025 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12808236/HADOOP-13203-branch-2-003.patch JIRA Issue HADOOP-13203 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux cd11cfa13ba6 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / c11c2ee Default Java 1.7.0_101 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/9665/artifact/patchprocess/diff-checkstyle-root.txt whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/9665/artifact/patchprocess/whitespace-eol.txt JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9665/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9665/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          It looks like, as people note, the move may make forward seeking, or a mix of seek + read() calls more expensive.More specifically, it could well accelerate a sequence of readFully() offset calls, but not handle so well situations of ) + read(pos, n) + seek(pos + n + n2) , stuff the forward skipping could handle.

          Even regarding readFully() calls, it isn't going to handle well any mix of read()+readFully(), as the first read will have triggered a to-end-of-file read.

          It seems to me that one could actually do something of both where all reads specified a block length, such as 64KB. On sustained forward reads, when the boundary was triggered it'd read forward. On mixed seek/read operations, ones where the range of the read is unknown, this would significantly optimise any random access use, rather than those which exclusively used on read operation.

          And here's the problem: right now we don't know what are the API/file use modes in widespread use against s3. We don't have the data. I can see what you're highlighting: the current mechanism is very expensive for backwards seeks —but we have just optimised forward seeking and instrumented the code to collect detail on what's actually going on.

          1. I don't want to rush into a change which has the potential to make some existing codepaths worse —especially as we don't know how the FS gets used.
          2. I'd really like to see collected statistics on FS usage across a broad dataset. Anyone here is welcome to contribute to this —it should include statistics gathered in downstream use.

          I'm very tempted to argue this should be an S3a phase III improvement: it has ramifications, and we should do it well. We are, with the metrics, in a position to understand those ramifications and, if not in a rush, implement something which works well for a broad set of uses

          Show
          stevel@apache.org Steve Loughran added a comment - It looks like, as people note, the move may make forward seeking, or a mix of seek + read() calls more expensive.More specifically, it could well accelerate a sequence of readFully() offset calls, but not handle so well situations of ) + read(pos, n) + seek(pos + n + n2) , stuff the forward skipping could handle. Even regarding readFully() calls, it isn't going to handle well any mix of read()+readFully(), as the first read will have triggered a to-end-of-file read. It seems to me that one could actually do something of both where all reads specified a block length, such as 64KB. On sustained forward reads, when the boundary was triggered it'd read forward. On mixed seek/read operations, ones where the range of the read is unknown, this would significantly optimise any random access use, rather than those which exclusively used on read operation. And here's the problem: right now we don't know what are the API/file use modes in widespread use against s3. We don't have the data. I can see what you're highlighting: the current mechanism is very expensive for backwards seeks —but we have just optimised forward seeking and instrumented the code to collect detail on what's actually going on. I don't want to rush into a change which has the potential to make some existing codepaths worse —especially as we don't know how the FS gets used. I'd really like to see collected statistics on FS usage across a broad dataset. Anyone here is welcome to contribute to this —it should include statistics gathered in downstream use. I'm very tempted to argue this should be an S3a phase III improvement: it has ramifications, and we should do it well. We are, with the metrics, in a position to understand those ramifications and, if not in a rush, implement something which works well for a broad set of uses
          Hide
          cnauroth Chris Nauroth added a comment -

          Rajesh, thank you for patch 003. That addresses my comments 1 and 2, though it looks like we still need to come to consensus on point 3 (optimized forward seek/scan vs. optimized backward seek).

          Show
          cnauroth Chris Nauroth added a comment - Rajesh, thank you for patch 003. That addresses my comments 1 and 2, though it looks like we still need to come to consensus on point 3 (optimized forward seek/scan vs. optimized backward seek).
          Hide
          stevel@apache.org Steve Loughran added a comment -

          I'm thinking of something more sophisticated, which I'm willing to do, it's just a phase III kind of problem (post hadoop 2.8)

          1. we have the notion of a read block size, say 64KB. This block size should be consistent with the block sizes used in the aws code/httpclient
          2. we always read aligned with the block size.
          3. for a simple seek()/read() at position P, the read would be from P to the next block size > P.
          4. if the number of bytes to be read is known seek+read(bytes), read-fully, read-positioned, positioned read-fully, we'd read up to the next block size, or, if the full read would span a block, up to the next block past the final read position.
          5. During a read, an EOF exception would trigger a new read (and/or the latest block size is tracked and managed in the S3AInputStream
          6. whenever a seek/positioned read in a new location is needed, the data up to the end of the next block is read in.
          7. for forward seeks, where the data is in the current block, skip the bytes
          8. for forward seeks where the data is in a later block, read to the end of the block, then to a read from the next location to the end of that block.

          It means that for forward scan through the file, the number of blocks read in a file is file/blocksize.
          For backward seeks of any kind, the amount of data read is the remainder of the current block + the data read.
          For forward seeks, if the data is in block, the amount of data is readLocation-currentLocation
          For forward seeks, if the data is not in the block, the cost of a seek equals that of a backward seek.

          So: short hop forward seeks and sequential reading is not very expensive; backwards and long-distance forward seeks have a predictable cost —one which is the same irrespective of the destination of the seek.

          Show
          stevel@apache.org Steve Loughran added a comment - I'm thinking of something more sophisticated, which I'm willing to do, it's just a phase III kind of problem (post hadoop 2.8) we have the notion of a read block size, say 64KB. This block size should be consistent with the block sizes used in the aws code/httpclient we always read aligned with the block size. for a simple seek()/read() at position P, the read would be from P to the next block size > P. if the number of bytes to be read is known seek+read(bytes), read-fully, read-positioned, positioned read-fully , we'd read up to the next block size, or, if the full read would span a block, up to the next block past the final read position. During a read, an EOF exception would trigger a new read (and/or the latest block size is tracked and managed in the S3AInputStream whenever a seek/positioned read in a new location is needed, the data up to the end of the next block is read in. for forward seeks, where the data is in the current block, skip the bytes for forward seeks where the data is in a later block, read to the end of the block, then to a read from the next location to the end of that block. It means that for forward scan through the file, the number of blocks read in a file is file/blocksize . For backward seeks of any kind, the amount of data read is the remainder of the current block + the data read. For forward seeks, if the data is in block, the amount of data is readLocation-currentLocation For forward seeks, if the data is not in the block, the cost of a seek equals that of a backward seek. So: short hop forward seeks and sequential reading is not very expensive; backwards and long-distance forward seeks have a predictable cost —one which is the same irrespective of the destination of the seek.
          Hide
          rajesh.balamohan Rajesh Balamohan added a comment -

          Thanks for sharing the details. Would that mean GetObjectRequest.withRange(targetPos, requestedStreamLen) would be block aligned as well?.
          Forward seek cost in latest patch is fileLen/blockSize. Backward seek is soft close of the current connection (by reading the remaining data till the block size) and requesting for fresh data.

          Show
          rajesh.balamohan Rajesh Balamohan added a comment - Thanks for sharing the details. Would that mean GetObjectRequest.withRange(targetPos, requestedStreamLen) would be block aligned as well?. Forward seek cost in latest patch is fileLen/blockSize . Backward seek is soft close of the current connection (by reading the remaining data till the block size) and requesting for fresh data.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          bq, Would that mean GetObjectRequest.withRange(targetPos, requestedStreamLen) would be block aligned as well?.

          yes, I think we'd always round up the end of a request to the next block (or EOF, whichever comes first). That way, if there is another positioned read directly or within that same block as the previous one: no need to make another request.

          Show
          stevel@apache.org Steve Loughran added a comment - bq, Would that mean GetObjectRequest.withRange(targetPos, requestedStreamLen) would be block aligned as well?. yes, I think we'd always round up the end of a request to the next block (or EOF, whichever comes first). That way, if there is another positioned read directly or within that same block as the previous one: no need to make another request.
          Hide
          rajesh.balamohan Rajesh Balamohan added a comment -

          There is a corner case, wherein closing the stream should make use of requestedStreamLen instead of contentLength to avoid connection abort. This would be visible in long running services in the cluster tries to access this codepath. Fixed this in the latest patch.

          Also, got the stream access profiles for couple of TPC-DS and TPC-H queries, wherein I printed the stream statistics during close in the cluster where i tested it. Attaching those logs here with. Please note that this was done with ORC data format which tries to read the footer and then starts reading the stripe information.

          1. In TPC-DS most of the files are small so they end up having single backwards seeks during file reading. I.e Reader reads
          the postscript/footer/meta details as the first operation and then seeks backwards to read the data portion of the file. Without the patch, it would abort the connection as the difference between file length and the current position would be much higher than CLOSE_THRESHOLD.

          e.g log

          2016-06-15 09:00:31,546 [INFO] [TezChild] |s3a.S3AFileSystem|: S3AInputStream{s3a://xyz/tpcds_bin_partitioned_orc_200.db/store_sales/ss_sold_date_sk=2450967/000456_0 pos=4162453 nextReadPos=4162453 contentLength=7630589 StreamStatistics{OpenOperations=4, CloseOperations=4, Closed=4, Aborted=0, SeekOperations=3, ReadExceptions=0, ForwardSeekOperations=2, BackwardSeekOperations=1, BytesSkippedOnSeek=5963, BytesBackwardsOnSeek=7629525, BytesRead=740946, BytesRead excluding skipped=734983, ReadOperations=91, ReadFullyOperations=0, ReadsIncomplete=85}}
          

          There are file accesses without any backward seeks, where in they access standard 16KB information to read the footer details and closes the file without any additional reads.
          e.g log

          2016-06-15 09:00:28,590 [INFO] [TezChild] |s3a.S3AFileSystem|: S3AInputStream{s3a://xyz/tpcds_bin_partitioned_orc_200.db/store_sales/ss_sold_date_sk=2450993/000213_0 pos=7549954 nextReadPos=7549954 contentLength=7549954 StreamStatistics{OpenOperations=1, CloseOperations=1, Closed=1, Aborted=0, SeekOperations=0, ReadExceptions=0, ForwardSeekOperations=0, BackwardSeekOperations=0, BytesSkippedOnSeek=0, BytesBackwardsOnSeek=0, BytesRead=16384, BytesRead excluding skipped=16384, ReadOperations=1, ReadFullyOperations=0, ReadsIncomplete=0}}
          

          2. In TPC-H dataset, relatively large files are present (e.g each file in lineitem dataset would be around 1 GB in size in the overall 1 TB tpc-h dataset). In such cases, equal amount of forward-seeks and backward-seeks happen (e.g around 24 times in per file in the log). Patch avoids connection aborts with backward seeks.
          e.g log

          2016-06-15 09:26:26,671 [INFO] [TezChild] |s3a.S3AFileSystem|: S3AInputStream{s3a://xyz/tpch_flat_orc_1000.db/lineitem/000041_0 pos=728756230 nextReadPos=728756230 contentLength=739566852 StreamStatistics{OpenOperations=72, CloseOperations=72, Closed=72, Aborted=0, SeekOperations=48, ReadExceptions=0, ForwardSeekOperations=24, BackwardSeekOperations=24, BytesSkippedOnSeek=167662, BytesBackwardsOnSeek=737556392, BytesRead=244894978, BytesRead excluding skipped=244727316, ReadOperations=28457, ReadFullyOperations=0, ReadsIncomplete=28217}}
          
          Show
          rajesh.balamohan Rajesh Balamohan added a comment - There is a corner case, wherein closing the stream should make use of requestedStreamLen instead of contentLength to avoid connection abort. This would be visible in long running services in the cluster tries to access this codepath. Fixed this in the latest patch. Also, got the stream access profiles for couple of TPC-DS and TPC-H queries, wherein I printed the stream statistics during close in the cluster where i tested it. Attaching those logs here with. Please note that this was done with ORC data format which tries to read the footer and then starts reading the stripe information. 1. In TPC-DS most of the files are small so they end up having single backwards seeks during file reading. I.e Reader reads the postscript/footer/meta details as the first operation and then seeks backwards to read the data portion of the file. Without the patch, it would abort the connection as the difference between file length and the current position would be much higher than CLOSE_THRESHOLD. e.g log 2016-06-15 09:00:31,546 [INFO] [TezChild] |s3a.S3AFileSystem|: S3AInputStream{s3a://xyz/tpcds_bin_partitioned_orc_200.db/store_sales/ss_sold_date_sk=2450967/000456_0 pos=4162453 nextReadPos=4162453 contentLength=7630589 StreamStatistics{OpenOperations=4, CloseOperations=4, Closed=4, Aborted=0, SeekOperations=3, ReadExceptions=0, ForwardSeekOperations=2, BackwardSeekOperations=1, BytesSkippedOnSeek=5963, BytesBackwardsOnSeek=7629525, BytesRead=740946, BytesRead excluding skipped=734983, ReadOperations=91, ReadFullyOperations=0, ReadsIncomplete=85}} There are file accesses without any backward seeks, where in they access standard 16KB information to read the footer details and closes the file without any additional reads. e.g log 2016-06-15 09:00:28,590 [INFO] [TezChild] |s3a.S3AFileSystem|: S3AInputStream{s3a://xyz/tpcds_bin_partitioned_orc_200.db/store_sales/ss_sold_date_sk=2450993/000213_0 pos=7549954 nextReadPos=7549954 contentLength=7549954 StreamStatistics{OpenOperations=1, CloseOperations=1, Closed=1, Aborted=0, SeekOperations=0, ReadExceptions=0, ForwardSeekOperations=0, BackwardSeekOperations=0, BytesSkippedOnSeek=0, BytesBackwardsOnSeek=0, BytesRead=16384, BytesRead excluding skipped=16384, ReadOperations=1, ReadFullyOperations=0, ReadsIncomplete=0}} 2. In TPC-H dataset, relatively large files are present (e.g each file in lineitem dataset would be around 1 GB in size in the overall 1 TB tpc-h dataset). In such cases, equal amount of forward-seeks and backward-seeks happen (e.g around 24 times in per file in the log). Patch avoids connection aborts with backward seeks. e.g log 2016-06-15 09:26:26,671 [INFO] [TezChild] |s3a.S3AFileSystem|: S3AInputStream{s3a://xyz/tpch_flat_orc_1000.db/lineitem/000041_0 pos=728756230 nextReadPos=728756230 contentLength=739566852 StreamStatistics{OpenOperations=72, CloseOperations=72, Closed=72, Aborted=0, SeekOperations=48, ReadExceptions=0, ForwardSeekOperations=24, BackwardSeekOperations=24, BytesSkippedOnSeek=167662, BytesBackwardsOnSeek=737556392, BytesRead=244894978, BytesRead excluding skipped=244727316, ReadOperations=28457, ReadFullyOperations=0, ReadsIncomplete=28217}}
          Hide
          stevel@apache.org Steve Loughran added a comment -

          -1

          I accept your contention that it works well for your benchmark. However it does so at the expense of being pathologically bad for anything trying to read a file in different ways. I can demonstrate this with the log for one of my SPARK-7481 runs. Essentially, to read a 20MB .csv.gz file has gone from < 20s to about 6 minutes: 20x slower.

          2016-06-16 18:34:21,350 INFO  scheduler.DAGScheduler (Logging.scala:logInfo(58)) - Job 0 finished: count at S3LineCount.scala:99, took 350.460510 s
          2016-06-16 18:34:21,355 INFO  examples.S3LineCount (Logging.scala:logInfo(58)) - Duration of  count s3a://landsat-pds/scene_list.gz = 350,666,373,013 ns
          2016-06-16 18:34:21,355 INFO  examples.S3LineCount (Logging.scala:logInfo(58)) - line count = 514524
          2016-06-16 18:34:21,356 INFO  examples.S3LineCount (Logging.scala:logInfo(58)) - File System = S3AFileSystem{uri=s3a://landsat-pds, workingDir=s3a://landsat-pds/user/stevel, partSize=5242880, enableMultiObjectsDelete=true, maxKeys=5000, readAhead=65536, blockSize=1048576, multiPartThreshold=5242880, statistics {22626526 bytes read, 0 bytes written, 3 read ops, 0 large read ops, 0 write ops}, metrics {{Context=S3AFileSystem} {FileSystemId=03e96d8b-c5d4-4b3c-8b9d-04931588912b-landsat-pds} {fsURI=s3a://landsat-pds/scene_list.gz} {files_created=0} {files_copied=0} {files_copied_bytes=0} {files_deleted=0} {directories_created=0} {directories_deleted=0} {ignored_errors=1} {invocations_copyfromlocalfile=0} {invocations_exists=0} {invocations_getfilestatus=3} {invocations_globstatus=1} {invocations_is_directory=0} {invocations_is_file=0} {invocations_listfiles=0} {invocations_listlocatedstatus=0} {invocations_liststatus=0} {invocations_mdkirs=0} {invocations_rename=0} {object_copy_requests=0} {object_delete_requests=0} {object_list_requests=0} {object_metadata_requests=3} {object_multipart_aborted=0} {object_put_bytes=0} {object_put_requests=0} {streamReadOperations=1584} {streamForwardSeekOperations=0} {streamBytesRead=22626526} {streamSeekOperations=0} {streamReadExceptions=0} {streamOpened=1584} {streamReadOperationsIncomplete=1584} {streamAborted=0} {streamReadFullyOperations=0} {streamClosed=1584} {streamBytesSkippedOnSeek=0} {streamCloseOperations=1584} {streamBytesBackwardsOnSeek=0} {streamBackwardSeekOperations=0} }}
          
          

          And without the patch

          2016-06-16 18:37:55,688 INFO  scheduler.DAGScheduler (Logging.scala:logInfo(58)) - Job 0 finished: count at S3LineCount.scala:99, took 15.853566 s
          2016-06-16 18:37:55,693 INFO  examples.S3LineCount (Logging.scala:logInfo(58)) - Duration of  count s3a://landsat-pds/scene_list.gz = 16,143,975,760 ns
          2016-06-16 18:37:55,693 INFO  examples.S3LineCount (Logging.scala:logInfo(58)) - line count = 514524
          2016-06-16 18:37:55,694 INFO  examples.S3LineCount (Logging.scala:logInfo(58)) - File System = S3AFileSystem{uri=s3a://landsat-pds, workingDir=s3a://landsat-pds/user/stevel, partSize=5242880, enableMultiObjectsDelete=true, maxKeys=5000, readAhead=65536, blockSize=1048576, multiPartThreshold=5242880, statistics {22626526 bytes read, 0 bytes written, 3 read ops, 0 large read ops, 0 write ops}, metrics {{Context=S3AFileSystem} {FileSystemId=96650849-6e33-441f-a976-e74443239ad6-landsat-pds} {fsURI=s3a://landsat-pds/scene_list.gz} {files_created=0} {files_copied=0} {files_copied_bytes=0} {files_deleted=0} {directories_created=0} {directories_deleted=0} {ignored_errors=1} {invocations_copyfromlocalfile=0} {invocations_exists=0} {invocations_getfilestatus=3} {invocations_globstatus=1} {invocations_is_directory=0} {invocations_is_file=0} {invocations_listfiles=0} {invocations_listlocatedstatus=0} {invocations_liststatus=0} {invocations_mdkirs=0} {invocations_rename=0} {object_copy_requests=0} {object_delete_requests=0} {object_list_requests=0} {object_metadata_requests=3} {object_multipart_aborted=0} {object_put_bytes=0} {object_put_requests=0} {streamReadOperations=2601} {streamForwardSeekOperations=0} {streamBytesRead=22626526} {streamSeekOperations=0} {streamReadExceptions=0} {streamOpened=1} {streamReadOperationsIncomplete=2601} {streamAborted=0} {streamReadFullyOperations=0} {streamClosed=1} {streamBytesSkippedOnSeek=0} {streamCloseOperations=1} {streamBytesBackwardsOnSeek=0} {streamBackwardSeekOperations=0} }}
          2016-06-16 18:37:55,694 INFO  examples.S3LineCount (Logging.scala:logInfo(58)) - Stopping Spark Context
          

          The test, I believe, simply reads in the whole file: no seeks, no skipping. I see the no of stream opened calls has gone from 1 to 1584. I suspect that is what's at play here

          I think this code needs what I suggested, some block mechanism which works with open read() calls along with read operations where the full length is known. It also needs to handle the scenario of a read(bytes[]) which overshoots the currently being read block, not by closing the current read and discarding the data, but by reading in all the data it can from the current block, then starting to read the new block.

          Also

          • The default block size needs to be significantly bigger
          • A new S3Scale test to do the full byte-by-byte scan through the file; this will pick up performance problems immediately. ( i may put that in myself anyway, to catch similar problems in other patches)
          • I think some more instrumentation would be good here. Specifically "bytes from current read discarded".
          Show
          stevel@apache.org Steve Loughran added a comment - -1 I accept your contention that it works well for your benchmark. However it does so at the expense of being pathologically bad for anything trying to read a file in different ways. I can demonstrate this with the log for one of my SPARK-7481 runs. Essentially, to read a 20MB .csv.gz file has gone from < 20s to about 6 minutes: 20x slower. 2016-06-16 18:34:21,350 INFO scheduler.DAGScheduler (Logging.scala:logInfo(58)) - Job 0 finished: count at S3LineCount.scala:99, took 350.460510 s 2016-06-16 18:34:21,355 INFO examples.S3LineCount (Logging.scala:logInfo(58)) - Duration of count s3a: //landsat-pds/scene_list.gz = 350,666,373,013 ns 2016-06-16 18:34:21,355 INFO examples.S3LineCount (Logging.scala:logInfo(58)) - line count = 514524 2016-06-16 18:34:21,356 INFO examples.S3LineCount (Logging.scala:logInfo(58)) - File System = S3AFileSystem{uri=s3a: //landsat-pds, workingDir=s3a://landsat-pds/user/stevel, partSize=5242880, enableMultiObjectsDelete= true , maxKeys=5000, readAhead=65536, blockSize=1048576, multiPartThreshold=5242880, statistics {22626526 bytes read, 0 bytes written, 3 read ops, 0 large read ops, 0 write ops}, metrics {{Context=S3AFileSystem} {FileSystemId=03e96d8b-c5d4-4b3c-8b9d-04931588912b-landsat-pds} {fsURI=s3a://landsat-pds/scene_list.gz} {files_created=0} {files_copied=0} {files_copied_bytes=0} {files_deleted=0} {directories_created=0} {directories_deleted=0} {ignored_errors=1} {invocations_copyfromlocalfile=0} {invocations_exists=0} {invocations_getfilestatus=3} {invocations_globstatus=1} {invocations_is_directory=0} {invocations_is_file=0} {invocations_listfiles=0} {invocations_listlocatedstatus=0} {invocations_liststatus=0} {invocations_mdkirs=0} {invocations_rename=0} {object_copy_requests=0} {object_delete_requests=0} {object_list_requests=0} {object_metadata_requests=3} {object_multipart_aborted=0} {object_put_bytes=0} {object_put_requests=0} {streamReadOperations=1584} {streamForwardSeekOperations=0} {streamBytesRead=22626526} {streamSeekOperations=0} {streamReadExceptions=0} {streamOpened=1584} {streamReadOperationsIncomplete=1584} {streamAborted=0} {streamReadFullyOperations=0} {streamClosed=1584} {streamBytesSkippedOnSeek=0} {streamCloseOperations=1584} {streamBytesBackwardsOnSeek=0} {streamBackwardSeekOperations=0} }} And without the patch 2016-06-16 18:37:55,688 INFO scheduler.DAGScheduler (Logging.scala:logInfo(58)) - Job 0 finished: count at S3LineCount.scala:99, took 15.853566 s 2016-06-16 18:37:55,693 INFO examples.S3LineCount (Logging.scala:logInfo(58)) - Duration of count s3a: //landsat-pds/scene_list.gz = 16,143,975,760 ns 2016-06-16 18:37:55,693 INFO examples.S3LineCount (Logging.scala:logInfo(58)) - line count = 514524 2016-06-16 18:37:55,694 INFO examples.S3LineCount (Logging.scala:logInfo(58)) - File System = S3AFileSystem{uri=s3a: //landsat-pds, workingDir=s3a://landsat-pds/user/stevel, partSize=5242880, enableMultiObjectsDelete= true , maxKeys=5000, readAhead=65536, blockSize=1048576, multiPartThreshold=5242880, statistics {22626526 bytes read, 0 bytes written, 3 read ops, 0 large read ops, 0 write ops}, metrics {{Context=S3AFileSystem} {FileSystemId=96650849-6e33-441f-a976-e74443239ad6-landsat-pds} {fsURI=s3a://landsat-pds/scene_list.gz} {files_created=0} {files_copied=0} {files_copied_bytes=0} {files_deleted=0} {directories_created=0} {directories_deleted=0} {ignored_errors=1} {invocations_copyfromlocalfile=0} {invocations_exists=0} {invocations_getfilestatus=3} {invocations_globstatus=1} {invocations_is_directory=0} {invocations_is_file=0} {invocations_listfiles=0} {invocations_listlocatedstatus=0} {invocations_liststatus=0} {invocations_mdkirs=0} {invocations_rename=0} {object_copy_requests=0} {object_delete_requests=0} {object_list_requests=0} {object_metadata_requests=3} {object_multipart_aborted=0} {object_put_bytes=0} {object_put_requests=0} {streamReadOperations=2601} {streamForwardSeekOperations=0} {streamBytesRead=22626526} {streamSeekOperations=0} {streamReadExceptions=0} {streamOpened=1} {streamReadOperationsIncomplete=2601} {streamAborted=0} {streamReadFullyOperations=0} {streamClosed=1} {streamBytesSkippedOnSeek=0} {streamCloseOperations=1} {streamBytesBackwardsOnSeek=0} {streamBackwardSeekOperations=0} }} 2016-06-16 18:37:55,694 INFO examples.S3LineCount (Logging.scala:logInfo(58)) - Stopping Spark Context The test, I believe, simply reads in the whole file: no seeks, no skipping. I see the no of stream opened calls has gone from 1 to 1584. I suspect that is what's at play here I think this code needs what I suggested, some block mechanism which works with open read() calls along with read operations where the full length is known. It also needs to handle the scenario of a read(bytes[]) which overshoots the currently being read block, not by closing the current read and discarding the data, but by reading in all the data it can from the current block, then starting to read the new block. Also The default block size needs to be significantly bigger A new S3Scale test to do the full byte-by-byte scan through the file; this will pick up performance problems immediately. ( i may put that in myself anyway, to catch similar problems in other patches) I think some more instrumentation would be good here. Specifically "bytes from current read discarded".
          Hide
          cnauroth Chris Nauroth added a comment -

          Steve, thanks for posting the details of the instrumentation. I think the key point there is {streamOpened=1584} vs. {streamOpened=1}. If the Spark test only triggered 1 streamOpened (without this patch), then it must have been a full forward-scan usage pattern. This matches with my earlier observation about TestS3AInputStreamPerformance#testReadAheadDefault, so for future testing, that's a test case we can focus on decoupled from Spark.

          Show
          cnauroth Chris Nauroth added a comment - Steve, thanks for posting the details of the instrumentation. I think the key point there is {streamOpened=1584} vs. {streamOpened=1}. If the Spark test only triggered 1 streamOpened (without this patch), then it must have been a full forward-scan usage pattern. This matches with my earlier observation about TestS3AInputStreamPerformance#testReadAheadDefault , so for future testing, that's a test case we can focus on decoupled from Spark.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Chris, it's work over a .gz file; that codec has to go through the entire file.

          We also have do do some measurements of seeks on large server-side-encrypted files: if the decryption is in blocks, seeks should be affordable. If it has to start from the front each time, we'd expect the duration of open(), seek(pos) read() to be O(pos)

          Show
          stevel@apache.org Steve Loughran added a comment - Chris, it's work over a .gz file; that codec has to go through the entire file. We also have do do some measurements of seeks on large server-side-encrypted files: if the decryption is in blocks, seeks should be affordable. If it has to start from the front each time, we'd expect the duration of open(), seek(pos) read() to be O(pos)
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Performance of the HADOOP-13286 patch

          testDecompression128K: Decompress with a 128K readahead
          
          2016-06-17 17:14:57,072 [Thread-0] INFO  compress.CodecPool (CodecPool.java:getDecompressor(181)) - Got brand-new decompressor [.gz]
          2016-06-17 17:15:32,986 [Thread-0] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1262)) - Duration of Time to read 514690 lines [99896260 bytes expanded, 22633778 raw] with readahead = 131072: 36,078,064,490 nS
          2016-06-17 17:15:32,986 [Thread-0] INFO  scale.TestS3AInputStreamPerformance (TestS3AInputStreamPerformance.java:logTimePerIOP(144)) - Time per IOP: 70,096 nS
          2016-06-17 17:15:32,987 [Thread-0] INFO  scale.TestS3AInputStreamPerformance (TestS3AInputStreamPerformance.java:logStreamStatistics(306)) - Stream Statistics
          StreamStatistics{OpenOperations=175, CloseOperations=175, Closed=175, Aborted=0, SeekOperations=0, ReadExceptions=0, ForwardSeekOperations=0, BackwardSeekOperations=0, BytesSkippedOnSeek=0, BytesBackwardsOnSeek=0, BytesRead=22633778, BytesRead excluding skipped=22633778, ReadOperations=6680, ReadFullyOperations=0, ReadsIncomplete=1583}
          
          Show
          stevel@apache.org Steve Loughran added a comment - Performance of the HADOOP-13286 patch testDecompression128K: Decompress with a 128K readahead 2016-06-17 17:14:57,072 [ Thread -0] INFO compress.CodecPool (CodecPool.java:getDecompressor(181)) - Got brand- new decompressor [.gz] 2016-06-17 17:15:32,986 [ Thread -0] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1262)) - Duration of Time to read 514690 lines [99896260 bytes expanded, 22633778 raw] with readahead = 131072: 36,078,064,490 nS 2016-06-17 17:15:32,986 [ Thread -0] INFO scale.TestS3AInputStreamPerformance (TestS3AInputStreamPerformance.java:logTimePerIOP(144)) - Time per IOP: 70,096 nS 2016-06-17 17:15:32,987 [ Thread -0] INFO scale.TestS3AInputStreamPerformance (TestS3AInputStreamPerformance.java:logStreamStatistics(306)) - Stream Statistics StreamStatistics{OpenOperations=175, CloseOperations=175, Closed=175, Aborted=0, SeekOperations=0, ReadExceptions=0, ForwardSeekOperations=0, BackwardSeekOperations=0, BytesSkippedOnSeek=0, BytesBackwardsOnSeek=0, BytesRead=22633778, BytesRead excluding skipped=22633778, ReadOperations=6680, ReadFullyOperations=0, ReadsIncomplete=1583}
          Hide
          stevel@apache.org Steve Loughran added a comment -

          patch 005. This is a WiP, just wanted to push it up to show where I'm going here.

          The key change is that it introduces the notion of an InputStrategy to S3a, currently: general, positioned, sequential

          As of now, there's also no diff between positioned and general: they both say "to end of stream"; I think general may want to consider having slightly shorter range, though still
          something big.

          Logic of seekInStream enhanced to not try seeking if the end of the range passed in is beyond the end of the current read.

          The metrics track more details on range overshoot

          Limits
          -now need to test both codepaths. The strategy can be set on an instantiated FS instance to allow testing without recreating FS instances.
          -still wasteful of data in the current read if the next read overshoots (maybe counter could track the missed quantitiy there), then go to having read(bytes[]) return the amount of available data, with the readFully() calls handling the incomplete response by asking for more.
          -what would a good policy for "general" be? Not positioned, clearly...but is sequential it?

          Show
          stevel@apache.org Steve Loughran added a comment - patch 005. This is a WiP, just wanted to push it up to show where I'm going here. The key change is that it introduces the notion of an InputStrategy to S3a, currently: general, positioned, sequential As of now, there's also no diff between positioned and general: they both say "to end of stream"; I think general may want to consider having slightly shorter range, though still something big. Logic of seekInStream enhanced to not try seeking if the end of the range passed in is beyond the end of the current read. The metrics track more details on range overshoot Limits -now need to test both codepaths. The strategy can be set on an instantiated FS instance to allow testing without recreating FS instances. -still wasteful of data in the current read if the next read overshoots (maybe counter could track the missed quantitiy there), then go to having read(bytes[]) return the amount of available data, with the readFully() calls handling the incomplete response by asking for more. -what would a good policy for "general" be? Not positioned, clearly...but is sequential it?
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 19s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 0m 19s Maven dependency ordering for branch
          +1 mvninstall 6m 37s branch-2 passed
          +1 compile 5m 30s branch-2 passed with JDK v1.8.0_91
          +1 compile 6m 21s branch-2 passed with JDK v1.7.0_101
          +1 checkstyle 1m 21s branch-2 passed
          +1 mvnsite 1m 19s branch-2 passed
          +1 mvneclipse 0m 33s branch-2 passed
          +1 findbugs 2m 12s branch-2 passed
          +1 javadoc 0m 59s branch-2 passed with JDK v1.8.0_91
          +1 javadoc 1m 11s branch-2 passed with JDK v1.7.0_101
          0 mvndep 0m 14s Maven dependency ordering for patch
          +1 mvninstall 0m 56s the patch passed
          +1 compile 7m 0s the patch passed with JDK v1.8.0_91
          +1 javac 7m 0s the patch passed
          +1 compile 6m 43s the patch passed with JDK v1.7.0_101
          +1 javac 6m 43s the patch passed
          -1 checkstyle 1m 21s root: The patch generated 5 new + 44 unchanged - 7 fixed = 49 total (was 51)
          +1 mvnsite 1m 20s the patch passed
          +1 mvneclipse 0m 28s the patch passed
          -1 whitespace 0m 0s The patch has 50 line(s) that end in whitespace. Use git apply --whitespace=fix.
          -1 findbugs 0m 46s hadoop-tools/hadoop-aws generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
          +1 javadoc 0m 57s the patch passed with JDK v1.8.0_91
          +1 javadoc 1m 13s the patch passed with JDK v1.7.0_101
          +1 unit 7m 50s hadoop-common in the patch passed with JDK v1.7.0_101.
          +1 unit 0m 16s hadoop-aws in the patch passed with JDK v1.7.0_101.
          +1 asflicense 0m 21s The patch does not generate ASF License warnings.
          67m 21s



          Reason Tests
          FindBugs module:hadoop-tools/hadoop-aws
            Unread field:S3AInputStream.java:[line 173]



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:d1c475d
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12811427/HADOOP-13203-branch-2-005.patch
          JIRA Issue HADOOP-13203
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 4849147f1ea3 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2 / 35c6b72
          Default Java 1.7.0_101
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/9817/artifact/patchprocess/diff-checkstyle-root.txt
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/9817/artifact/patchprocess/whitespace-eol.txt
          findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/9817/artifact/patchprocess/new-findbugs-hadoop-tools_hadoop-aws.html
          JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9817/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9817/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 0m 19s Maven dependency ordering for branch +1 mvninstall 6m 37s branch-2 passed +1 compile 5m 30s branch-2 passed with JDK v1.8.0_91 +1 compile 6m 21s branch-2 passed with JDK v1.7.0_101 +1 checkstyle 1m 21s branch-2 passed +1 mvnsite 1m 19s branch-2 passed +1 mvneclipse 0m 33s branch-2 passed +1 findbugs 2m 12s branch-2 passed +1 javadoc 0m 59s branch-2 passed with JDK v1.8.0_91 +1 javadoc 1m 11s branch-2 passed with JDK v1.7.0_101 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 0m 56s the patch passed +1 compile 7m 0s the patch passed with JDK v1.8.0_91 +1 javac 7m 0s the patch passed +1 compile 6m 43s the patch passed with JDK v1.7.0_101 +1 javac 6m 43s the patch passed -1 checkstyle 1m 21s root: The patch generated 5 new + 44 unchanged - 7 fixed = 49 total (was 51) +1 mvnsite 1m 20s the patch passed +1 mvneclipse 0m 28s the patch passed -1 whitespace 0m 0s The patch has 50 line(s) that end in whitespace. Use git apply --whitespace=fix. -1 findbugs 0m 46s hadoop-tools/hadoop-aws generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) +1 javadoc 0m 57s the patch passed with JDK v1.8.0_91 +1 javadoc 1m 13s the patch passed with JDK v1.7.0_101 +1 unit 7m 50s hadoop-common in the patch passed with JDK v1.7.0_101. +1 unit 0m 16s hadoop-aws in the patch passed with JDK v1.7.0_101. +1 asflicense 0m 21s The patch does not generate ASF License warnings. 67m 21s Reason Tests FindBugs module:hadoop-tools/hadoop-aws   Unread field:S3AInputStream.java: [line 173] Subsystem Report/Notes Docker Image:yetus/hadoop:d1c475d JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12811427/HADOOP-13203-branch-2-005.patch JIRA Issue HADOOP-13203 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 4849147f1ea3 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / 35c6b72 Default Java 1.7.0_101 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/9817/artifact/patchprocess/diff-checkstyle-root.txt whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/9817/artifact/patchprocess/whitespace-eol.txt findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/9817/artifact/patchprocess/new-findbugs-hadoop-tools_hadoop-aws.html JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9817/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9817/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          HADOOP-13203 Patch 006. Explicit policies based on fadvise terminology(); configuration file option uses "experimental" in its name, to indicate that it is. Explicit test of random IO shows 4x speedup over sequential IO. Also: cut back on some of the scale tests that were just doing sequential seek+read with different readahead sizes. They don't add much, just take up another 10-20s

          Show
          stevel@apache.org Steve Loughran added a comment - HADOOP-13203 Patch 006. Explicit policies based on fadvise terminology(); configuration file option uses "experimental" in its name, to indicate that it is. Explicit test of random IO shows 4x speedup over sequential IO. Also: cut back on some of the scale tests that were just doing sequential seek+read with different readahead sizes. They don't add much, just take up another 10-20s
          Hide
          stevel@apache.org Steve Loughran added a comment -

          HADOOP-13203 Patch 007 cleanup, including findbugs and checkstyle

          While this patch is ready for some review, there's one feature I want to write a test for and then address: a read which starts in the current requested range but which goes past it causes the stream to be closed, starting again at the new position. This can be fixed.

          I plan to do it by having the read(bytes[]) return only the bytes in the current request; this meets the semantics of read(bytes[]). The readFullly() calls already iterate on the reads(), so this is handled at that level...there is no need to be clever further down.

          Show
          stevel@apache.org Steve Loughran added a comment - HADOOP-13203 Patch 007 cleanup, including findbugs and checkstyle While this patch is ready for some review, there's one feature I want to write a test for and then address: a read which starts in the current requested range but which goes past it causes the stream to be closed, starting again at the new position. This can be fixed. I plan to do it by having the read(bytes[]) return only the bytes in the current request; this meets the semantics of read(bytes[]) . The readFullly() calls already iterate on the reads(), so this is handled at that level...there is no need to be clever further down.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Here is the comparison of a small sequence of forward/backward read operations, between the random and sequential policies

          "random" keeps buffer sizes in requests down to a minimum, hence no bytes wasted in close or aborts BytesReadInClose=0, BytesDiscardedInAbort=0

          "sequential" expects a read from 0-len, requests the entire range. Forward seeks can be skipped, backward seeks trigger retreat. However, the default block size (32K) is too low for any forward skip (should we change this to something like 128K?), then there's an abort, leading to the values BytesReadInClose=0, BytesDiscardedInAbort=80771308. Those abort bytes never get read in, but they do measure how oversized the request was

          testRandomIO_RandomPolicy: Random IO with policy "random"
          
          2016-06-20 20:35:15,680 [Thread-0] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:setInputPolicy(566)) - Setting input strategy: random
          2016-06-20 20:35:15,680 [Thread-0] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:open(617)) - Opening 's3a://landsat-pds/scene_list.gz' for reading.
          2016-06-20 20:35:15,680 [Thread-0] DEBUG s3a.S3AFileSystem (S3AStorageStatistics.java:incrementCounter(60)) - invocations_getfilestatus += 1  ->  2
          2016-06-20 20:35:15,680 [Thread-0] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:getFileStatus(1421)) - Getting path status for s3a://landsat-pds/scene_list.gz  (scene_list.gz)
          2016-06-20 20:35:15,681 [Thread-0] DEBUG s3a.S3AFileSystem (S3AStorageStatistics.java:incrementCounter(60)) - object_metadata_requests += 1  ->  2
          2016-06-20 20:35:15,846 [Thread-0] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:getFileStatus(1432)) - Found exact file: normal file
          2016-06-20 20:35:15,848 [Thread-0] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:setInputPolicy(566)) - Setting input strategy: normal
          2016-06-20 20:35:15,850 [Thread-0] DEBUG s3a.S3AFileSystem (S3AInputStream.java:reopen(145)) - reopen(s3a://landsat-pds/scene_list.gz) for read from new offset at targetPos=2097152, length=131072, requestedStreamLen=2228224, streamPosition=0, nextReadPosition=2097152
          2016-06-20 20:35:16,069 [Thread-0] DEBUG s3a.S3AFileSystem (S3AInputStream.java:closeStream(470)) - Stream s3a://landsat-pds/scene_list.gz closed: seekInStream(); streamPos=2228224, nextReadPos=131072,request range 2097152-2228224 length=2228224
          2016-06-20 20:35:16,069 [Thread-0] DEBUG s3a.S3AFileSystem (S3AInputStream.java:reopen(145)) - reopen(s3a://landsat-pds/scene_list.gz) for read from new offset at targetPos=131072, length=131072, requestedStreamLen=262144, streamPosition=131072, nextReadPosition=131072
          2016-06-20 20:35:16,259 [Thread-0] DEBUG s3a.S3AFileSystem (S3AInputStream.java:closeStream(470)) - Stream s3a://landsat-pds/scene_list.gz closed: seekInStream(); streamPos=262144, nextReadPos=5242880,request range 131072-262144 length=262144
          2016-06-20 20:35:16,259 [Thread-0] DEBUG s3a.S3AFileSystem (S3AInputStream.java:reopen(145)) - reopen(s3a://landsat-pds/scene_list.gz) for read from new offset at targetPos=5242880, length=65536, requestedStreamLen=5308416, streamPosition=5242880, nextReadPosition=5242880
          2016-06-20 20:35:16,437 [Thread-0] DEBUG s3a.S3AFileSystem (S3AInputStream.java:closeStream(470)) - Stream s3a://landsat-pds/scene_list.gz closed: seekInStream(); streamPos=5308416, nextReadPos=1048576,request range 5242880-5308416 length=5308416
          2016-06-20 20:35:16,437 [Thread-0] DEBUG s3a.S3AFileSystem (S3AInputStream.java:reopen(145)) - reopen(s3a://landsat-pds/scene_list.gz) for read from new offset at targetPos=1048576, length=1048576, requestedStreamLen=2097152, streamPosition=1048576, nextReadPosition=1048576
          2016-06-20 20:35:16,994 [Thread-0] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1262)) - Duration of Time to execute 4 reads of total size 1376256 bytes: 1,141,400,611 nS
          2016-06-20 20:35:16,994 [Thread-0] DEBUG s3a.S3AFileSystem (S3AInputStream.java:closeStream(470)) - Stream s3a://landsat-pds/scene_list.gz closed: close() operation; streamPos=2097152, nextReadPos=0,request range 1048576-2097152 length=2097152
          2016-06-20 20:35:16,995 [Thread-0] INFO  scale.TestS3AInputStreamPerformance (TestS3AInputStreamPerformance.java:logTimePerIOP(165)) - Time per byte read: 829 nS
          2016-06-20 20:35:16,996 [Thread-0] INFO  scale.TestS3AInputStreamPerformance (TestS3AInputStreamPerformance.java:executeRandomIO(388)) - Effective bandwidth 1.205761 MB/S
          2016-06-20 20:35:16,997 [Thread-0] INFO  scale.TestS3AInputStreamPerformance (TestS3AInputStreamPerformance.java:logStreamStatistics(292)) - Stream Statistics
          StreamStatistics{OpenOperations=4, CloseOperations=4, Closed=4, Aborted=0, SeekOperations=2, ReadExceptions=0, ForwardSeekOperations=0, BackwardSeekOperations=2, BytesSkippedOnSeek=0, BytesBackwardsOnSeek=6356992, BytesRead=1376256, BytesRead excluding skipped=1376256, ReadOperations=164, ReadFullyOperations=4, ReadsIncomplete=160, SeekRangeOvershot=0, BytesReadInClose=0, BytesDiscardedInAbort=0}
          
          
          testRandomIO_NormalPolicy: Random IO with policy "normal"
          
          2016-06-20 20:35:20,433 [Thread-5] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:setInputPolicy(566)) - Setting input strategy: normal
          2016-06-20 20:35:20,433 [Thread-5] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:open(617)) - Opening 's3a://landsat-pds/scene_list.gz' for reading.
          2016-06-20 20:35:20,433 [Thread-5] DEBUG s3a.S3AFileSystem (S3AStorageStatistics.java:incrementCounter(60)) - invocations_getfilestatus += 1  ->  5
          2016-06-20 20:35:20,434 [Thread-5] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:getFileStatus(1421)) - Getting path status for s3a://landsat-pds/scene_list.gz  (scene_list.gz)
          2016-06-20 20:35:20,434 [Thread-5] DEBUG s3a.S3AFileSystem (S3AStorageStatistics.java:incrementCounter(60)) - object_metadata_requests += 1  ->  6
          2016-06-20 20:35:20,597 [Thread-5] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:getFileStatus(1432)) - Found exact file: normal file
          2016-06-20 20:35:20,597 [Thread-5] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:setInputPolicy(566)) - Setting input strategy: normal
          2016-06-20 20:35:20,598 [Thread-5] DEBUG s3a.S3AFileSystem (S3AInputStream.java:reopen(145)) - reopen(s3a://landsat-pds/scene_list.gz) for read from new offset at targetPos=2097152, length=131072, requestedStreamLen=22666811, streamPosition=0, nextReadPosition=2097152
          2016-06-20 20:35:20,792 [Thread-5] DEBUG s3a.S3AFileSystem (S3AInputStream.java:closeStream(470)) - Stream s3a://landsat-pds/scene_list.gz aborted: seekInStream(); streamPos=2228224, nextReadPos=131072,request range 2097152-22666811 length=22666811
          2016-06-20 20:35:20,792 [Thread-5] DEBUG s3a.S3AFileSystem (S3AInputStream.java:reopen(145)) - reopen(s3a://landsat-pds/scene_list.gz) for read from new offset at targetPos=131072, length=131072, requestedStreamLen=22666811, streamPosition=131072, nextReadPosition=131072
          2016-06-20 20:35:22,117 [Thread-5] DEBUG s3a.S3AFileSystem (S3AInputStream.java:closeStream(470)) - Stream s3a://landsat-pds/scene_list.gz aborted: seekInStream(); streamPos=262144, nextReadPos=5242880,request range 131072-22666811 length=22666811
          2016-06-20 20:35:22,117 [Thread-5] DEBUG s3a.S3AFileSystem (S3AInputStream.java:reopen(145)) - reopen(s3a://landsat-pds/scene_list.gz) for read from new offset at targetPos=5242880, length=65536, requestedStreamLen=22666811, streamPosition=5242880, nextReadPosition=5242880
          2016-06-20 20:35:23,302 [Thread-5] DEBUG s3a.S3AFileSystem (S3AInputStream.java:closeStream(470)) - Stream s3a://landsat-pds/scene_list.gz aborted: seekInStream(); streamPos=5308416, nextReadPos=1048576,request range 5242880-22666811 length=22666811
          2016-06-20 20:35:23,302 [Thread-5] DEBUG s3a.S3AFileSystem (S3AInputStream.java:reopen(145)) - reopen(s3a://landsat-pds/scene_list.gz) for read from new offset at targetPos=1048576, length=1048576, requestedStreamLen=22666811, streamPosition=1048576, nextReadPosition=1048576
          2016-06-20 20:35:25,240 [Thread-5] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1262)) - Duration of Time to execute 4 reads of total size 1376256 bytes: 4,641,800,492 nS
          2016-06-20 20:35:25,240 [Thread-5] DEBUG s3a.S3AFileSystem (S3AInputStream.java:closeStream(470)) - Stream s3a://landsat-pds/scene_list.gz aborted: close() operation; streamPos=2097152, nextReadPos=0,request range 1048576-22666811 length=22666811
          2016-06-20 20:35:25,241 [Thread-5] INFO  scale.TestS3AInputStreamPerformance (TestS3AInputStreamPerformance.java:logTimePerIOP(165)) - Time per byte read: 3,372 nS
          2016-06-20 20:35:25,241 [Thread-5] INFO  scale.TestS3AInputStreamPerformance (TestS3AInputStreamPerformance.java:executeRandomIO(388)) - Effective bandwidth 0.296492 MB/S
          2016-06-20 20:35:25,241 [Thread-5] INFO  scale.TestS3AInputStreamPerformance (TestS3AInputStreamPerformance.java:logStreamStatistics(292)) - Stream Statistics
          StreamStatistics{OpenOperations=4, CloseOperations=4, Closed=0, Aborted=4, SeekOperations=2, ReadExceptions=0, ForwardSeekOperations=0, BackwardSeekOperations=2, BytesSkippedOnSeek=0, BytesBackwardsOnSeek=6356992, BytesRead=1376256, BytesRead excluding skipped=1376256, ReadOperations=158, ReadFullyOperations=4, ReadsIncomplete=154, SeekRangeOvershot=0, BytesReadInClose=0, BytesDiscardedInAbort=80771308}
          2016-06-20 20:35:25,241 [Thread-5] INFO  scale.S3AScaleTestBase (S3AScaleTestBase.java:describe(155)) - 
          
          
          Show
          stevel@apache.org Steve Loughran added a comment - Here is the comparison of a small sequence of forward/backward read operations, between the random and sequential policies "random" keeps buffer sizes in requests down to a minimum, hence no bytes wasted in close or aborts BytesReadInClose=0, BytesDiscardedInAbort=0 "sequential" expects a read from 0-len, requests the entire range. Forward seeks can be skipped, backward seeks trigger retreat. However, the default block size (32K) is too low for any forward skip (should we change this to something like 128K?), then there's an abort, leading to the values BytesReadInClose=0, BytesDiscardedInAbort=80771308 . Those abort bytes never get read in, but they do measure how oversized the request was testRandomIO_RandomPolicy: Random IO with policy "random" 2016-06-20 20:35:15,680 [ Thread -0] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:setInputPolicy(566)) - Setting input strategy: random 2016-06-20 20:35:15,680 [ Thread -0] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:open(617)) - Opening 's3a: //landsat-pds/scene_list.gz' for reading. 2016-06-20 20:35:15,680 [ Thread -0] DEBUG s3a.S3AFileSystem (S3AStorageStatistics.java:incrementCounter(60)) - invocations_getfilestatus += 1 -> 2 2016-06-20 20:35:15,680 [ Thread -0] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:getFileStatus(1421)) - Getting path status for s3a: //landsat-pds/scene_list.gz (scene_list.gz) 2016-06-20 20:35:15,681 [ Thread -0] DEBUG s3a.S3AFileSystem (S3AStorageStatistics.java:incrementCounter(60)) - object_metadata_requests += 1 -> 2 2016-06-20 20:35:15,846 [ Thread -0] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:getFileStatus(1432)) - Found exact file: normal file 2016-06-20 20:35:15,848 [ Thread -0] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:setInputPolicy(566)) - Setting input strategy: normal 2016-06-20 20:35:15,850 [ Thread -0] DEBUG s3a.S3AFileSystem (S3AInputStream.java:reopen(145)) - reopen(s3a: //landsat-pds/scene_list.gz) for read from new offset at targetPos=2097152, length=131072, requestedStreamLen=2228224, streamPosition=0, nextReadPosition=2097152 2016-06-20 20:35:16,069 [ Thread -0] DEBUG s3a.S3AFileSystem (S3AInputStream.java:closeStream(470)) - Stream s3a: //landsat-pds/scene_list.gz closed: seekInStream(); streamPos=2228224, nextReadPos=131072,request range 2097152-2228224 length=2228224 2016-06-20 20:35:16,069 [ Thread -0] DEBUG s3a.S3AFileSystem (S3AInputStream.java:reopen(145)) - reopen(s3a: //landsat-pds/scene_list.gz) for read from new offset at targetPos=131072, length=131072, requestedStreamLen=262144, streamPosition=131072, nextReadPosition=131072 2016-06-20 20:35:16,259 [ Thread -0] DEBUG s3a.S3AFileSystem (S3AInputStream.java:closeStream(470)) - Stream s3a: //landsat-pds/scene_list.gz closed: seekInStream(); streamPos=262144, nextReadPos=5242880,request range 131072-262144 length=262144 2016-06-20 20:35:16,259 [ Thread -0] DEBUG s3a.S3AFileSystem (S3AInputStream.java:reopen(145)) - reopen(s3a: //landsat-pds/scene_list.gz) for read from new offset at targetPos=5242880, length=65536, requestedStreamLen=5308416, streamPosition=5242880, nextReadPosition=5242880 2016-06-20 20:35:16,437 [ Thread -0] DEBUG s3a.S3AFileSystem (S3AInputStream.java:closeStream(470)) - Stream s3a: //landsat-pds/scene_list.gz closed: seekInStream(); streamPos=5308416, nextReadPos=1048576,request range 5242880-5308416 length=5308416 2016-06-20 20:35:16,437 [ Thread -0] DEBUG s3a.S3AFileSystem (S3AInputStream.java:reopen(145)) - reopen(s3a: //landsat-pds/scene_list.gz) for read from new offset at targetPos=1048576, length=1048576, requestedStreamLen=2097152, streamPosition=1048576, nextReadPosition=1048576 2016-06-20 20:35:16,994 [ Thread -0] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1262)) - Duration of Time to execute 4 reads of total size 1376256 bytes: 1,141,400,611 nS 2016-06-20 20:35:16,994 [ Thread -0] DEBUG s3a.S3AFileSystem (S3AInputStream.java:closeStream(470)) - Stream s3a: //landsat-pds/scene_list.gz closed: close() operation; streamPos=2097152, nextReadPos=0,request range 1048576-2097152 length=2097152 2016-06-20 20:35:16,995 [ Thread -0] INFO scale.TestS3AInputStreamPerformance (TestS3AInputStreamPerformance.java:logTimePerIOP(165)) - Time per byte read: 829 nS 2016-06-20 20:35:16,996 [ Thread -0] INFO scale.TestS3AInputStreamPerformance (TestS3AInputStreamPerformance.java:executeRandomIO(388)) - Effective bandwidth 1.205761 MB/S 2016-06-20 20:35:16,997 [ Thread -0] INFO scale.TestS3AInputStreamPerformance (TestS3AInputStreamPerformance.java:logStreamStatistics(292)) - Stream Statistics StreamStatistics{OpenOperations=4, CloseOperations=4, Closed=4, Aborted=0, SeekOperations=2, ReadExceptions=0, ForwardSeekOperations=0, BackwardSeekOperations=2, BytesSkippedOnSeek=0, BytesBackwardsOnSeek=6356992, BytesRead=1376256, BytesRead excluding skipped=1376256, ReadOperations=164, ReadFullyOperations=4, ReadsIncomplete=160, SeekRangeOvershot=0, BytesReadInClose=0, BytesDiscardedInAbort=0} testRandomIO_NormalPolicy: Random IO with policy "normal" 2016-06-20 20:35:20,433 [ Thread -5] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:setInputPolicy(566)) - Setting input strategy: normal 2016-06-20 20:35:20,433 [ Thread -5] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:open(617)) - Opening 's3a: //landsat-pds/scene_list.gz' for reading. 2016-06-20 20:35:20,433 [ Thread -5] DEBUG s3a.S3AFileSystem (S3AStorageStatistics.java:incrementCounter(60)) - invocations_getfilestatus += 1 -> 5 2016-06-20 20:35:20,434 [ Thread -5] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:getFileStatus(1421)) - Getting path status for s3a: //landsat-pds/scene_list.gz (scene_list.gz) 2016-06-20 20:35:20,434 [ Thread -5] DEBUG s3a.S3AFileSystem (S3AStorageStatistics.java:incrementCounter(60)) - object_metadata_requests += 1 -> 6 2016-06-20 20:35:20,597 [ Thread -5] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:getFileStatus(1432)) - Found exact file: normal file 2016-06-20 20:35:20,597 [ Thread -5] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:setInputPolicy(566)) - Setting input strategy: normal 2016-06-20 20:35:20,598 [ Thread -5] DEBUG s3a.S3AFileSystem (S3AInputStream.java:reopen(145)) - reopen(s3a: //landsat-pds/scene_list.gz) for read from new offset at targetPos=2097152, length=131072, requestedStreamLen=22666811, streamPosition=0, nextReadPosition=2097152 2016-06-20 20:35:20,792 [ Thread -5] DEBUG s3a.S3AFileSystem (S3AInputStream.java:closeStream(470)) - Stream s3a: //landsat-pds/scene_list.gz aborted: seekInStream(); streamPos=2228224, nextReadPos=131072,request range 2097152-22666811 length=22666811 2016-06-20 20:35:20,792 [ Thread -5] DEBUG s3a.S3AFileSystem (S3AInputStream.java:reopen(145)) - reopen(s3a: //landsat-pds/scene_list.gz) for read from new offset at targetPos=131072, length=131072, requestedStreamLen=22666811, streamPosition=131072, nextReadPosition=131072 2016-06-20 20:35:22,117 [ Thread -5] DEBUG s3a.S3AFileSystem (S3AInputStream.java:closeStream(470)) - Stream s3a: //landsat-pds/scene_list.gz aborted: seekInStream(); streamPos=262144, nextReadPos=5242880,request range 131072-22666811 length=22666811 2016-06-20 20:35:22,117 [ Thread -5] DEBUG s3a.S3AFileSystem (S3AInputStream.java:reopen(145)) - reopen(s3a: //landsat-pds/scene_list.gz) for read from new offset at targetPos=5242880, length=65536, requestedStreamLen=22666811, streamPosition=5242880, nextReadPosition=5242880 2016-06-20 20:35:23,302 [ Thread -5] DEBUG s3a.S3AFileSystem (S3AInputStream.java:closeStream(470)) - Stream s3a: //landsat-pds/scene_list.gz aborted: seekInStream(); streamPos=5308416, nextReadPos=1048576,request range 5242880-22666811 length=22666811 2016-06-20 20:35:23,302 [ Thread -5] DEBUG s3a.S3AFileSystem (S3AInputStream.java:reopen(145)) - reopen(s3a: //landsat-pds/scene_list.gz) for read from new offset at targetPos=1048576, length=1048576, requestedStreamLen=22666811, streamPosition=1048576, nextReadPosition=1048576 2016-06-20 20:35:25,240 [ Thread -5] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1262)) - Duration of Time to execute 4 reads of total size 1376256 bytes: 4,641,800,492 nS 2016-06-20 20:35:25,240 [ Thread -5] DEBUG s3a.S3AFileSystem (S3AInputStream.java:closeStream(470)) - Stream s3a: //landsat-pds/scene_list.gz aborted: close() operation; streamPos=2097152, nextReadPos=0,request range 1048576-22666811 length=22666811 2016-06-20 20:35:25,241 [ Thread -5] INFO scale.TestS3AInputStreamPerformance (TestS3AInputStreamPerformance.java:logTimePerIOP(165)) - Time per byte read: 3,372 nS 2016-06-20 20:35:25,241 [ Thread -5] INFO scale.TestS3AInputStreamPerformance (TestS3AInputStreamPerformance.java:executeRandomIO(388)) - Effective bandwidth 0.296492 MB/S 2016-06-20 20:35:25,241 [ Thread -5] INFO scale.TestS3AInputStreamPerformance (TestS3AInputStreamPerformance.java:logStreamStatistics(292)) - Stream Statistics StreamStatistics{OpenOperations=4, CloseOperations=4, Closed=0, Aborted=4, SeekOperations=2, ReadExceptions=0, ForwardSeekOperations=0, BackwardSeekOperations=2, BytesSkippedOnSeek=0, BytesBackwardsOnSeek=6356992, BytesRead=1376256, BytesRead excluding skipped=1376256, ReadOperations=158, ReadFullyOperations=4, ReadsIncomplete=154, SeekRangeOvershot=0, BytesReadInClose=0, BytesDiscardedInAbort=80771308} 2016-06-20 20:35:25,241 [ Thread -5] INFO scale.S3AScaleTestBase (S3AScaleTestBase.java:describe(155)) -
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Patch 008; tested against s3 ireland.

          This revision has the test to demonstrate what I suspected: reads spanning block boundaries were going to have problems —and it has the fix. Which consists of always calling seekInStream(pos, len) before a read, even if targetPos==currentPos —and in that situation, closing the current stream if the currentPos is at the end of the current request range (i.e. there's no seek, but no data either). The test does block-spanning reads, on a file built up with the byte at each position being (position % 64) ... this is used in the tests to verify the bytes returned really are the bytes in the file at the specific read positions.

          BTW, note that some of the -Len fields in the input stream now refer to range start and finish; Len isn't appropriate now the range of the HTTP request may be less than the length of the actual blob. It was getting confusing.

          Show
          stevel@apache.org Steve Loughran added a comment - Patch 008; tested against s3 ireland. This revision has the test to demonstrate what I suspected: reads spanning block boundaries were going to have problems —and it has the fix. Which consists of always calling seekInStream(pos, len) before a read, even if targetPos==currentPos —and in that situation, closing the current stream if the currentPos is at the end of the current request range (i.e. there's no seek, but no data either). The test does block-spanning reads, on a file built up with the byte at each position being (position % 64) ... this is used in the tests to verify the bytes returned really are the bytes in the file at the specific read positions. BTW, note that some of the -Len fields in the input stream now refer to range start and finish; Len isn't appropriate now the range of the HTTP request may be less than the length of the actual blob. It was getting confusing.
          Hide
          cnauroth Chris Nauroth added a comment -

          Steve and Rajesh, this looks great to me. We'll get the best of both worlds. Thank you very much.

          All of the random vs. sequential logic looks correct to me. All tests passed for me against a bucket in US-west-2, barring the known failure related to a secret with a '+' in it, which is tracked elsewhere. I only have a few minor nitpicks on patch 008.

          1. Please add audience and stability annotations to S3AInputPolicy.

             * Optimised purely for random seek+reed/positionedRead operations;
          

          2. s/reed/read

              // Better to set it to the value requested by higher level layer.
              // In case this is set to contentLength, expect lots of connection
              // closes when backwards-seeks are executed.
              // Note that abort would force the internal connection to be
              // closed and makes it un-usable.
          

          3. I think that comment can be removed. I don't think it's relevant anymore.

              LOG.info("Stream Statistics\n{}", streamStatistics);
          

          4. I suggest changing to this for platform-agnostic line endings:

              LOG.info(String.format("Stream Statistics%n{}"), streamStatistics);
          
          Show
          cnauroth Chris Nauroth added a comment - Steve and Rajesh, this looks great to me. We'll get the best of both worlds. Thank you very much. All of the random vs. sequential logic looks correct to me. All tests passed for me against a bucket in US-west-2, barring the known failure related to a secret with a '+' in it, which is tracked elsewhere. I only have a few minor nitpicks on patch 008. 1. Please add audience and stability annotations to S3AInputPolicy . * Optimised purely for random seek+reed/positionedRead operations; 2. s/reed/read // Better to set it to the value requested by higher level layer. // In case this is set to contentLength, expect lots of connection // closes when backwards-seeks are executed. // Note that abort would force the internal connection to be // closed and makes it un-usable. 3. I think that comment can be removed. I don't think it's relevant anymore. LOG.info( "Stream Statistics\n{}" , streamStatistics); 4. I suggest changing to this for platform-agnostic line endings: LOG.info( String .format( "Stream Statistics%n{}" ), streamStatistics);
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 29s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          0 mvndep 0m 42s Maven dependency ordering for branch
          +1 mvninstall 6m 40s branch-2 passed
          +1 compile 6m 2s branch-2 passed with JDK v1.8.0_91
          +1 compile 6m 40s branch-2 passed with JDK v1.7.0_101
          +1 checkstyle 1m 22s branch-2 passed
          +1 mvnsite 1m 18s branch-2 passed
          +1 mvneclipse 0m 30s branch-2 passed
          +1 findbugs 2m 14s branch-2 passed
          +1 javadoc 0m 59s branch-2 passed with JDK v1.8.0_91
          +1 javadoc 1m 11s branch-2 passed with JDK v1.7.0_101
          0 mvndep 0m 13s Maven dependency ordering for patch
          +1 mvninstall 1m 2s the patch passed
          +1 compile 5m 53s the patch passed with JDK v1.8.0_91
          +1 javac 5m 53s the patch passed
          +1 compile 6m 31s the patch passed with JDK v1.7.0_101
          +1 javac 6m 31s the patch passed
          -1 checkstyle 1m 21s root: The patch generated 33 new + 43 unchanged - 9 fixed = 76 total (was 52)
          +1 mvnsite 1m 20s the patch passed
          +1 mvneclipse 0m 30s the patch passed
          -1 whitespace 0m 0s The patch has 50 line(s) that end in whitespace. Use git apply --whitespace=fix.
          -1 findbugs 0m 47s hadoop-tools/hadoop-aws generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0)
          +1 javadoc 1m 1s the patch passed with JDK v1.8.0_91
          +1 javadoc 1m 11s the patch passed with JDK v1.7.0_101
          +1 unit 8m 26s hadoop-common in the patch passed with JDK v1.7.0_101.
          +1 unit 0m 16s hadoop-aws in the patch passed with JDK v1.7.0_101.
          +1 asflicense 0m 22s The patch does not generate ASF License warnings.
          68m 21s



          Reason Tests
          FindBugs module:hadoop-tools/hadoop-aws
            Inconsistent synchronization of org.apache.hadoop.fs.s3a.S3AInputStream.contentRangeFinish; locked 75% of time Unsynchronized access at S3AInputStream.java:75% of time Unsynchronized access at S3AInputStream.java:[line 514]
            Inconsistent synchronization of org.apache.hadoop.fs.s3a.S3AInputStream.pos; locked 87% of time Unsynchronized access at S3AInputStream.java:87% of time Unsynchronized access at S3AInputStream.java:[line 497]
          JDK v1.8.0_91 Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:d1c475d
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12812220/HADOOP-13203-branch-2-008.patch
          JIRA Issue HADOOP-13203
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 58e81c49cfc3 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2 / 65b4f26
          Default Java 1.7.0_101
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/9845/artifact/patchprocess/diff-checkstyle-root.txt
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/9845/artifact/patchprocess/whitespace-eol.txt
          findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/9845/artifact/patchprocess/new-findbugs-hadoop-tools_hadoop-aws.html
          JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9845/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9845/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 29s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 0m 42s Maven dependency ordering for branch +1 mvninstall 6m 40s branch-2 passed +1 compile 6m 2s branch-2 passed with JDK v1.8.0_91 +1 compile 6m 40s branch-2 passed with JDK v1.7.0_101 +1 checkstyle 1m 22s branch-2 passed +1 mvnsite 1m 18s branch-2 passed +1 mvneclipse 0m 30s branch-2 passed +1 findbugs 2m 14s branch-2 passed +1 javadoc 0m 59s branch-2 passed with JDK v1.8.0_91 +1 javadoc 1m 11s branch-2 passed with JDK v1.7.0_101 0 mvndep 0m 13s Maven dependency ordering for patch +1 mvninstall 1m 2s the patch passed +1 compile 5m 53s the patch passed with JDK v1.8.0_91 +1 javac 5m 53s the patch passed +1 compile 6m 31s the patch passed with JDK v1.7.0_101 +1 javac 6m 31s the patch passed -1 checkstyle 1m 21s root: The patch generated 33 new + 43 unchanged - 9 fixed = 76 total (was 52) +1 mvnsite 1m 20s the patch passed +1 mvneclipse 0m 30s the patch passed -1 whitespace 0m 0s The patch has 50 line(s) that end in whitespace. Use git apply --whitespace=fix. -1 findbugs 0m 47s hadoop-tools/hadoop-aws generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) +1 javadoc 1m 1s the patch passed with JDK v1.8.0_91 +1 javadoc 1m 11s the patch passed with JDK v1.7.0_101 +1 unit 8m 26s hadoop-common in the patch passed with JDK v1.7.0_101. +1 unit 0m 16s hadoop-aws in the patch passed with JDK v1.7.0_101. +1 asflicense 0m 22s The patch does not generate ASF License warnings. 68m 21s Reason Tests FindBugs module:hadoop-tools/hadoop-aws   Inconsistent synchronization of org.apache.hadoop.fs.s3a.S3AInputStream.contentRangeFinish; locked 75% of time Unsynchronized access at S3AInputStream.java:75% of time Unsynchronized access at S3AInputStream.java: [line 514]   Inconsistent synchronization of org.apache.hadoop.fs.s3a.S3AInputStream.pos; locked 87% of time Unsynchronized access at S3AInputStream.java:87% of time Unsynchronized access at S3AInputStream.java: [line 497] JDK v1.8.0_91 Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics Subsystem Report/Notes Docker Image:yetus/hadoop:d1c475d JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12812220/HADOOP-13203-branch-2-008.patch JIRA Issue HADOOP-13203 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 58e81c49cfc3 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / 65b4f26 Default Java 1.7.0_101 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/9845/artifact/patchprocess/diff-checkstyle-root.txt whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/9845/artifact/patchprocess/whitespace-eol.txt findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/9845/artifact/patchprocess/new-findbugs-hadoop-tools_hadoop-aws.html JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9845/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9845/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          patch 009: docs, checkstyle and findbugs

          I have not addressed those bits of the checkstyle complaints about constants called _128K in tests, as it's only testing and the name is the value.

          Show
          stevel@apache.org Steve Loughran added a comment - patch 009: docs, checkstyle and findbugs I have not addressed those bits of the checkstyle complaints about constants called _128K in tests, as it's only testing and the name is the value.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          patch 010: chris's review, +one other IDE complaint about mixed-sync use of a field.

          test run in progress

          Show
          stevel@apache.org Steve Loughran added a comment - patch 010: chris's review, +one other IDE complaint about mixed-sync use of a field. test run in progress
          Hide
          stevel@apache.org Steve Loughran added a comment -

          parallel test run against s3 ireland: completed in < 9 minutes; failure of TestS3AContractRootDir which went away when run standalone ... some race conditions/consistency condition to look at there

          Show
          stevel@apache.org Steve Loughran added a comment - parallel test run against s3 ireland: completed in < 9 minutes; failure of TestS3AContractRootDir which went away when run standalone ... some race conditions/consistency condition to look at there
          Hide
          stevel@apache.org Steve Loughran added a comment -

          BTW, this patch enhances the range validation checks in FSInputStream so that on a block read where the length > buffer capacity, the details of the request are included in the exception. You'll appreciate this if you ever have problems here.

          Show
          stevel@apache.org Steve Loughran added a comment - BTW, this patch enhances the range validation checks in FSInputStream so that on a block read where the length > buffer capacity, the details of the request are included in the exception. You'll appreciate this if you ever have problems here.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 27s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          0 mvndep 0m 15s Maven dependency ordering for branch
          +1 mvninstall 7m 9s branch-2 passed
          +1 compile 7m 2s branch-2 passed with JDK v1.8.0_91
          +1 compile 6m 46s branch-2 passed with JDK v1.7.0_101
          +1 checkstyle 1m 24s branch-2 passed
          +1 mvnsite 1m 23s branch-2 passed
          +1 mvneclipse 0m 29s branch-2 passed
          +1 findbugs 2m 14s branch-2 passed
          +1 javadoc 1m 2s branch-2 passed with JDK v1.8.0_91
          +1 javadoc 1m 14s branch-2 passed with JDK v1.7.0_101
          0 mvndep 0m 14s Maven dependency ordering for patch
          +1 mvninstall 0m 59s the patch passed
          +1 compile 6m 46s the patch passed with JDK v1.8.0_91
          +1 javac 6m 46s the patch passed
          +1 compile 8m 19s the patch passed with JDK v1.7.0_101
          +1 javac 8m 19s the patch passed
          -1 checkstyle 1m 33s root: The patch generated 20 new + 43 unchanged - 9 fixed = 63 total (was 52)
          +1 mvnsite 1m 40s the patch passed
          +1 mvneclipse 0m 33s the patch passed
          -1 whitespace 0m 0s The patch has 50 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 findbugs 3m 2s the patch passed
          +1 javadoc 1m 15s the patch passed with JDK v1.8.0_91
          +1 javadoc 1m 21s the patch passed with JDK v1.7.0_101
          -1 unit 9m 2s hadoop-common in the patch failed with JDK v1.7.0_101.
          +1 unit 0m 17s hadoop-aws in the patch passed with JDK v1.7.0_101.
          +1 asflicense 0m 23s The patch does not generate ASF License warnings.
          76m 39s



          Reason Tests
          JDK v1.7.0_101 Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics
            hadoop.security.ssl.TestReloadingX509TrustManager



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:d1c475d
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12812246/HADOOP-13203-branch-2-009.patch
          JIRA Issue HADOOP-13203
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux b133a7b498c8 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2 / 65b4f26
          Default Java 1.7.0_101
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/9846/artifact/patchprocess/diff-checkstyle-root.txt
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/9846/artifact/patchprocess/whitespace-eol.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9846/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_101.txt
          JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9846/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9846/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 27s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 0m 15s Maven dependency ordering for branch +1 mvninstall 7m 9s branch-2 passed +1 compile 7m 2s branch-2 passed with JDK v1.8.0_91 +1 compile 6m 46s branch-2 passed with JDK v1.7.0_101 +1 checkstyle 1m 24s branch-2 passed +1 mvnsite 1m 23s branch-2 passed +1 mvneclipse 0m 29s branch-2 passed +1 findbugs 2m 14s branch-2 passed +1 javadoc 1m 2s branch-2 passed with JDK v1.8.0_91 +1 javadoc 1m 14s branch-2 passed with JDK v1.7.0_101 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 0m 59s the patch passed +1 compile 6m 46s the patch passed with JDK v1.8.0_91 +1 javac 6m 46s the patch passed +1 compile 8m 19s the patch passed with JDK v1.7.0_101 +1 javac 8m 19s the patch passed -1 checkstyle 1m 33s root: The patch generated 20 new + 43 unchanged - 9 fixed = 63 total (was 52) +1 mvnsite 1m 40s the patch passed +1 mvneclipse 0m 33s the patch passed -1 whitespace 0m 0s The patch has 50 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 findbugs 3m 2s the patch passed +1 javadoc 1m 15s the patch passed with JDK v1.8.0_91 +1 javadoc 1m 21s the patch passed with JDK v1.7.0_101 -1 unit 9m 2s hadoop-common in the patch failed with JDK v1.7.0_101. +1 unit 0m 17s hadoop-aws in the patch passed with JDK v1.7.0_101. +1 asflicense 0m 23s The patch does not generate ASF License warnings. 76m 39s Reason Tests JDK v1.7.0_101 Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics   hadoop.security.ssl.TestReloadingX509TrustManager Subsystem Report/Notes Docker Image:yetus/hadoop:d1c475d JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12812246/HADOOP-13203-branch-2-009.patch JIRA Issue HADOOP-13203 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux b133a7b498c8 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / 65b4f26 Default Java 1.7.0_101 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/9846/artifact/patchprocess/diff-checkstyle-root.txt whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/9846/artifact/patchprocess/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9846/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_101.txt JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9846/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9846/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          cnauroth Chris Nauroth added a comment -

          +1 for patch 010, pending pre-commit. I'll also kick off another full test run against S3.

          failure of TestS3AContractRootDir which went away when run standalone ... some race conditions/consistency condition to look at there

          If this was a run with both -Pparallel-tests and -Dtest=TestS3A*, then it's probably the problem we discussed elsewhere that passing these arguments would erroneously include TestS3AContractRootDir in the parallel testing phase.

          Show
          cnauroth Chris Nauroth added a comment - +1 for patch 010, pending pre-commit. I'll also kick off another full test run against S3. failure of TestS3AContractRootDir which went away when run standalone ... some race conditions/consistency condition to look at there If this was a run with both -Pparallel-tests and -Dtest=TestS3A* , then it's probably the problem we discussed elsewhere that passing these arguments would erroneously include TestS3AContractRootDir in the parallel testing phase.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          thanks...given the precommit went through I'll take the +1 as confirmed, run another test against the patch rebased onto branch 2

          regarding the failure, it is in the serialzied set; HADOOP-13271 looks at it.

          I think it's a delayed delete visibility issue, or similar; the test is in hadoop-common so might need some working to isolate and handle consistency...probably initially pull it out into its own. The fact that it doesn't fail standalone makes it hard though.

          Show
          stevel@apache.org Steve Loughran added a comment - thanks...given the precommit went through I'll take the +1 as confirmed, run another test against the patch rebased onto branch 2 regarding the failure, it is in the serialzied set; HADOOP-13271 looks at it. I think it's a delayed delete visibility issue, or similar; the test is in hadoop-common so might need some working to isolate and handle consistency...probably initially pull it out into its own. The fact that it doesn't fail standalone makes it hard though.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #9999 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9999/)
          HADOOP-13203 S3A: Support fadvise "random" mode for high performance (stevel: rev 4ee3543625c77c06d566fe81644d21c607d6d74d)

          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Statistic.java
          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AInstrumentation.java
          • hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/scale/TestS3AInputStreamPerformance.java
          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
          • hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3AInputPolicies.java
          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AInputPolicy.java
          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/AbstractContractSeekTest.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FSInputStream.java
          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AInputStream.java
          • hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #9999 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9999/ ) HADOOP-13203 S3A: Support fadvise "random" mode for high performance (stevel: rev 4ee3543625c77c06d566fe81644d21c607d6d74d) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Statistic.java hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AInstrumentation.java hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/scale/TestS3AInputStreamPerformance.java hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3AInputPolicies.java hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AInputPolicy.java hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/AbstractContractSeekTest.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FSInputStream.java hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AInputStream.java hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md
          Hide
          mmokhtar Mostafa Mokhtar added a comment -
          Show
          mmokhtar Mostafa Mokhtar added a comment - Sailesh Mukil FYI
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Mostafa: that's not really a constructive use of comments on ASF JIRAs, as it ends up sending an email to everyone who has watched or worked on a JIRA. Just point your colleagues at them. Thanks

          Show
          stevel@apache.org Steve Loughran added a comment - Mostafa: that's not really a constructive use of comments on ASF JIRAs, as it ends up sending an email to everyone who has watched or worked on a JIRA. Just point your colleagues at them. Thanks

            People

            • Assignee:
              rajesh.balamohan Rajesh Balamohan
              Reporter:
              rajesh.balamohan Rajesh Balamohan
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development