Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13017

Implementations of InputStream.read(buffer, offset, bytes) to exit 0 if bytes==0

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.8.0
    • Fix Version/s: 2.8.0
    • Component/s: fs, io
    • Labels:
      None

      Description

      HDFS-10277 showed that HDFS was return -1 on read(buf[], 0, 0) when there was no data left in the stream; Java IO says

      If len is zero, then no bytes are read and 0 is returned; otherwise, there is an attempt to read at least one byte.

      Review the implementations of {{IOStream.(buffer, offset, bytes)} and, where necessary and considered safe, add a fast exit if the length is 0.

      1. HADOOP-13017-002.patch
        6 kB
        Steve Loughran
      2. HDFS-13017-001.patch
        5 kB
        Steve Loughran

        Issue Links

          Activity

          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 11m 39s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          0 mvndep 0m 58s Maven dependency ordering for branch
          +1 mvninstall 6m 56s trunk passed
          +1 compile 6m 5s trunk passed with JDK v1.8.0_77
          +1 compile 6m 43s trunk passed with JDK v1.7.0_95
          +1 checkstyle 1m 7s trunk passed
          +1 mvnsite 1m 58s trunk passed
          +1 mvneclipse 0m 53s trunk passed
          +1 findbugs 3m 6s trunk passed
          +1 javadoc 1m 32s trunk passed with JDK v1.8.0_77
          +1 javadoc 1m 47s trunk passed with JDK v1.7.0_95
          0 mvndep 0m 15s Maven dependency ordering for patch
          +1 mvninstall 1m 25s the patch passed
          +1 compile 6m 48s the patch passed with JDK v1.8.0_77
          +1 javac 6m 48s the patch passed
          +1 compile 6m 57s the patch passed with JDK v1.7.0_95
          +1 javac 6m 57s the patch passed
          +1 checkstyle 1m 7s the patch passed
          +1 mvnsite 1m 54s the patch passed
          +1 mvneclipse 0m 52s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 3m 54s the patch passed
          +1 javadoc 1m 35s the patch passed with JDK v1.8.0_77
          +1 javadoc 1m 47s the patch passed with JDK v1.7.0_95
          -1 unit 7m 21s hadoop-common in the patch failed with JDK v1.8.0_77.
          +1 unit 8m 17s hadoop-distcp in the patch passed with JDK v1.8.0_77.
          +1 unit 0m 12s hadoop-openstack in the patch passed with JDK v1.8.0_77.
          +1 unit 0m 12s hadoop-aws in the patch passed with JDK v1.8.0_77.
          +1 unit 7m 39s hadoop-common in the patch passed with JDK v1.7.0_95.
          +1 unit 7m 28s hadoop-distcp in the patch passed with JDK v1.7.0_95.
          +1 unit 0m 14s hadoop-openstack in the patch passed with JDK v1.7.0_95.
          +1 unit 0m 17s hadoop-aws in the patch passed with JDK v1.7.0_95.
          +1 asflicense 0m 24s Patch does not generate ASF License warnings.
          103m 4s



          Reason Tests
          JDK v1.8.0_77 Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:fbe3e86
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12798112/HDFS-13017-001.patch
          JIRA Issue HADOOP-13017
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 96148087324a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 44bbc50
          Default Java 1.7.0_95
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9060/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_77.txt
          unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/9060/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_77.txt
          JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9060/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-distcp hadoop-tools/hadoop-openstack hadoop-tools/hadoop-aws U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9060/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 11m 39s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 0m 58s Maven dependency ordering for branch +1 mvninstall 6m 56s trunk passed +1 compile 6m 5s trunk passed with JDK v1.8.0_77 +1 compile 6m 43s trunk passed with JDK v1.7.0_95 +1 checkstyle 1m 7s trunk passed +1 mvnsite 1m 58s trunk passed +1 mvneclipse 0m 53s trunk passed +1 findbugs 3m 6s trunk passed +1 javadoc 1m 32s trunk passed with JDK v1.8.0_77 +1 javadoc 1m 47s trunk passed with JDK v1.7.0_95 0 mvndep 0m 15s Maven dependency ordering for patch +1 mvninstall 1m 25s the patch passed +1 compile 6m 48s the patch passed with JDK v1.8.0_77 +1 javac 6m 48s the patch passed +1 compile 6m 57s the patch passed with JDK v1.7.0_95 +1 javac 6m 57s the patch passed +1 checkstyle 1m 7s the patch passed +1 mvnsite 1m 54s the patch passed +1 mvneclipse 0m 52s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 3m 54s the patch passed +1 javadoc 1m 35s the patch passed with JDK v1.8.0_77 +1 javadoc 1m 47s the patch passed with JDK v1.7.0_95 -1 unit 7m 21s hadoop-common in the patch failed with JDK v1.8.0_77. +1 unit 8m 17s hadoop-distcp in the patch passed with JDK v1.8.0_77. +1 unit 0m 12s hadoop-openstack in the patch passed with JDK v1.8.0_77. +1 unit 0m 12s hadoop-aws in the patch passed with JDK v1.8.0_77. +1 unit 7m 39s hadoop-common in the patch passed with JDK v1.7.0_95. +1 unit 7m 28s hadoop-distcp in the patch passed with JDK v1.7.0_95. +1 unit 0m 14s hadoop-openstack in the patch passed with JDK v1.7.0_95. +1 unit 0m 17s hadoop-aws in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 24s Patch does not generate ASF License warnings. 103m 4s Reason Tests JDK v1.8.0_77 Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12798112/HDFS-13017-001.patch JIRA Issue HADOOP-13017 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 96148087324a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 44bbc50 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/9060/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_77.txt unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/9060/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_77.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9060/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-distcp hadoop-tools/hadoop-openstack hadoop-tools/hadoop-aws U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9060/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          There's a test for this in the test seek contract test, but an explicit one can be added to another contract test, as it isn't directly seek related; it just went in as part of the positioned readable

          Show
          stevel@apache.org Steve Loughran added a comment - There's a test for this in the test seek contract test, but an explicit one can be added to another contract test, as it isn't directly seek related; it just went in as part of the positioned readable
          Hide
          iwasakims Masatake Iwasaki added a comment -

          The patch looks good, though the part fixing S3InputStream needs update.

          HarFsInputStream.read and WebHdfsInputStream.read seems to be able to do fast exit too.

          Should the issue title say InputStream.read rather than IOStream.read?

          Show
          iwasakims Masatake Iwasaki added a comment - The patch looks good, though the part fixing S3InputStream needs update. HarFsInputStream.read and WebHdfsInputStream.read seems to be able to do fast exit too. Should the issue title say InputStream.read rather than IOStream.read?
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          -1 patch 0m 5s HADOOP-13017 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



          Subsystem Report/Notes
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12798112/HDFS-13017-001.patch
          JIRA Issue HADOOP-13017
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10076/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 5s HADOOP-13017 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12798112/HDFS-13017-001.patch JIRA Issue HADOOP-13017 Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10076/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Patch 002; adds Har and WebHdfs streams; cuts s3InputStream out

          Show
          stevel@apache.org Steve Loughran added a comment - Patch 002; adds Har and WebHdfs streams; cuts s3InputStream out
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          0 mvndep 1m 37s Maven dependency ordering for branch
          +1 mvninstall 7m 25s trunk passed
          +1 compile 6m 58s trunk passed
          +1 checkstyle 1m 29s trunk passed
          +1 mvnsite 2m 9s trunk passed
          +1 mvneclipse 0m 53s trunk passed
          +1 findbugs 3m 31s trunk passed
          +1 javadoc 1m 34s trunk passed
          0 mvndep 0m 15s Maven dependency ordering for patch
          +1 mvninstall 1m 53s the patch passed
          +1 compile 7m 59s the patch passed
          +1 javac 7m 59s the patch passed
          +1 checkstyle 1m 38s the patch passed
          +1 mvnsite 2m 24s the patch passed
          +1 mvneclipse 1m 1s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 4m 54s the patch passed
          +1 javadoc 1m 47s the patch passed
          +1 unit 7m 40s hadoop-common in the patch passed.
          +1 unit 1m 14s hadoop-hdfs-client in the patch passed.
          +1 unit 10m 59s hadoop-distcp in the patch passed.
          +1 unit 0m 19s hadoop-openstack in the patch passed.
          +1 asflicense 0m 27s The patch does not generate ASF License warnings.
          91m 44s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HADOOP-13017
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12834982/HADOOP-13017-002.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux d46896bbdb93 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / b18f35f
          Default Java 1.8.0_101
          findbugs v3.0.0
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10876/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-tools/hadoop-distcp hadoop-tools/hadoop-openstack U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10876/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 1m 37s Maven dependency ordering for branch +1 mvninstall 7m 25s trunk passed +1 compile 6m 58s trunk passed +1 checkstyle 1m 29s trunk passed +1 mvnsite 2m 9s trunk passed +1 mvneclipse 0m 53s trunk passed +1 findbugs 3m 31s trunk passed +1 javadoc 1m 34s trunk passed 0 mvndep 0m 15s Maven dependency ordering for patch +1 mvninstall 1m 53s the patch passed +1 compile 7m 59s the patch passed +1 javac 7m 59s the patch passed +1 checkstyle 1m 38s the patch passed +1 mvnsite 2m 24s the patch passed +1 mvneclipse 1m 1s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 4m 54s the patch passed +1 javadoc 1m 47s the patch passed +1 unit 7m 40s hadoop-common in the patch passed. +1 unit 1m 14s hadoop-hdfs-client in the patch passed. +1 unit 10m 59s hadoop-distcp in the patch passed. +1 unit 0m 19s hadoop-openstack in the patch passed. +1 asflicense 0m 27s The patch does not generate ASF License warnings. 91m 44s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HADOOP-13017 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12834982/HADOOP-13017-002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux d46896bbdb93 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / b18f35f Default Java 1.8.0_101 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10876/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-tools/hadoop-distcp hadoop-tools/hadoop-openstack U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10876/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          iwasakims Masatake Iwasaki added a comment -

          +1, committing this.

          Show
          iwasakims Masatake Iwasaki added a comment - +1, committing this.
          Hide
          iwasakims Masatake Iwasaki added a comment -

          Committed. Thanks, Steve Loughran.

          Show
          iwasakims Masatake Iwasaki added a comment - Committed. Thanks, Steve Loughran .
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10696 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10696/)
          HADOOP-13017. Implementations of InputStream.read(buffer, offset, bytes) (iwasakims: rev 0bdd263d82a4510f16df49238d57c9f78ac28ae7)

          • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/HarFileSystem.java
          • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SaslInputStream.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
          • (edit) hadoop-tools/hadoop-openstack/src/main/java/org/apache/hadoop/fs/swift/http/HttpInputStreamWithRelease.java
          • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LimitInputStream.java
          • (edit) hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/util/ThrottledInputStream.java
          • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SaslRpcClient.java
          • (edit) hadoop-tools/hadoop-openstack/src/main/java/org/apache/hadoop/fs/swift/snative/SwiftNativeInputStream.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10696 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10696/ ) HADOOP-13017 . Implementations of InputStream.read(buffer, offset, bytes) (iwasakims: rev 0bdd263d82a4510f16df49238d57c9f78ac28ae7) (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/HarFileSystem.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SaslInputStream.java (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java (edit) hadoop-tools/hadoop-openstack/src/main/java/org/apache/hadoop/fs/swift/http/HttpInputStreamWithRelease.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LimitInputStream.java (edit) hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/util/ThrottledInputStream.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SaslRpcClient.java (edit) hadoop-tools/hadoop-openstack/src/main/java/org/apache/hadoop/fs/swift/snative/SwiftNativeInputStream.java
          Hide
          stevel@apache.org Steve Loughran added a comment -

          thanks for the review

          Show
          stevel@apache.org Steve Loughran added a comment - thanks for the review
          Hide
          ajisakaa Akira Ajisaka added a comment -

          TestHadoopArchives times out due to this change. Hi Masatake Iwasaki and Steve Loughran, would you review the patch in HADOOP-13798?

          Show
          ajisakaa Akira Ajisaka added a comment - TestHadoopArchives times out due to this change. Hi Masatake Iwasaki and Steve Loughran , would you review the patch in HADOOP-13798 ?

            People

            • Assignee:
              stevel@apache.org Steve Loughran
              Reporter:
              stevel@apache.org Steve Loughran
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development