Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-14680

Azure: IndexOutOfBoundsException in BlockBlobInputStream

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-beta1
    • Component/s: fs/azure
    • Labels:
      None

      Description

      https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/BlockBlobInputStream.java#L361

      On certain conditions, BlockBlobInputStream can throw IndexOutOfBoundsException. Following is an example

      length:297898, offset:4194304, buf.len:4492202, writePos:4194304 :
      In this case, MemoryOutputStream::capacity() would end up returning negative value and can cause IndexOutOfBoundsException

      It should be return buffer.length - offset; to determine current capacity.

      1. HADOOP-14680-001.patch
        3 kB
        Thomas Marquardt
      2. HADOOP-14680-branch-2.01.patch
        4 kB
        Jitendra Nath Pandey

        Issue Links

          Activity

          Hide
          shanem Shane Mainali added a comment -

          You can assign this to Thomas Marquardt, he will help fix it as it looks related to the seek/random read changes.

          Show
          shanem Shane Mainali added a comment - You can assign this to Thomas Marquardt , he will help fix it as it looks related to the seek/random read changes.
          Hide
          tmarquardt Thomas Marquardt added a comment -

          I'm looking at it now...

          Show
          tmarquardt Thomas Marquardt added a comment - I'm looking at it now...
          Hide
          tmarquardt Thomas Marquardt added a comment -

          Attaching HADOOP-14680-001.patch.

          This fixes the issure where MemoryOutputStream.capacity() returns the wrong value. The function was supposed to return the total capacity of the stream and it now does that.

          A new test case was added. The new test case fails without the fix.

          Show
          tmarquardt Thomas Marquardt added a comment - Attaching HADOOP-14680 -001.patch. This fixes the issure where MemoryOutputStream.capacity() returns the wrong value. The function was supposed to return the total capacity of the stream and it now does that. A new test case was added. The new test case fails without the fix.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
                trunk Compile Tests
          +1 mvninstall 15m 5s trunk passed
          +1 compile 0m 19s trunk passed
          +1 checkstyle 0m 14s trunk passed
          +1 mvnsite 0m 21s trunk passed
          +1 findbugs 0m 29s trunk passed
          +1 javadoc 0m 14s trunk passed
                Patch Compile Tests
          +1 mvninstall 0m 17s the patch passed
          +1 compile 0m 16s the patch passed
          +1 javac 0m 16s the patch passed
          -0 checkstyle 0m 11s hadoop-tools/hadoop-azure: The patch generated 1 new + 26 unchanged - 2 fixed = 27 total (was 28)
          +1 mvnsite 0m 18s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 0m 33s the patch passed
          +1 javadoc 0m 11s the patch passed
                Other Tests
          +1 unit 1m 24s hadoop-azure in the patch passed.
          +1 asflicense 0m 18s The patch does not generate ASF License warnings.
          21m 38s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:14b5c93
          JIRA Issue HADOOP-14680
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12878857/HADOOP-14680-001.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux ef1170dae841 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / f81a4ef
          Default Java 1.8.0_131
          findbugs v3.1.0-RC1
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/12855/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-azure.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12855/testReport/
          modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12855/console
          Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       trunk Compile Tests +1 mvninstall 15m 5s trunk passed +1 compile 0m 19s trunk passed +1 checkstyle 0m 14s trunk passed +1 mvnsite 0m 21s trunk passed +1 findbugs 0m 29s trunk passed +1 javadoc 0m 14s trunk passed       Patch Compile Tests +1 mvninstall 0m 17s the patch passed +1 compile 0m 16s the patch passed +1 javac 0m 16s the patch passed -0 checkstyle 0m 11s hadoop-tools/hadoop-azure: The patch generated 1 new + 26 unchanged - 2 fixed = 27 total (was 28) +1 mvnsite 0m 18s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 33s the patch passed +1 javadoc 0m 11s the patch passed       Other Tests +1 unit 1m 24s hadoop-azure in the patch passed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 21m 38s Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HADOOP-14680 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12878857/HADOOP-14680-001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux ef1170dae841 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / f81a4ef Default Java 1.8.0_131 findbugs v3.1.0-RC1 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/12855/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-azure.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12855/testReport/ modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12855/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          tmarquardt Thomas Marquardt added a comment -

          All WASB tests are passing with the HADOOP-14680-001.patch except one unrelated test failure:

          Tests run: 762, Failures: 1, Errors: 0, Skipped: 70

          Failed tests: TestWasbRemoteCallHelper.testWhenOneInstanceIsDown:359

          The test failure is unrelated to the HADOOP-14680-001.patch. It may be related to HADOOP-14642 (commit #2843c6)

          Show
          tmarquardt Thomas Marquardt added a comment - All WASB tests are passing with the HADOOP-14680 -001.patch except one unrelated test failure: Tests run: 762, Failures: 1, Errors: 0, Skipped: 70 Failed tests: TestWasbRemoteCallHelper.testWhenOneInstanceIsDown:359 The test failure is unrelated to the HADOOP-14680 -001.patch. It may be related to HADOOP-14642 (commit #2843c6)
          Hide
          rajesh.balamohan Rajesh Balamohan added a comment -

          Thanks for the patch Thomas Marquardt. Patch lgtm (non-binding). I tried out the patch on multi-node cluster and it works fine.

          Show
          rajesh.balamohan Rajesh Balamohan added a comment - Thanks for the patch Thomas Marquardt . Patch lgtm (non-binding). I tried out the patch on multi-node cluster and it works fine.
          Hide
          shanem Shane Mainali added a comment -

          Reviewed the patch with Thomas as well and it looks good. Thanks Thomas Marquardt!

          Show
          shanem Shane Mainali added a comment - Reviewed the patch with Thomas as well and it looks good. Thanks Thomas Marquardt !
          Hide
          jnp Jitendra Nath Pandey added a comment -

          +1. I will commit the patch shortly.

          Show
          jnp Jitendra Nath Pandey added a comment - +1. I will commit the patch shortly.
          Hide
          jnp Jitendra Nath Pandey added a comment - - edited

          I have committed this to trunk. Attached branch-2 patch for jenkins.

          Show
          jnp Jitendra Nath Pandey added a comment - - edited I have committed this to trunk. Attached branch-2 patch for jenkins.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12055 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12055/)
          HADOOP-14680. Azure: IndexOutOfBoundsException in BlockBlobInputStream. (jitendra: rev a92bf39e2313d4bfccd641ce0ccefe26f4903a69)

          • (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/BlockBlobInputStream.java
          • (edit) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestBlockBlobInputStream.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12055 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12055/ ) HADOOP-14680 . Azure: IndexOutOfBoundsException in BlockBlobInputStream. (jitendra: rev a92bf39e2313d4bfccd641ce0ccefe26f4903a69) (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/BlockBlobInputStream.java (edit) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestBlockBlobInputStream.java
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 15s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
                branch-2 Compile Tests
          +1 mvninstall 7m 14s branch-2 passed
          +1 compile 0m 18s branch-2 passed with JDK v1.8.0_131
          +1 compile 0m 20s branch-2 passed with JDK v1.7.0_131
          +1 checkstyle 0m 16s branch-2 passed
          +1 mvnsite 0m 25s branch-2 passed
          +1 findbugs 0m 36s branch-2 passed
          +1 javadoc 0m 14s branch-2 passed with JDK v1.8.0_131
          +1 javadoc 0m 15s branch-2 passed with JDK v1.7.0_131
                Patch Compile Tests
          +1 mvninstall 0m 19s the patch passed
          +1 compile 0m 16s the patch passed with JDK v1.8.0_131
          +1 javac 0m 16s the patch passed
          +1 compile 0m 17s the patch passed with JDK v1.7.0_131
          +1 javac 0m 17s the patch passed
          -0 checkstyle 0m 25s hadoop-tools/hadoop-azure: The patch generated 1 new + 26 unchanged - 2 fixed = 27 total (was 28)
          +1 mvnsite 0m 24s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 0m 44s the patch passed
          +1 javadoc 0m 11s the patch passed with JDK v1.8.0_131
          +1 javadoc 0m 14s the patch passed with JDK v1.7.0_131
                Other Tests
          +1 unit 1m 39s hadoop-azure in the patch passed with JDK v1.7.0_131.
          +1 asflicense 0m 19s The patch does not generate ASF License warnings.
          17m 55s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:5e40efe
          JIRA Issue HADOOP-14680
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12878923/HADOOP-14680-branch-2.01.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux daff016a03cc 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2 / 00f4a80
          Default Java 1.7.0_131
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_131 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_131
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/12856/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-azure.txt
          JDK v1.7.0_131 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12856/testReport/
          modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12856/console
          Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       branch-2 Compile Tests +1 mvninstall 7m 14s branch-2 passed +1 compile 0m 18s branch-2 passed with JDK v1.8.0_131 +1 compile 0m 20s branch-2 passed with JDK v1.7.0_131 +1 checkstyle 0m 16s branch-2 passed +1 mvnsite 0m 25s branch-2 passed +1 findbugs 0m 36s branch-2 passed +1 javadoc 0m 14s branch-2 passed with JDK v1.8.0_131 +1 javadoc 0m 15s branch-2 passed with JDK v1.7.0_131       Patch Compile Tests +1 mvninstall 0m 19s the patch passed +1 compile 0m 16s the patch passed with JDK v1.8.0_131 +1 javac 0m 16s the patch passed +1 compile 0m 17s the patch passed with JDK v1.7.0_131 +1 javac 0m 17s the patch passed -0 checkstyle 0m 25s hadoop-tools/hadoop-azure: The patch generated 1 new + 26 unchanged - 2 fixed = 27 total (was 28) +1 mvnsite 0m 24s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 44s the patch passed +1 javadoc 0m 11s the patch passed with JDK v1.8.0_131 +1 javadoc 0m 14s the patch passed with JDK v1.7.0_131       Other Tests +1 unit 1m 39s hadoop-azure in the patch passed with JDK v1.7.0_131. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 17m 55s Subsystem Report/Notes Docker Image:yetus/hadoop:5e40efe JIRA Issue HADOOP-14680 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12878923/HADOOP-14680-branch-2.01.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux daff016a03cc 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / 00f4a80 Default Java 1.7.0_131 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_131 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_131 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/12856/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-azure.txt JDK v1.7.0_131 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12856/testReport/ modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12856/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          its a bit moot now the patch is in, but please be aware that the error message raised when assertEquals fails only makes sense if the constant/expected value is the first arg, not the second. That is assertEquals(numBytesReadV1, KILOBYTE) should have been assertEquals( KILOBYTE, numBytesReadV1). It doesn't merit a patch here, but its something to remember in future. thanks

          Show
          stevel@apache.org Steve Loughran added a comment - its a bit moot now the patch is in, but please be aware that the error message raised when assertEquals fails only makes sense if the constant/expected value is the first arg, not the second. That is assertEquals(numBytesReadV1, KILOBYTE) should have been assertEquals( KILOBYTE, numBytesReadV1) . It doesn't merit a patch here, but its something to remember in future. thanks

            People

            • Assignee:
              tmarquardt Thomas Marquardt
              Reporter:
              rajesh.balamohan Rajesh Balamohan
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development