Details
-
Type:
Bug
-
Status: Resolved
-
Priority:
Minor
-
Resolution: Fixed
-
Affects Version/s: None
-
Fix Version/s: 2.9.0, 3.0.0-beta1
-
Component/s: fs/azure
-
Labels:None
-
Target Version/s:
-
Release Note:Bug fix to Azure Filesystem related to
HADOOP-14535
Description
On certain conditions, BlockBlobInputStream can throw IndexOutOfBoundsException. Following is an example
length:297898, offset:4194304, buf.len:4492202, writePos:4194304 :
In this case, MemoryOutputStream::capacity() would end up returning negative value and can cause IndexOutOfBoundsException
It should be return buffer.length - offset; to determine current capacity.
Issue Links
- is depended upon by
-
HADOOP-14535
wasb: implement high-performance random access and seek of block blobs
-
- Resolved
-
- is related to
-
HADOOP-14552
Über-jira: WASB client phase II: performance and testing
-
- Open
-
Activity
- All
- Comments
- Work Log
- History
- Activity
- Transitions
I'm looking at it now...
Attaching HADOOP-14680-001.patch.
This fixes the issure where MemoryOutputStream.capacity() returns the wrong value. The function was supposed to return the total capacity of the stream and it now does that.
A new test case was added. The new test case fails without the fix.
 +1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | reexec | 0m 18s | Docker mode activated. |
| Prechecks | |||
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
| trunk Compile Tests | |||
| +1 | mvninstall | 15m 5s | trunk passed |
| +1 | compile | 0m 19s | trunk passed |
| +1 | checkstyle | 0m 14s | trunk passed |
| +1 | mvnsite | 0m 21s | trunk passed |
| +1 | findbugs | 0m 29s | trunk passed |
| +1 | javadoc | 0m 14s | trunk passed |
| Patch Compile Tests | |||
| +1 | mvninstall | 0m 17s | the patch passed |
| +1 | compile | 0m 16s | the patch passed |
| +1 | javac | 0m 16s | the patch passed |
| -0 | checkstyle | 0m 11s | hadoop-tools/hadoop-azure: The patch generated 1 new + 26 unchanged - 2 fixed = 27 total (was 28) |
| +1 | mvnsite | 0m 18s | the patch passed |
| +1 | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 | findbugs | 0m 33s | the patch passed |
| +1 | javadoc | 0m 11s | the patch passed |
| Other Tests | |||
| +1 | unit | 1m 24s | hadoop-azure in the patch passed. |
| +1 | asflicense | 0m 18s | The patch does not generate ASF License warnings. |
| 21m 38s |
| Subsystem | Report/Notes |
|---|---|
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12878857/HADOOP-14680-001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle |
| uname | Linux ef1170dae841 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / f81a4ef |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| checkstyle | https://builds.apache.org/job/PreCommit-HADOOP-Build/12855/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-azure.txt |
| Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/12855/testReport/ |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/12855/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
All WASB tests are passing with the HADOOP-14680-001.patch except one unrelated test failure:
Tests run: 762, Failures: 1, Errors: 0, Skipped: 70
Failed tests: TestWasbRemoteCallHelper.testWhenOneInstanceIsDown:359
The test failure is unrelated to the HADOOP-14680-001.patch. It may be related to HADOOP-14642 (commit #2843c6)
Thanks for the patch Thomas Marquardt. Patch lgtm (non-binding). I tried out the patch on multi-node cluster and it works fine.
Reviewed the patch with Thomas as well and it looks good. Thanks Thomas Marquardt!
+1. I will commit the patch shortly.
I have committed this to trunk. Attached branch-2 patch for jenkins.
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12055 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12055/)
HADOOP-14680. Azure: IndexOutOfBoundsException in BlockBlobInputStream. (jitendra: rev a92bf39e2313d4bfccd641ce0ccefe26f4903a69)
- (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/BlockBlobInputStream.java
- (edit) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestBlockBlobInputStream.java
| +1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | reexec | 0m 15s | Docker mode activated. |
| Prechecks | |||
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
| branch-2 Compile Tests | |||
| +1 | mvninstall | 7m 14s | branch-2 passed |
| +1 | compile | 0m 18s | branch-2 passed with JDK v1.8.0_131 |
| +1 | compile | 0m 20s | branch-2 passed with JDK v1.7.0_131 |
| +1 | checkstyle | 0m 16s | branch-2 passed |
| +1 | mvnsite | 0m 25s | branch-2 passed |
| +1 | findbugs | 0m 36s | branch-2 passed |
| +1 | javadoc | 0m 14s | branch-2 passed with JDK v1.8.0_131 |
| +1 | javadoc | 0m 15s | branch-2 passed with JDK v1.7.0_131 |
| Patch Compile Tests | |||
| +1 | mvninstall | 0m 19s | the patch passed |
| +1 | compile | 0m 16s | the patch passed with JDK v1.8.0_131 |
| +1 | javac | 0m 16s | the patch passed |
| +1 | compile | 0m 17s | the patch passed with JDK v1.7.0_131 |
| +1 | javac | 0m 17s | the patch passed |
| -0 | checkstyle | 0m 25s | hadoop-tools/hadoop-azure: The patch generated 1 new + 26 unchanged - 2 fixed = 27 total (was 28) |
| +1 | mvnsite | 0m 24s | the patch passed |
| +1 | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 | findbugs | 0m 44s | the patch passed |
| +1 | javadoc | 0m 11s | the patch passed with JDK v1.8.0_131 |
| +1 | javadoc | 0m 14s | the patch passed with JDK v1.7.0_131 |
| Other Tests | |||
| +1 | unit | 1m 39s | hadoop-azure in the patch passed with JDK v1.7.0_131. |
| +1 | asflicense | 0m 19s | The patch does not generate ASF License warnings. |
| 17m 55s |
| Subsystem | Report/Notes |
|---|---|
| Docker | Image:yetus/hadoop:5e40efe |
| JIRA Issue | |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12878923/HADOOP-14680-branch-2.01.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle |
| uname | Linux daff016a03cc 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
| git revision | branch-2 / 00f4a80 |
| Default Java | 1.7.0_131 |
| Multi-JDK versions | /usr/lib/jvm/java-8-oracle:1.8.0_131 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_131 |
| findbugs | v3.0.0 |
| checkstyle | https://builds.apache.org/job/PreCommit-HADOOP-Build/12856/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-azure.txt |
| JDK v1.7.0_131 Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/12856/testReport/ |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/12856/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
its a bit moot now the patch is in, but please be aware that the error message raised when assertEquals fails only makes sense if the constant/expected value is the first arg, not the second. That is assertEquals(numBytesReadV1, KILOBYTE) should have been assertEquals( KILOBYTE, numBytesReadV1). It doesn't merit a patch here, but its something to remember in future. thanks
You can assign this to Thomas Marquardt, he will help fix it as it looks related to the seek/random read changes.