Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
2.7.1
-
Reviewed
Description
Azure FileSystem PageBlobInputStream does not return -1 on EOF. This is some scenarios causes infinite hands on reading files (e.g. copyToLocal can hang forever).
Attachments
Attachments
- HADOOP-12073.2.patch
- 12 kB
- Ivan Mitic
- HADOOP-12073.3.patch
- 12 kB
- Ivan Mitic
- HADOOP-12073.patch
- 12 kB
- Ivan Mitic
Activity
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | pre-patch | 15m 9s | Pre-patch trunk compilation is healthy. |
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | tests included | 0m 0s | The patch appears to include 2 new or modified test files. |
-1 | javac | 7m 28s | The applied patch generated 1 additional warning messages. |
+1 | javadoc | 9m 36s | There were no new javadoc warning messages. |
+1 | release audit | 0m 22s | The applied patch does not increase the total number of release audit warnings. |
+1 | checkstyle | 0m 22s | There were no new checkstyle issues. |
+1 | whitespace | 0m 1s | The patch has no lines that end in whitespace. |
+1 | install | 1m 34s | mvn install still works. |
+1 | eclipse:eclipse | 0m 34s | The patch built with eclipse:eclipse. |
+1 | findbugs | 0m 49s | The patch does not introduce any new Findbugs (version 3.0.0) warnings. |
+1 | tools/hadoop tests | 1m 11s | Tests passed in hadoop-azure. |
37m 10s |
Subsystem | Report/Notes |
---|---|
Patch URL | http://issues.apache.org/jira/secure/attachment/12738267/HADOOP-12073.patch |
Optional Tests | javadoc javac unit findbugs checkstyle |
git revision | trunk / bd69ea4 |
javac | https://builds.apache.org/job/PreCommit-HADOOP-Build/6936/artifact/patchprocess/diffJavacWarnings.txt |
hadoop-azure test log | https://builds.apache.org/job/PreCommit-HADOOP-Build/6936/artifact/patchprocess/testrun_hadoop-azure.txt |
Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/6936/testReport/ |
Java | 1.7.0_55 |
uname | Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/6936/console |
This message was automatically generated.
+1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | pre-patch | 15m 13s | Pre-patch trunk compilation is healthy. |
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | tests included | 0m 0s | The patch appears to include 2 new or modified test files. |
+1 | javac | 7m 28s | There were no new javac warning messages. |
+1 | javadoc | 9m 35s | There were no new javadoc warning messages. |
+1 | release audit | 0m 23s | The applied patch does not increase the total number of release audit warnings. |
+1 | checkstyle | 0m 21s | There were no new checkstyle issues. |
+1 | whitespace | 0m 1s | The patch has no lines that end in whitespace. |
+1 | install | 1m 34s | mvn install still works. |
+1 | eclipse:eclipse | 0m 33s | The patch built with eclipse:eclipse. |
+1 | findbugs | 0m 46s | The patch does not introduce any new Findbugs (version 3.0.0) warnings. |
+1 | tools/hadoop tests | 1m 11s | Tests passed in hadoop-azure. |
37m 8s |
Subsystem | Report/Notes |
---|---|
Patch URL | http://issues.apache.org/jira/secure/attachment/12738276/HADOOP-12073.2.patch |
Optional Tests | javadoc javac unit findbugs checkstyle |
git revision | trunk / a6cb489 |
hadoop-azure test log | https://builds.apache.org/job/PreCommit-HADOOP-Build/6937/artifact/patchprocess/testrun_hadoop-azure.txt |
Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/6937/testReport/ |
Java | 1.7.0_55 |
uname | Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/6937/console |
This message was automatically generated.
Hi ivanmi. The patch looks good. I just have one question. Since the patch adds the closed flag to implement idempotence of PageBlobOutputStream#close, do we also need to mark that method synchronized for thread-safety?
Since the patch adds the closed flag to implement idempotence of PageBlobOutputStream#close, do we also need to mark that method synchronized for thread-safety?
Good catch, thanks Chris. Fixed in the updated patch.
+1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | pre-patch | 15m 31s | Pre-patch trunk compilation is healthy. |
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | tests included | 0m 0s | The patch appears to include 2 new or modified test files. |
+1 | javac | 7m 44s | There were no new javac warning messages. |
+1 | javadoc | 9m 58s | There were no new javadoc warning messages. |
+1 | release audit | 0m 23s | The applied patch does not increase the total number of release audit warnings. |
+1 | checkstyle | 0m 23s | There were no new checkstyle issues. |
+1 | whitespace | 0m 1s | The patch has no lines that end in whitespace. |
+1 | install | 1m 37s | mvn install still works. |
+1 | eclipse:eclipse | 0m 33s | The patch built with eclipse:eclipse. |
+1 | findbugs | 0m 47s | The patch does not introduce any new Findbugs (version 3.0.0) warnings. |
+1 | tools/hadoop tests | 1m 11s | Tests passed in hadoop-azure. |
38m 12s |
Subsystem | Report/Notes |
---|---|
Patch URL | http://issues.apache.org/jira/secure/attachment/12738479/HADOOP-12073.3.patch |
Optional Tests | javadoc javac unit findbugs checkstyle |
git revision | trunk / fc2ed4a |
hadoop-azure test log | https://builds.apache.org/job/PreCommit-HADOOP-Build/6940/artifact/patchprocess/testrun_hadoop-azure.txt |
Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/6940/testReport/ |
Java | 1.7.0_55 |
uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/6940/console |
This message was automatically generated.
+1 for the patch. I have committed this to trunk and branch-2. Ivan, thank you for the contribution.
FAILURE: Integrated in Hadoop-trunk-Commit #7996 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7996/)
HADOOP-12073. Azure FileSystem PageBlobInputStream does not return -1 on EOF. Contributed by Ivan Mitic. (cnauroth: rev c45784bc9031353b938f4756473937cca759b3dc)
- hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/PageBlobOutputStream.java
- hadoop-common-project/hadoop-common/CHANGES.txt
- hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/AzureNativeFileSystemStore.java
- hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/PageBlobInputStream.java
- hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestNativeAzureFileSystemContractPageBlobLive.java
- hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/NativeAzureFileSystemBaseTest.java
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #223 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/223/)
HADOOP-12073. Azure FileSystem PageBlobInputStream does not return -1 on EOF. Contributed by Ivan Mitic. (cnauroth: rev c45784bc9031353b938f4756473937cca759b3dc)
- hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/NativeAzureFileSystemBaseTest.java
- hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/AzureNativeFileSystemStore.java
- hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestNativeAzureFileSystemContractPageBlobLive.java
- hadoop-common-project/hadoop-common/CHANGES.txt
- hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/PageBlobOutputStream.java
- hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/PageBlobInputStream.java
FAILURE: Integrated in Hadoop-Yarn-trunk #953 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/953/)
HADOOP-12073. Azure FileSystem PageBlobInputStream does not return -1 on EOF. Contributed by Ivan Mitic. (cnauroth: rev c45784bc9031353b938f4756473937cca759b3dc)
- hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/PageBlobOutputStream.java
- hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/NativeAzureFileSystemBaseTest.java
- hadoop-common-project/hadoop-common/CHANGES.txt
- hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/PageBlobInputStream.java
- hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/AzureNativeFileSystemStore.java
- hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestNativeAzureFileSystemContractPageBlobLive.java
FAILURE: Integrated in Hadoop-Hdfs-trunk #2151 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2151/)
HADOOP-12073. Azure FileSystem PageBlobInputStream does not return -1 on EOF. Contributed by Ivan Mitic. (cnauroth: rev c45784bc9031353b938f4756473937cca759b3dc)
- hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/NativeAzureFileSystemBaseTest.java
- hadoop-common-project/hadoop-common/CHANGES.txt
- hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/PageBlobInputStream.java
- hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/PageBlobOutputStream.java
- hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/AzureNativeFileSystemStore.java
- hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestNativeAzureFileSystemContractPageBlobLive.java
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #212 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/212/)
HADOOP-12073. Azure FileSystem PageBlobInputStream does not return -1 on EOF. Contributed by Ivan Mitic. (cnauroth: rev c45784bc9031353b938f4756473937cca759b3dc)
- hadoop-common-project/hadoop-common/CHANGES.txt
- hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestNativeAzureFileSystemContractPageBlobLive.java
- hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/PageBlobOutputStream.java
- hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/PageBlobInputStream.java
- hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/AzureNativeFileSystemStore.java
- hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/NativeAzureFileSystemBaseTest.java
FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #221 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/221/)
HADOOP-12073. Azure FileSystem PageBlobInputStream does not return -1 on EOF. Contributed by Ivan Mitic. (cnauroth: rev c45784bc9031353b938f4756473937cca759b3dc)
- hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/AzureNativeFileSystemStore.java
- hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestNativeAzureFileSystemContractPageBlobLive.java
- hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/NativeAzureFileSystemBaseTest.java
- hadoop-common-project/hadoop-common/CHANGES.txt
- hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/PageBlobInputStream.java
- hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/PageBlobOutputStream.java
FAILURE: Integrated in Hadoop-Mapreduce-trunk #2169 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2169/)
HADOOP-12073. Azure FileSystem PageBlobInputStream does not return -1 on EOF. Contributed by Ivan Mitic. (cnauroth: rev c45784bc9031353b938f4756473937cca759b3dc)
- hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/PageBlobOutputStream.java
- hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/NativeAzureFileSystemBaseTest.java
- hadoop-common-project/hadoop-common/CHANGES.txt
- hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/AzureNativeFileSystemStore.java
- hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestNativeAzureFileSystemContractPageBlobLive.java
- hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/PageBlobInputStream.java
Is this serious enough that we should apply to 2.7.2? As it does sound it
I'm applying the 2.7.2-candidate label to get this on the radar for inclusion in the 2.7.2 release, as Steve suggested.
Attaching the patch.
I enabled additional FileSystem contract tests from Hadoop-common on page blobs to increase test coverage of these code paths. This caught another minor bug in PageBlobOutputStream where closing the stream multiple times does not succeed, so I fixed this as well.