Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-14598

Blacklist Http/HttpsFileSystem in FsUrlStreamHandlerFactory

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.9.0, 3.0.0-alpha4
    • Fix Version/s: 2.9.0, 3.0.0-beta1
    • Component/s: fs/azure, test
    • Labels:
      None

      Description

      my downstream-of-spark cloud integration tests (where I haven't been running the azure ones for a while) now have a few of the tests failing

       org.apache.hadoop.fs.azure.AzureException: com.microsoft.azure.storage.StorageException: org.apache.hadoop.fs.FsUrlConnection cannot be cast to java.net.HttpURLConnection
      

      No obvious cause, and it's only apparently happening in some of the (scalatest) tests.

      1. HADOOP-14598-002.patch
        6 kB
        Steve Loughran
      2. HADOOP-14598-003.patch
        7 kB
        Steve Loughran
      3. HADOOP-14598-004.patch
        7 kB
        Steve Loughran
      4. HADOOP-14598-005.patch
        7 kB
        Steve Loughran

        Issue Links

          Activity

          Hide
          stevel@apache.org Steve Loughran added a comment -

          FYI., on a "similar topic but not actually caused by the new work" is the discovery that you get an NPE if you try to use an hdfs:// URL For a JAR in the classloader: SPARK-21697

          That is: you cannot safely use any of the Hadoop filesystems as supported schemas for classloaders, due to a loop in commons-logging setup & use (at the very least, there may be other examples too). Not caused by the recent changes to FsUrlStreamHandlerFactory or the new schemas, just a new codepath that's not been looked at before

          Show
          stevel@apache.org Steve Loughran added a comment - FYI., on a "similar topic but not actually caused by the new work" is the discovery that you get an NPE if you try to use an hdfs:// URL For a JAR in the classloader: SPARK-21697 That is: you cannot safely use any of the Hadoop filesystems as supported schemas for classloaders, due to a loop in commons-logging setup & use (at the very least, there may be other examples too). Not caused by the recent changes to FsUrlStreamHandlerFactory or the new schemas, just a new codepath that's not been looked at before
          Hide
          stevel@apache.org Steve Loughran added a comment -

          thanks for committing it. It's a funny one. FWIW the class cast wasn't downstream. it was in an external library. We will have to assume this is commonplace across the java codebase.

          Show
          stevel@apache.org Steve Loughran added a comment - thanks for committing it. It's a funny one. FWIW the class cast wasn't downstream. it was in an external library. We will have to assume this is commonplace across the java codebase.
          Hide
          wheat9 Haohui Mai added a comment -

          Committed to trunk and branch-2. Thanks Steve Loughran for contribution.

          Show
          wheat9 Haohui Mai added a comment - Committed to trunk and branch-2. Thanks Steve Loughran for contribution.
          Hide
          wheat9 Haohui Mai added a comment -

          So essentially the downstream users assume that an URLConnection can be always casted to HttpConnection then. It's not ideal but yes, many people do that.

          The patch looks good to me. +1. I'll commit it shortly.

          Show
          wheat9 Haohui Mai added a comment - So essentially the downstream users assume that an URLConnection can be always casted to HttpConnection then. It's not ideal but yes, many people do that. The patch looks good to me. +1. I'll commit it shortly.
          Hide
          jzhuge John Zhuge added a comment -

          +1 LGTM. Wrap LOG.debug with if (LOG.isDebugEnabled()) in fast paths?

          Show
          jzhuge John Zhuge added a comment - +1 LGTM. Wrap LOG.debug with if (LOG.isDebugEnabled()) in fast paths?
          Hide
          stevel@apache.org Steve Loughran added a comment -

          HADOOP-14383 was designed to allow anyone to use an http or https URL as a source of data in anything which takes a filesystem for reading things. This is good, and changing the schema to anything other than http/https doesn't make sense.

          All thats problematic is that the bit of code which exports every Hadoop FS client as a URL via the JVM mustn't register the core JVM HTTP/HTTPS clients, as those work very well and other bits of code (here: Azure SDK), have assumptions/requirements about the class returned when you try to open such URLs.

          This patch stops the new schemas from being registered, sets things up for future schemas to go in too.

          Show
          stevel@apache.org Steve Loughran added a comment - HADOOP-14383 was designed to allow anyone to use an http or https URL as a source of data in anything which takes a filesystem for reading things. This is good, and changing the schema to anything other than http/https doesn't make sense. All thats problematic is that the bit of code which exports every Hadoop FS client as a URL via the JVM mustn't register the core JVM HTTP/HTTPS clients, as those work very well and other bits of code (here: Azure SDK), have assumptions/requirements about the class returned when you try to open such URLs. This patch stops the new schemas from being registered, sets things up for future schemas to go in too.
          Hide
          jzhuge John Zhuge added a comment -

          If HADOOP-14383 is reverted or modified to use different schemes, this JIRA no longer needs to be fixed, right?

          Show
          jzhuge John Zhuge added a comment - If HADOOP-14383 is reverted or modified to use different schemes, this JIRA no longer needs to be fixed, right?
          Hide
          esmanii Esfandiar Manii added a comment -

          +1, thanks!

          Show
          esmanii Esfandiar Manii added a comment - +1, thanks!
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 17s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
                trunk Compile Tests
          0 mvndep 1m 21s Maven dependency ordering for branch
          +1 mvninstall 13m 48s trunk passed
          +1 compile 13m 37s trunk passed
          +1 checkstyle 1m 57s trunk passed
          +1 mvnsite 2m 28s trunk passed
          -1 findbugs 1m 50s hadoop-hdfs-project/hadoop-hdfs in trunk has 9 extant Findbugs warnings.
          +1 javadoc 1m 40s trunk passed
                Patch Compile Tests
          0 mvndep 0m 16s Maven dependency ordering for patch
          +1 mvninstall 1m 31s the patch passed
          +1 compile 10m 22s the patch passed
          +1 javac 10m 22s the patch passed
          +1 checkstyle 2m 1s root: The patch generated 0 new + 8 unchanged - 1 fixed = 8 total (was 9)
          +1 mvnsite 2m 31s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 37s the patch passed
          +1 javadoc 1m 46s the patch passed
                Other Tests
          -1 unit 8m 9s hadoop-common in the patch failed.
          -1 unit 65m 4s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 37s The patch does not generate ASF License warnings.
          156m 27s



          Reason Tests
          Failed junit tests hadoop.security.TestKDiag
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy
            hadoop.hdfs.server.balancer.TestBalancer



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:14b5c93
          JIRA Issue HADOOP-14598
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12880380/HADOOP-14598-005.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux cd62903903a4 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 7fc324a
          Default Java 1.8.0_131
          findbugs v3.1.0-RC1
          findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/12951/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/12951/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/12951/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12951/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12951/console
          Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       trunk Compile Tests 0 mvndep 1m 21s Maven dependency ordering for branch +1 mvninstall 13m 48s trunk passed +1 compile 13m 37s trunk passed +1 checkstyle 1m 57s trunk passed +1 mvnsite 2m 28s trunk passed -1 findbugs 1m 50s hadoop-hdfs-project/hadoop-hdfs in trunk has 9 extant Findbugs warnings. +1 javadoc 1m 40s trunk passed       Patch Compile Tests 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvninstall 1m 31s the patch passed +1 compile 10m 22s the patch passed +1 javac 10m 22s the patch passed +1 checkstyle 2m 1s root: The patch generated 0 new + 8 unchanged - 1 fixed = 8 total (was 9) +1 mvnsite 2m 31s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 37s the patch passed +1 javadoc 1m 46s the patch passed       Other Tests -1 unit 8m 9s hadoop-common in the patch failed. -1 unit 65m 4s hadoop-hdfs in the patch failed. +1 asflicense 0m 37s The patch does not generate ASF License warnings. 156m 27s Reason Tests Failed junit tests hadoop.security.TestKDiag   hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy   hadoop.hdfs.server.balancer.TestBalancer Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HADOOP-14598 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12880380/HADOOP-14598-005.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux cd62903903a4 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 7fc324a Default Java 1.8.0_131 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/12951/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html unit https://builds.apache.org/job/PreCommit-HADOOP-Build/12951/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/12951/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12951/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12951/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Patch 005: address Esfandiar Manii's comments by

          • moving list of protocols into a static string with a comment explaining importance
          • test handling of an unknown protocol (nobody will ever add a Gopher connector, right?)

          Also, clean up the test suite slightly

          Show
          stevel@apache.org Steve Loughran added a comment - Patch 005: address Esfandiar Manii's comments by moving list of protocols into a static string with a comment explaining importance test handling of an unknown protocol (nobody will ever add a Gopher connector, right?) Also, clean up the test suite slightly
          Hide
          esmanii Esfandiar Manii added a comment -

          FsUrlStreamHandlerFactory L73-74, Could you please add a few lines of comment on why the protocols are added there so the reason wont be forgotten in the future.
          FsUrlStreamHandlerFactory L73-74, I would create a util/private method which gets/(exists in) the factory and call put on all the list of protocols.
          TestUrlStreamHandler.java, do we also need to include a test for invalid protocols?

          Show
          esmanii Esfandiar Manii added a comment - FsUrlStreamHandlerFactory L73-74, Could you please add a few lines of comment on why the protocols are added there so the reason wont be forgotten in the future. FsUrlStreamHandlerFactory L73-74, I would create a util/private method which gets/(exists in) the factory and call put on all the list of protocols. TestUrlStreamHandler.java, do we also need to include a test for invalid protocols?
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Can I get a review of this? I do consider it a blocker for the next releases. Thx

          Show
          stevel@apache.org Steve Loughran added a comment - Can I get a review of this? I do consider it a blocker for the next releases. Thx
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 13s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
                trunk Compile Tests
          0 mvndep 0m 14s Maven dependency ordering for branch
          +1 mvninstall 13m 15s trunk passed
          +1 compile 14m 14s trunk passed
          +1 checkstyle 1m 52s trunk passed
          +1 mvnsite 2m 26s trunk passed
          -1 findbugs 1m 48s hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings.
          +1 javadoc 1m 46s trunk passed
                Patch Compile Tests
          0 mvndep 0m 17s Maven dependency ordering for patch
          +1 mvninstall 1m 42s the patch passed
          +1 compile 10m 59s the patch passed
          +1 javac 10m 59s the patch passed
          +1 checkstyle 1m 59s the patch passed
          +1 mvnsite 2m 31s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 59s the patch passed
          +1 javadoc 1m 45s the patch passed
                Other Tests
          +1 unit 8m 45s hadoop-common in the patch passed.
          -1 unit 64m 49s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 42s The patch does not generate ASF License warnings.
          156m 20s



          Reason Tests
          Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:14b5c93
          JIRA Issue HADOOP-14598
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12877054/HADOOP-14598-004.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 176b6d59fd1d 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / cf0d084
          Default Java 1.8.0_131
          findbugs v3.1.0-RC1
          findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/12772/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/12772/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12772/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12772/console
          Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       trunk Compile Tests 0 mvndep 0m 14s Maven dependency ordering for branch +1 mvninstall 13m 15s trunk passed +1 compile 14m 14s trunk passed +1 checkstyle 1m 52s trunk passed +1 mvnsite 2m 26s trunk passed -1 findbugs 1m 48s hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings. +1 javadoc 1m 46s trunk passed       Patch Compile Tests 0 mvndep 0m 17s Maven dependency ordering for patch +1 mvninstall 1m 42s the patch passed +1 compile 10m 59s the patch passed +1 javac 10m 59s the patch passed +1 checkstyle 1m 59s the patch passed +1 mvnsite 2m 31s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 59s the patch passed +1 javadoc 1m 45s the patch passed       Other Tests +1 unit 8m 45s hadoop-common in the patch passed. -1 unit 64m 49s hadoop-hdfs in the patch failed. +1 asflicense 0m 42s The patch does not generate ASF License warnings. 156m 20s Reason Tests Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070 Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HADOOP-14598 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12877054/HADOOP-14598-004.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 176b6d59fd1d 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / cf0d084 Default Java 1.8.0_131 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/12772/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html unit https://builds.apache.org/job/PreCommit-HADOOP-Build/12772/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12772/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12772/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          patch 004; checkstyle warned of unused imports. fixed

          Show
          stevel@apache.org Steve Loughran added a comment - patch 004; checkstyle warned of unused imports. fixed
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Test failures are unrelated. First is ordering of tgt tickets; the HDFS onea are all about network problems

          Show
          stevel@apache.org Steve Loughran added a comment - Test failures are unrelated. First is ordering of tgt tickets; the HDFS onea are all about network problems
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 19s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 1m 32s Maven dependency ordering for branch
          +1 mvninstall 18m 56s trunk passed
          +1 compile 20m 28s trunk passed
          +1 checkstyle 2m 22s trunk passed
          +1 mvnsite 2m 55s trunk passed
          -1 findbugs 2m 26s hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings.
          +1 javadoc 2m 11s trunk passed
          0 mvndep 0m 21s Maven dependency ordering for patch
          +1 mvninstall 2m 18s the patch passed
          +1 compile 14m 54s the patch passed
          +1 javac 14m 54s the patch passed
          -0 checkstyle 2m 14s root: The patch generated 3 new + 9 unchanged - 0 fixed = 12 total (was 9)
          +1 mvnsite 2m 39s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 4m 23s the patch passed
          +1 javadoc 1m 54s the patch passed
          -1 unit 8m 41s hadoop-common in the patch failed.
          -1 unit 81m 59s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 38s The patch does not generate ASF License warnings.
          194m 38s



          Reason Tests
          Failed junit tests hadoop.security.TestRaceWhenRelogin
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140
            hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
            hadoop.hdfs.server.namenode.TestDecommissioningStatus
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
            hadoop.hdfs.web.TestWebHdfsTimeouts
            hadoop.hdfs.server.datanode.TestDirectoryScanner



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:14b5c93
          JIRA Issue HADOOP-14598
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12875249/HADOOP-14598-003.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux bf793a890bf7 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 3be2659
          Default Java 1.8.0_131
          findbugs v3.1.0-RC1
          findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/12687/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/12687/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/12687/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/12687/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12687/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12687/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 1m 32s Maven dependency ordering for branch +1 mvninstall 18m 56s trunk passed +1 compile 20m 28s trunk passed +1 checkstyle 2m 22s trunk passed +1 mvnsite 2m 55s trunk passed -1 findbugs 2m 26s hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings. +1 javadoc 2m 11s trunk passed 0 mvndep 0m 21s Maven dependency ordering for patch +1 mvninstall 2m 18s the patch passed +1 compile 14m 54s the patch passed +1 javac 14m 54s the patch passed -0 checkstyle 2m 14s root: The patch generated 3 new + 9 unchanged - 0 fixed = 12 total (was 9) +1 mvnsite 2m 39s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 4m 23s the patch passed +1 javadoc 1m 54s the patch passed -1 unit 8m 41s hadoop-common in the patch failed. -1 unit 81m 59s hadoop-hdfs in the patch failed. +1 asflicense 0m 38s The patch does not generate ASF License warnings. 194m 38s Reason Tests Failed junit tests hadoop.security.TestRaceWhenRelogin   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.hdfs.server.namenode.TestDecommissioningStatus   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting   hadoop.hdfs.web.TestWebHdfsTimeouts   hadoop.hdfs.server.datanode.TestDirectoryScanner Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HADOOP-14598 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12875249/HADOOP-14598-003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux bf793a890bf7 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 3be2659 Default Java 1.8.0_131 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/12687/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/12687/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/12687/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/12687/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12687/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12687/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Patch 003 preregisters http and https as unsupported, so{{FsUrlStreamHandlerFactory}} doesn't look for any filesystems with those schemas, instead returning null. This will cause the JVM to fall back to the default handler.

          Before that fix is in, the two new tests fail.

          Note also I patched the test class to set up the test TestUrlStreamHandler to set up the handler for the JVM in a before-class call. The existing code only worked provided the first test in the source, testDfsUrls was the first test executed; an ordering which hasn't been guaranteed since Java 7. It's possibly just been luck (or accidental sorting order of test names?) which has caused this to continue to work.

          java.lang.AssertionError: Handler for HTTP is the Hadoop one expected null, but was:<org.apache.hadoop.fs.FsUrlStreamHandler@27ae2fd0>
          
          	at org.junit.Assert.fail(Assert.java:88)
          	at org.junit.Assert.failNotNull(Assert.java:664)
          	at org.junit.Assert.assertNull(Assert.java:646)
          	at org.apache.hadoop.fs.TestUrlStreamHandler.testHttpDefaultHandler(TestUrlStreamHandler.java:170)
          	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          	at java.lang.reflect.Method.invoke(Method.java:498)
          	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
          	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
          	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
          	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
          	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
          	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
          	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
          	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
          	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
          	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
          	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
          	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
          	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
          	at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
          	at org.junit.runner.JUnitCore.run(JUnitCore.java:160)
          	at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
          	at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:51)
          	at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
          	at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
          
          
          java.lang.AssertionError: Handler for HTTPS is the Hadoop one expected null, but was:<org.apache.hadoop.fs.FsUrlStreamHandler@36c88a32>
          
          	at org.junit.Assert.fail(Assert.java:88)
          	at org.junit.Assert.failNotNull(Assert.java:664)
          	at org.junit.Assert.assertNull(Assert.java:646)
          	at org.apache.hadoop.fs.TestUrlStreamHandler.testHttpsDefaultHandler(TestUrlStreamHandler.java:176)
          	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          	at java.lang.reflect.Method.invoke(Method.java:498)
          	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
          	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
          	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
          	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
          	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
          	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
          	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
          	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
          	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
          	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
          	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
          	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
          	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
          	at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
          	at org.junit.runner.JUnitCore.run(JUnitCore.java:160)
          	at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
          	at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:51)
          	at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
          	at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
          

          As well as this unit test, I have done a full integration test downstream. All is well.

          Show
          stevel@apache.org Steve Loughran added a comment - Patch 003 preregisters http and https as unsupported, so{{FsUrlStreamHandlerFactory}} doesn't look for any filesystems with those schemas, instead returning null. This will cause the JVM to fall back to the default handler. Before that fix is in, the two new tests fail. Note also I patched the test class to set up the test TestUrlStreamHandler to set up the handler for the JVM in a before-class call. The existing code only worked provided the first test in the source, testDfsUrls was the first test executed; an ordering which hasn't been guaranteed since Java 7. It's possibly just been luck (or accidental sorting order of test names?) which has caused this to continue to work. java.lang.AssertionError: Handler for HTTP is the Hadoop one expected null , but was:<org.apache.hadoop.fs.FsUrlStreamHandler@27ae2fd0> at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotNull(Assert.java:664) at org.junit.Assert.assertNull(Assert.java:646) at org.apache.hadoop.fs.TestUrlStreamHandler.testHttpDefaultHandler(TestUrlStreamHandler.java:170) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.run(ParentRunner.java:309) at org.junit.runner.JUnitCore.run(JUnitCore.java:160) at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68) at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:51) at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242) at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70) java.lang.AssertionError: Handler for HTTPS is the Hadoop one expected null , but was:<org.apache.hadoop.fs.FsUrlStreamHandler@36c88a32> at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotNull(Assert.java:664) at org.junit.Assert.assertNull(Assert.java:646) at org.apache.hadoop.fs.TestUrlStreamHandler.testHttpsDefaultHandler(TestUrlStreamHandler.java:176) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.run(ParentRunner.java:309) at org.junit.runner.JUnitCore.run(JUnitCore.java:160) at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68) at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:51) at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242) at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70) As well as this unit test, I have done a full integration test downstream. All is well.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 2m 5s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 15s Maven dependency ordering for branch
          +1 mvninstall 13m 36s trunk passed
          +1 compile 13m 52s trunk passed
          +1 checkstyle 1m 56s trunk passed
          +1 mvnsite 1m 35s trunk passed
          +1 findbugs 2m 9s trunk passed
          +1 javadoc 1m 16s trunk passed
          0 mvndep 0m 17s Maven dependency ordering for patch
          +1 mvninstall 1m 8s the patch passed
          +1 compile 13m 26s the patch passed
          +1 javac 13m 26s the patch passed
          -0 checkstyle 2m 15s root: The patch generated 3 new + 1 unchanged - 0 fixed = 4 total (was 1)
          +1 mvnsite 1m 53s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 53s the patch passed
          +1 javadoc 1m 31s the patch passed
          -1 unit 9m 25s hadoop-common in the patch failed.
          +1 unit 1m 51s hadoop-azure in the patch passed.
          +1 asflicense 0m 42s The patch does not generate ASF License warnings.
          93m 50s



          Reason Tests
          Failed junit tests hadoop.ha.TestZKFailoverController



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:14b5c93
          JIRA Issue HADOOP-14598
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12874910/HADOOP-14598-002.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 1749d1604419 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / ee243e5
          Default Java 1.8.0_131
          findbugs v3.1.0-RC1
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/12649/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/12649/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12649/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-azure U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12649/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 2m 5s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 15s Maven dependency ordering for branch +1 mvninstall 13m 36s trunk passed +1 compile 13m 52s trunk passed +1 checkstyle 1m 56s trunk passed +1 mvnsite 1m 35s trunk passed +1 findbugs 2m 9s trunk passed +1 javadoc 1m 16s trunk passed 0 mvndep 0m 17s Maven dependency ordering for patch +1 mvninstall 1m 8s the patch passed +1 compile 13m 26s the patch passed +1 javac 13m 26s the patch passed -0 checkstyle 2m 15s root: The patch generated 3 new + 1 unchanged - 0 fixed = 4 total (was 1) +1 mvnsite 1m 53s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 53s the patch passed +1 javadoc 1m 31s the patch passed -1 unit 9m 25s hadoop-common in the patch failed. +1 unit 1m 51s hadoop-azure in the patch passed. +1 asflicense 0m 42s The patch does not generate ASF License warnings. 93m 50s Reason Tests Failed junit tests hadoop.ha.TestZKFailoverController Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HADOOP-14598 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12874910/HADOOP-14598-002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 1749d1604419 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / ee243e5 Default Java 1.8.0_131 findbugs v3.1.0-RC1 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/12649/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/12649/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12649/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-azure U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12649/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Patch 002. This doesn't fix it, but provided me with enough debug level logging to confirm that the http schemas were resulting in connections via HttpFileSystem, not real HttpURLConnection instances.

          Show
          stevel@apache.org Steve Loughran added a comment - Patch 002. This doesn't fix it, but provided me with enough debug level logging to confirm that the http schemas were resulting in connections via HttpFileSystem , not real HttpURLConnection instances.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          More details. Issue is that the HttpFileSystem connection is getting found when an http: URL comes in, so: disaster

          2017-06-28 16:54:59,217 [ScalaTest-main-running-AzureDataFrameSuite] INFO  netty.NettyBlockTransferService (Logging.scala:logInfo(54)) - Server created on 192.168.1.38:57795
          2017-06-28 16:55:01,284 [ScalaTest-main-running-AzureDataFrameSuite] DEBUG fs.FsUrlStreamHandlerFactory (FsUrlStreamHandlerFactory.java:createURLStreamHandler(77)) - Creating handler for protocol http
          2017-06-28 16:55:01,285 [ScalaTest-main-running-AzureDataFrameSuite] DEBUG fs.FsUrlStreamHandlerFactory (FsUrlStreamHandlerFactory.java:createURLStreamHandler(77)) - Creating handler for protocol file
          2017-06-28 16:55:01,285 [ScalaTest-main-running-AzureDataFrameSuite] DEBUG fs.FsUrlStreamHandlerFactory (FsUrlStreamHandlerFactory.java:createURLStreamHandler(83)) - Found implementation of file: class org.apache.hadoop.fs.LocalFileSystem
          2017-06-28 16:55:01,285 [ScalaTest-main-running-AzureDataFrameSuite] DEBUG fs.FsUrlStreamHandlerFactory (FsUrlStreamHandlerFactory.java:createURLStreamHandler(91)) - Using handler for protocol file
          2017-06-28 16:55:01,291 [ScalaTest-main-running-AzureDataFrameSuite] DEBUG fs.FsUrlStreamHandlerFactory (FsUrlStreamHandlerFactory.java:createURLStreamHandler(83)) - Found implementation of http: class org.apache.hadoop.fs.http.HttpFileSystem
          2017-06-28 16:55:01,291 [ScalaTest-main-running-AzureDataFrameSuite] DEBUG fs.FsUrlStreamHandlerFactory (FsUrlStreamHandlerFactory.java:createURLStreamHandler(91)) - Using handler for protocol http
          2017-06-28 16:55:01,331 [ScalaTest-main-running-AzureDataFrameSuite] INFO  azure.AzureDataFrameSuite (Logging.scala:logInfo(54)) - Cleaning wasb://contract@contender.blob.core.windows.net/cloud-integration/DELAY_LISTING_ME/AzureDataFrameSuite
          2017-06-28 16:55:01,332 [ScalaTest-main-running-AzureDataFrameSuite] INFO  azure.AzureDataFrameSuite (Logging.scala:logInfo(54)) - During cleanup of filesystem: org.apache.hadoop.fs.azure.AzureException: com.microsoft.azure.storage.StorageException: org.apache.hadoop.fs.FsUrlConnection cannot be cast to java.net.HttpURLConnection
          - DataFrames *** FAILED ***
            java.io.IOException: Failed to delete wasb://contract@contender.blob.core.windows.net/cloud-integration/DELAY_LISTING_ME/AzureDataFrameSuite/dataframes/generated on org.apache.hadoop.fs.azure.NativeAzureFileSystem@7b3cbe6e org.apache.hadoop.fs.azure.AzureException: com.microsoft.azure.storage.StorageException: org.apache.hadoop.fs.FsUrlConnection cannot be cast to java.net.HttpURLConnection
          
          Show
          stevel@apache.org Steve Loughran added a comment - More details. Issue is that the HttpFileSystem connection is getting found when an http: URL comes in, so: disaster 2017-06-28 16:54:59,217 [ScalaTest-main-running-AzureDataFrameSuite] INFO netty.NettyBlockTransferService (Logging.scala:logInfo(54)) - Server created on 192.168.1.38:57795 2017-06-28 16:55:01,284 [ScalaTest-main-running-AzureDataFrameSuite] DEBUG fs.FsUrlStreamHandlerFactory (FsUrlStreamHandlerFactory.java:createURLStreamHandler(77)) - Creating handler for protocol http 2017-06-28 16:55:01,285 [ScalaTest-main-running-AzureDataFrameSuite] DEBUG fs.FsUrlStreamHandlerFactory (FsUrlStreamHandlerFactory.java:createURLStreamHandler(77)) - Creating handler for protocol file 2017-06-28 16:55:01,285 [ScalaTest-main-running-AzureDataFrameSuite] DEBUG fs.FsUrlStreamHandlerFactory (FsUrlStreamHandlerFactory.java:createURLStreamHandler(83)) - Found implementation of file: class org.apache.hadoop.fs.LocalFileSystem 2017-06-28 16:55:01,285 [ScalaTest-main-running-AzureDataFrameSuite] DEBUG fs.FsUrlStreamHandlerFactory (FsUrlStreamHandlerFactory.java:createURLStreamHandler(91)) - Using handler for protocol file 2017-06-28 16:55:01,291 [ScalaTest-main-running-AzureDataFrameSuite] DEBUG fs.FsUrlStreamHandlerFactory (FsUrlStreamHandlerFactory.java:createURLStreamHandler(83)) - Found implementation of http: class org.apache.hadoop.fs.http.HttpFileSystem 2017-06-28 16:55:01,291 [ScalaTest-main-running-AzureDataFrameSuite] DEBUG fs.FsUrlStreamHandlerFactory (FsUrlStreamHandlerFactory.java:createURLStreamHandler(91)) - Using handler for protocol http 2017-06-28 16:55:01,331 [ScalaTest-main-running-AzureDataFrameSuite] INFO azure.AzureDataFrameSuite (Logging.scala:logInfo(54)) - Cleaning wasb: //contract@contender.blob.core.windows.net/cloud-integration/DELAY_LISTING_ME/AzureDataFrameSuite 2017-06-28 16:55:01,332 [ScalaTest-main-running-AzureDataFrameSuite] INFO azure.AzureDataFrameSuite (Logging.scala:logInfo(54)) - During cleanup of filesystem: org.apache.hadoop.fs.azure.AzureException: com.microsoft.azure.storage.StorageException: org.apache.hadoop.fs.FsUrlConnection cannot be cast to java.net.HttpURLConnection - DataFrames *** FAILED *** java.io.IOException: Failed to delete wasb: //contract@contender.blob.core.windows.net/cloud-integration/DELAY_LISTING_ME/AzureDataFrameSuite/dataframes/generated on org.apache.hadoop.fs.azure.NativeAzureFileSystem@7b3cbe6e org.apache.hadoop.fs.azure.AzureException: com.microsoft.azure.storage.StorageException: org.apache.hadoop.fs.FsUrlConnection cannot be cast to java.net.HttpURLConnection
          Hide
          stevel@apache.org Steve Loughran added a comment -

          trying to understand this some more. The Hadoop stream handler factory should be transparent for any URL Scheme which isn't from a supported FS. But of course, wasb is. Somehow a wasb URL is getting down into the Azure library when it tries to connect to azure storage...the fs factory returns a URL, which cannot be cast to an HTTP URL.

          this should be isolatable into a local test

          Show
          stevel@apache.org Steve Loughran added a comment - trying to understand this some more. The Hadoop stream handler factory should be transparent for any URL Scheme which isn't from a supported FS. But of course, wasb is. Somehow a wasb URL is getting down into the Azure library when it tries to connect to azure storage...the fs factory returns a URL, which cannot be cast to an HTTP URL. this should be isolatable into a local test
          Hide
          stevel@apache.org Steve Loughran added a comment -

          In Spark's org.apache.spark.sql.internal.SharedState

          URL.setURLStreamHandlerFactory(new FsUrlStreamHandlerFactory())
          

          This is kicking in when Hive/ Spark SQLContext is being inited, which is why it only surfaces in some tests

          Show
          stevel@apache.org Steve Loughran added a comment - In Spark's org.apache.spark.sql.internal.SharedState URL.setURLStreamHandlerFactory( new FsUrlStreamHandlerFactory()) This is kicking in when Hive/ Spark SQLContext is being inited, which is why it only surfaces in some tests
          Hide
          stevel@apache.org Steve Loughran added a comment - - edited

          not related to the latest SDK update; happens if I switch to the previous one.

          Looking @ the code, it comes down to this: the URI to use

          builder = new UriQueryBuilder();
          final URL resourceUrl = builder.addToURI(uri).toURL();
          retConnection = (HttpURLConnection) resourceUrl.openConnection();   // *HERE*
          

          The implication is that (a) hadoop has patched in a URL factory and (b) its returning the wrong thing, or (c) I'm handing in the wrong URL to the FS. Which, if I instrument my code better, doesn't appear to be the case

           FileGenerator *** FAILED ***
            java.io.IOException: Failed to delete wasb://contract@contender.blob.core.windows.net/cloud-integration/DELAY_LISTING_ME/AzureFileGeneratorSuite/filegenerator on org.apache.hadoop.fs.azure.NativeAzureFileSystem@6fe77a97 org.apache.hadoop.fs.azure.AzureException: com.microsoft.azure.storage.StorageException: org.apache.hadoop.fs.FsUrlConnection cannot be cast to java.net.HttpURLConnection
            at com.hortonworks.spark.cloud.ObjectStoreOperations$class.rm(ObjectStoreOperations.scala:415)
            at com.hortonworks.spark.cloud.operations.CloudFileGenerator.rm(CloudFileGenerator.scala:33)
            at com.hortonworks.spark.cloud.operations.CloudFileGenerator.action(CloudFileGenerator.scala:81)
            at com.hortonworks.spark.cloud.ObjectStoreExample$class.action(ObjectStoreExample.scala:70)
            at com.hortonworks.spark.cloud.operations.CloudFileGenerator.action(CloudFileGenerator.scala:33)
            at com.hortonworks.spark.cloud.common.FileGeneratorTests.generate(FileGeneratorTests.scala:80)
            at com.hortonworks.spark.cloud.common.FileGeneratorTests$$anonfun$2.apply$mcV$sp(FileGeneratorTests.scala:40)
            at com.hortonworks.spark.cloud.CloudSuiteTrait$$anonfun$ctest$1.apply$mcV$sp(CloudSuiteTrait.scala:69)
            at com.hortonworks.spark.cloud.CloudSuiteTrait$$anonfun$ctest$1.apply(CloudSuiteTrait.scala:67)
            at com.hortonworks.spark.cloud.CloudSuiteTrait$$anonfun$ctest$1.apply(CloudSuiteTrait.scala:67)
            ...
            Cause: org.apache.hadoop.fs.azure.AzureException: com.microsoft.azure.st
          
          Show
          stevel@apache.org Steve Loughran added a comment - - edited not related to the latest SDK update; happens if I switch to the previous one. Looking @ the code, it comes down to this: the URI to use builder = new UriQueryBuilder(); final URL resourceUrl = builder.addToURI(uri).toURL(); retConnection = (HttpURLConnection) resourceUrl.openConnection(); // *HERE* The implication is that (a) hadoop has patched in a URL factory and (b) its returning the wrong thing, or (c) I'm handing in the wrong URL to the FS. Which, if I instrument my code better, doesn't appear to be the case FileGenerator *** FAILED *** java.io.IOException: Failed to delete wasb: //contract@contender.blob.core.windows.net/cloud-integration/DELAY_LISTING_ME/AzureFileGeneratorSuite/filegenerator on org.apache.hadoop.fs.azure.NativeAzureFileSystem@6fe77a97 org.apache.hadoop.fs.azure.AzureException: com.microsoft.azure.storage.StorageException: org.apache.hadoop.fs.FsUrlConnection cannot be cast to java.net.HttpURLConnection at com.hortonworks.spark.cloud.ObjectStoreOperations$class.rm(ObjectStoreOperations.scala:415) at com.hortonworks.spark.cloud.operations.CloudFileGenerator.rm(CloudFileGenerator.scala:33) at com.hortonworks.spark.cloud.operations.CloudFileGenerator.action(CloudFileGenerator.scala:81) at com.hortonworks.spark.cloud.ObjectStoreExample$class.action(ObjectStoreExample.scala:70) at com.hortonworks.spark.cloud.operations.CloudFileGenerator.action(CloudFileGenerator.scala:33) at com.hortonworks.spark.cloud.common.FileGeneratorTests.generate(FileGeneratorTests.scala:80) at com.hortonworks.spark.cloud.common.FileGeneratorTests$$anonfun$2.apply$mcV$sp(FileGeneratorTests.scala:40) at com.hortonworks.spark.cloud.CloudSuiteTrait$$anonfun$ctest$1.apply$mcV$sp(CloudSuiteTrait.scala:69) at com.hortonworks.spark.cloud.CloudSuiteTrait$$anonfun$ctest$1.apply(CloudSuiteTrait.scala:67) at com.hortonworks.spark.cloud.CloudSuiteTrait$$anonfun$ctest$1.apply(CloudSuiteTrait.scala:67) ... Cause: org.apache.hadoop.fs.azure.AzureException: com.microsoft.azure.st
          Hide
          stevel@apache.org Steve Loughran added a comment -
          - DataFrames *** FAILED ***
            org.apache.hadoop.fs.azure.AzureException: com.microsoft.azure.storage.StorageException: org.apache.hadoop.fs.FsUrlConnection cannot be cast to java.net.HttpURLConnection
            at org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.retrieveMetadata(AzureNativeFileSystemStore.java:2023)
            at org.apache.hadoop.fs.azure.NativeAzureFileSystem.delete(NativeAzureFileSystem.java:1846)
            at org.apache.hadoop.fs.azure.NativeAzureFileSystem.delete(NativeAzureFileSystem.java:1810)
            at com.hortonworks.spark.cloud.ObjectStoreOperations$class.rm(ObjectStoreOperations.scala:409)
            at com.hortonworks.spark.cloud.operations.CloudDataFrames.rm(CloudDataFrames.scala:36)
            at com.hortonworks.spark.cloud.operations.CloudDataFrames.action(CloudDataFrames.scala:82)
            at com.hortonworks.spark.cloud.ObjectStoreExample$class.action(ObjectStoreExample.scala:70)
            at com.hortonworks.spark.cloud.operations.CloudDataFrames.action(CloudDataFrames.scala:36)
            at com.hortonworks.spark.cloud.common.DataFrameTests$$anonfun$3.apply$mcV$sp(DataFrameTests.scala:45)
            at com.hortonworks.spark.cloud.CloudSuiteTrait$$anonfun$ctest$1.apply$mcV$sp(CloudSuiteTrait.scala:69)
            ...
            Cause: com.microsoft.azure.storage.StorageException: org.apache.hadoop.fs.FsUrlConnection cannot be cast to java.net.HttpURLConnection
            at com.microsoft.azure.storage.core.ExecutionEngine.setupStorageRequest(ExecutionEngine.java:332)
            at com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:95)
            at com.microsoft.azure.storage.blob.CloudBlob.exists(CloudBlob.java:1850)
            at com.microsoft.azure.storage.blob.CloudBlob.exists(CloudBlob.java:1837)
            at org.apache.hadoop.fs.azure.StorageInterfaceImpl$CloudBlobWrapperImpl.exists(StorageInterfaceImpl.java:333)
            at org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.retrieveMetadata(AzureNativeFileSystemStore.java:1964)
            at org.apache.hadoop.fs.azure.NativeAzureFileSystem.delete(NativeAzureFileSystem.java:1846)
            at org.apache.hadoop.fs.azure.NativeAzureFileSystem.delete(NativeAzureFileSystem.java:1810)
            at com.hortonworks.spark.cloud.ObjectStoreOperations$class.rm(ObjectStoreOperations.scala:409)
            at com.hortonworks.spark.cloud.operations.CloudDataFrames.rm(CloudDataFrames.scala:36)
            ...
            Cause: java.lang.ClassCastException: org.apache.hadoop.fs.FsUrlConnection cannot be cast to java.net.HttpURLConnection
            at com.microsoft.azure.storage.core.BaseRequest.createURLConnection(BaseRequest.java:185)
            at com.microsoft.azure.storage.core.BaseRequest.getProperties(BaseRequest.java:299)
            at com.microsoft.azure.storage.blob.BlobRequest.getProperties(BlobRequest.java:791)
            at com.microsoft.azure.storage.blob.BlobRequest.getBlobProperties(BlobRequest.java:570)
            at com.microsoft.azure.storage.blob.CloudBlob$10.buildRequest(CloudBlob.java:1867)
            at com.microsoft.azure.storage.blob.CloudBlob$10.buildRequest(CloudBlob.java:1857)
            at com.microsoft.azure.storage.core.ExecutionEngine.setupStorageRequest(ExecutionEngine.java:304)
            at com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:95)
            at com.microsoft.azure.storage.blob.CloudBlob.exists(CloudBlob.java:1850)
            at com.microsoft.azure.storage.blob.CloudBlob.exists(CloudBlob.java:1837)
          
          Show
          stevel@apache.org Steve Loughran added a comment - - DataFrames *** FAILED *** org.apache.hadoop.fs.azure.AzureException: com.microsoft.azure.storage.StorageException: org.apache.hadoop.fs.FsUrlConnection cannot be cast to java.net.HttpURLConnection at org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.retrieveMetadata(AzureNativeFileSystemStore.java:2023) at org.apache.hadoop.fs.azure.NativeAzureFileSystem.delete(NativeAzureFileSystem.java:1846) at org.apache.hadoop.fs.azure.NativeAzureFileSystem.delete(NativeAzureFileSystem.java:1810) at com.hortonworks.spark.cloud.ObjectStoreOperations$class.rm(ObjectStoreOperations.scala:409) at com.hortonworks.spark.cloud.operations.CloudDataFrames.rm(CloudDataFrames.scala:36) at com.hortonworks.spark.cloud.operations.CloudDataFrames.action(CloudDataFrames.scala:82) at com.hortonworks.spark.cloud.ObjectStoreExample$class.action(ObjectStoreExample.scala:70) at com.hortonworks.spark.cloud.operations.CloudDataFrames.action(CloudDataFrames.scala:36) at com.hortonworks.spark.cloud.common.DataFrameTests$$anonfun$3.apply$mcV$sp(DataFrameTests.scala:45) at com.hortonworks.spark.cloud.CloudSuiteTrait$$anonfun$ctest$1.apply$mcV$sp(CloudSuiteTrait.scala:69) ... Cause: com.microsoft.azure.storage.StorageException: org.apache.hadoop.fs.FsUrlConnection cannot be cast to java.net.HttpURLConnection at com.microsoft.azure.storage.core.ExecutionEngine.setupStorageRequest(ExecutionEngine.java:332) at com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:95) at com.microsoft.azure.storage.blob.CloudBlob.exists(CloudBlob.java:1850) at com.microsoft.azure.storage.blob.CloudBlob.exists(CloudBlob.java:1837) at org.apache.hadoop.fs.azure.StorageInterfaceImpl$CloudBlobWrapperImpl.exists(StorageInterfaceImpl.java:333) at org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.retrieveMetadata(AzureNativeFileSystemStore.java:1964) at org.apache.hadoop.fs.azure.NativeAzureFileSystem.delete(NativeAzureFileSystem.java:1846) at org.apache.hadoop.fs.azure.NativeAzureFileSystem.delete(NativeAzureFileSystem.java:1810) at com.hortonworks.spark.cloud.ObjectStoreOperations$class.rm(ObjectStoreOperations.scala:409) at com.hortonworks.spark.cloud.operations.CloudDataFrames.rm(CloudDataFrames.scala:36) ... Cause: java.lang.ClassCastException: org.apache.hadoop.fs.FsUrlConnection cannot be cast to java.net.HttpURLConnection at com.microsoft.azure.storage.core.BaseRequest.createURLConnection(BaseRequest.java:185) at com.microsoft.azure.storage.core.BaseRequest.getProperties(BaseRequest.java:299) at com.microsoft.azure.storage.blob.BlobRequest.getProperties(BlobRequest.java:791) at com.microsoft.azure.storage.blob.BlobRequest.getBlobProperties(BlobRequest.java:570) at com.microsoft.azure.storage.blob.CloudBlob$10.buildRequest(CloudBlob.java:1867) at com.microsoft.azure.storage.blob.CloudBlob$10.buildRequest(CloudBlob.java:1857) at com.microsoft.azure.storage.core.ExecutionEngine.setupStorageRequest(ExecutionEngine.java:304) at com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:95) at com.microsoft.azure.storage.blob.CloudBlob.exists(CloudBlob.java:1850) at com.microsoft.azure.storage.blob.CloudBlob.exists(CloudBlob.java:1837)

            People

            • Assignee:
              stevel@apache.org Steve Loughran
              Reporter:
              stevel@apache.org Steve Loughran
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development