Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.7.2
    • Fix Version/s: 2.9.0, 3.0.0-alpha1
    • Component/s: fs/s3
    • Labels:
      None
    • Environment:

      Description

      Some companies have restricted firewalls and don't allow the "direct access" to Web-pages or Internet, so their employees have to use the SOCKS or the HTTP proxy.

      This also means that the engineers who develop and test the Hadoop code and s3a:// -filesystem inside the company firewalls, need to access the "external" Amazon S3 service via the HTTP proxy.

      Unfortunately, there are S3A jUnit test failures when run with the HTTP proxy (see Environment field above).

      Details:

      • steps to reproduce:

      cd hadoop_git/hadoop-tools/hadoop-aws

      mvn clean test -Dtest=TestS3AConfiguration
      [...]
      Running org.apache.hadoop.fs.s3a.TestS3AConfiguration
      Tests run: 5, Failures: 1, Errors: 1, Skipped: 0, Time elapsed: 2.737 sec <<< FAILURE! - in org.apache.hadoop.fs.s3a.TestS3AConfiguration
      TestAutomaticProxyPortSelection(org.apache.hadoop.fs.s3a.TestS3AConfiguration) Time elapsed: 0.737 sec <<< FAILURE!
      java.lang.AssertionError: Expected a connection error for proxy server
      at org.junit.Assert.fail(Assert.java:88)
      at org.apache.hadoop.fs.s3a.TestS3AConfiguration.TestAutomaticProxyPortSelection(TestS3AConfiguration.java:130)

      TestProxyPortWithoutHost(org.apache.hadoop.fs.s3a.TestS3AConfiguration) Time elapsed: 0.624 sec <<< ERROR!
      com.amazonaws.AmazonClientException: Unable to execute HTTP request: Connection to http://127.0.0.1:1 refused
      at java.net.PlainSocketImpl.socketConnect(Native Method)
      at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
      at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
      at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
      at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
      at java.net.Socket.connect(Socket.java:589)
      at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:127)
      at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
      at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)
      at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:643)
      at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:479)
      at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
      at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
      at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:384)
      at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232)
      at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3528)
      at com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1031)
      at com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:994)
      at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:297)
      at org.apache.hadoop.fs.s3a.S3ATestUtils.createTestFileSystem(S3ATestUtils.java:51)
      at org.apache.hadoop.fs.s3a.TestS3AConfiguration.TestProxyPortWithoutHost(TestS3AConfiguration.java:111)

      Results :

      Failed tests:
      TestS3AConfiguration.TestAutomaticProxyPortSelection:130 Expected a connection error for proxy server

      Tests in error:
      TestS3AConfiguration.TestProxyPortWithoutHost:111 » AmazonClient Unable to exe...

      Tests run: 5, Failures: 1, Errors: 1, Skipped: 0

        Attachments

        1. HADOOP-12753-s3a-jUnits.diff
          2 kB
          Zoran Rajic
        2. HDFS-12753.002.patch
          2 kB
          Zoran Rajic
        3. HDFS-12753-branch-2.003.patch
          2 kB
          Zoran Rajic

          Activity

            People

            • Assignee:
              zoran.rajic@hgst.com Zoran Rajic
              Reporter:
              zoran.rajic@hgst.com Zoran Rajic
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - 24h
                24h
                Remaining:
                Remaining Estimate - 24h
                24h
                Logged:
                Time Spent - Not Specified
                Not Specified