[HADOOP-12753] S3A JUnit tests failing if using HTTP proxy

• Type: Sub-task
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.7.2
• Fix Version/s: 2.9.0, 3.0.0-alpha1
• Component/s: fs/s3

Environment

Test hardware:

• issue identified while running jUnit tests on MacBook Pro

PREREQUISITE:

• have a functional HTTP proxy running (ie. SQUID HTTP proxy @ http://127.0.0.1:3128)

• S3A test configuration (relevant options):
  <!-- NOTE: by not specifying endpoint, we'll use the default AWS S3 -->
  <property>
  <name>fs.s3a.endpoint</name>
  <value></value>
  </property>
  <!-- NOTE: please configure functional HTTP proxy with fs.s3a.proxy. {host,port}

  -->
  <property>
  <name>fs.s3a.proxy.host</name>
  <value>127.0.0.1</value>
  </property>
  <property>
  <name>fs.s3a.proxy.port</name>
  <value>3128</value>
  </property>
  <property>
  <name>fs.s3a.access.key</name>
  <value>_REAL_AWS_S3_ACCESS_KEY</value>
  </property>
  <property>
  <name>fs.s3a.secret.key</name>
  <value>_REAL_AWS_S3_SECRET_KEY</value>
  </property>

• Assignee: Zoran Rajic
• Reporter: Zoran Rajic
• Created: 01/Feb/16 19:08
• Updated: 30/Aug/16 01:19
• Resolved: 08/Apr/16 16:11

Description

Some companies have restricted firewalls and don't allow the "direct access" to Web-pages or Internet, so their employees have to use the SOCKS or the HTTP proxy.

This also means that the engineers who develop and test the Hadoop code and s3a:// -filesystem inside the company firewalls, need to access the "external" Amazon S3 service via the HTTP proxy.

Unfortunately, there are S3A jUnit test failures when run with the HTTP proxy (see Environment field above).

–

Details:

• steps to reproduce:

cd hadoop_git/hadoop-tools/hadoop-aws

mvn clean test -Dtest=TestS3AConfiguration
[...]
Running org.apache.hadoop.fs.s3a.TestS3AConfiguration
Tests run: 5, Failures: 1, Errors: 1, Skipped: 0, Time elapsed: 2.737 sec <<< FAILURE! - in org.apache.hadoop.fs.s3a.TestS3AConfiguration
TestAutomaticProxyPortSelection(org.apache.hadoop.fs.s3a.TestS3AConfiguration) Time elapsed: 0.737 sec <<< FAILURE!
java.lang.AssertionError: Expected a connection error for proxy server
at org.junit.Assert.fail(Assert.java:88)
at org.apache.hadoop.fs.s3a.TestS3AConfiguration.TestAutomaticProxyPortSelection(TestS3AConfiguration.java:130)

TestProxyPortWithoutHost(org.apache.hadoop.fs.s3a.TestS3AConfiguration) Time elapsed: 0.624 sec <<< ERROR!
com.amazonaws.AmazonClientException: Unable to execute HTTP request: Connection to http://127.0.0.1:1 refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:127)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:643)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:479)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:384)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3528)
at com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1031)
at com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:994)
at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:297)
at org.apache.hadoop.fs.s3a.S3ATestUtils.createTestFileSystem(S3ATestUtils.java:51)
at org.apache.hadoop.fs.s3a.TestS3AConfiguration.TestProxyPortWithoutHost(TestS3AConfiguration.java:111)

Results :

Failed tests:
TestS3AConfiguration.TestAutomaticProxyPortSelection:130 Expected a connection error for proxy server

Tests in error:
TestS3AConfiguration.TestProxyPortWithoutHost:111 » AmazonClient Unable to exe...

Tests run: 5, Failures: 1, Errors: 1, Skipped: 0