Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-11171

Enable using a proxy server to connect to S3a.

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.4.0
    • Fix Version/s: 2.7.0
    • Component/s: fs/s3
    • Labels:
    • Target Version/s:

      Description

      This exposes the AWS SDK config for a proxy (host and port) to s3a through config settings.

      1. HADOOP-11171.patch
        2 kB
        Thomas Demoor
      2. HADOOP-11171-2.patch
        4 kB
        Thomas Demoor
      3. HADOOP-11171-3.patch
        4 kB
        Thomas Demoor
      4. HADOOP-11171-4.patch
        4 kB
        Thomas Demoor
      5. HADOOP-11171-5.patch
        5 kB
        Thomas Demoor
      6. HADOOP-11171-6.patch
        7 kB
        Thomas Demoor
      7. HADOOP-11171-7.patch
        7 kB
        Thomas Demoor
      8. HADOOP-11171-8.patch
        9 kB
        Thomas Demoor
      9. HADOOP-11171-9.patch
        8 kB
        Thomas Demoor
      10. HADOOP-11171-10.patch
        8 kB
        Thomas Demoor

        Issue Links

          Activity

          Hide
          thodemoor Thomas Demoor added a comment -

          Straightforward patch. Sets proxyHost and proxyPort. Enables connecting to AWS S3 and other S3-compatible object stores through a proxy server.

          Show
          thodemoor Thomas Demoor added a comment - Straightforward patch. Sets proxyHost and proxyPort. Enables connecting to AWS S3 and other S3-compatible object stores through a proxy server.
          Hide
          stevel@apache.org Steve Loughran added a comment -
          1. can you use conf.getTrimmed() to get the host conf?
          2. there's no easy for test for this is there? what about setting the proxy to being 127.0.0.1/1 and expecting a connection failure? That would validate that the property was being picked up and applied
          Show
          stevel@apache.org Steve Loughran added a comment - can you use conf.getTrimmed() to get the host conf? there's no easy for test for this is there? what about setting the proxy to being 127.0.0.1/1 and expecting a connection failure? That would validate that the property was being picked up and applied
          Hide
          thodemoor Thomas Demoor added a comment -

          Patch 2 adds test and uses trimmed String as per Steve's request.

          Test ran successfully through

          mvn test -Dtest=TestS3AConfiguration

          This shouId suffice but I was not able to run the full test suite

           mvn test 

          as I have issues with the other test class which has following failed tests (that are unrelated to this patch, they also fail on trunk!):
          S3AFileSystemContractBaseTest.testRenameDirectory:193 null
          FileSystemContractBaseTest.testMkdirsWithUmask:186 expected:<461> but was:<511>
          FileSystemContractBaseTest.rename:489 expected:<false> but was:<true> -> this causes numerous tests to fail

          Any pointers to solving these are welcomed.

          Show
          thodemoor Thomas Demoor added a comment - Patch 2 adds test and uses trimmed String as per Steve's request. Test ran successfully through mvn test -Dtest=TestS3AConfiguration This shouId suffice but I was not able to run the full test suite mvn test as I have issues with the other test class which has following failed tests (that are unrelated to this patch, they also fail on trunk!): S3AFileSystemContractBaseTest.testRenameDirectory:193 null FileSystemContractBaseTest.testMkdirsWithUmask:186 expected:<461> but was:<511> FileSystemContractBaseTest.rename:489 expected:<false> but was:<true> -> this causes numerous tests to fail Any pointers to solving these are welcomed.
          Hide
          thodemoor Thomas Demoor added a comment -

          Patch 3 adds the apache license to the test class.

          Some extra info on the test: it only runs when the (not checked in) auth-keys.xml is present as advocated by Steve (cfr. other s3 connectors and openstack). The config setting "test.fs.s3a.name" is picked up from an xml.

          The test class is kept general as it is envisioned to hold tests for future patches involving config settings.

          Show
          thodemoor Thomas Demoor added a comment - Patch 3 adds the apache license to the test class. Some extra info on the test: it only runs when the (not checked in) auth-keys.xml is present as advocated by Steve (cfr. other s3 connectors and openstack). The config setting "test.fs.s3a.name" is picked up from an xml. The test class is kept general as it is envisioned to hold tests for future patches involving config settings.
          Hide
          thodemoor Thomas Demoor added a comment -

          Enables using a proxy server with the S3A Filesystem

          Show
          thodemoor Thomas Demoor added a comment - Enables using a proxy server with the S3A Filesystem
          Hide
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12678489/HADOOP-11171-3.patch
          against trunk revision d1828d9.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-tools/hadoop-aws.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/4995//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4995//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12678489/HADOOP-11171-3.patch against trunk revision d1828d9. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-tools/hadoop-aws. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/4995//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4995//console This message is automatically generated.
          Hide
          thodemoor Thomas Demoor added a comment -

          Patch 4:

          • Made test less brittle (checks: "host:port refused" thus allowing https )
          • Replaced hard coded Strings by the appropriate constants
          Show
          thodemoor Thomas Demoor added a comment - Patch 4: Made test less brittle (checks: "host:port refused" thus allowing https ) Replaced hard coded Strings by the appropriate constants
          Hide
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12679233/HADOOP-11171-4.patch
          against trunk revision 80d7d18.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-tools/hadoop-aws.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5041//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5041//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12679233/HADOOP-11171-4.patch against trunk revision 80d7d18. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-tools/hadoop-aws. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5041//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5041//console This message is automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          ...the current patch lets you set a port for a proxy without the hostname, and vice-versa. Should that be the case? Or should it be "if you set the host then you must have a port, and if you don't set the host the port value will be disregarded"

          Show
          stevel@apache.org Steve Loughran added a comment - ...the current patch lets you set a port for a proxy without the hostname, and vice-versa. Should that be the case? Or should it be "if you set the host then you must have a port, and if you don't set the host the port value will be disregarded"
          Hide
          thodemoor Thomas Demoor added a comment -

          The underlying aws lib does the following (HttpClientFactory.java):

           if (proxyHost != null && proxyPort > 0) {
          // proxy code goes here
          }
          

          When "half-configured" aws thus tries to connect directly, bypassing the proxy. Would you prefer raising an explicit error?

          Looking at that code reminded me to add the login credentials. So yet another patch on the way. I'll await your feedback before resubmitting.

          Show
          thodemoor Thomas Demoor added a comment - The underlying aws lib does the following (HttpClientFactory.java): if (proxyHost != null && proxyPort > 0) { // proxy code goes here } When "half-configured" aws thus tries to connect directly, bypassing the proxy. Would you prefer raising an explicit error? Looking at that code reminded me to add the login credentials. So yet another patch on the way. I'll await your feedback before resubmitting.
          Hide
          thodemoor Thomas Demoor added a comment -

          Patch 5: added username, password, domain, workstation. They are only picked up if proxy.host is set.

          Show
          thodemoor Thomas Demoor added a comment - Patch 5: added username, password, domain, workstation. They are only picked up if proxy.host is set.
          Hide
          thodemoor Thomas Demoor added a comment -

          If host is set without port the default of 80 / 443 is used (http / https) and a warning is logged. If port is set without host an error is thrown. Also refactored tests to use the (recently introduced) S3ATestUtils.

          Show
          thodemoor Thomas Demoor added a comment - If host is set without port the default of 80 / 443 is used (http / https) and a warning is logged. If port is set without host an error is thrown. Also refactored tests to use the (recently introduced) S3ATestUtils.
          Hide
          thodemoor Thomas Demoor added a comment -

          Added debug msg (containing all parameters) in case the proxy server is used.

          Show
          thodemoor Thomas Demoor added a comment - Added debug msg (containing all parameters) in case the proxy server is used.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          I can't test this; I have looked at the code and tests to see that (a) the new code doesn't break anything existing and (b) the tests do check that parameters are validated and that some bad settings are picked up.

          S3AFileSystem

          • p140: what if a field like password or domain is not supplied. Is setting them to null acceptable? I presume so, based on the testsl
          • p145, SLF4Jj handles the stringification of vararg elements, even null ones —so you can cut te String.valueOf.
          • TestS3AConfiguration. Can you rethrow invalid exceptions, such as
              if (!e.getMessage().contains(proxy + " refused")) {
                throw e;
              }
            
            

            This will help debug why a test is failing.

          Show
          stevel@apache.org Steve Loughran added a comment - I can't test this; I have looked at the code and tests to see that (a) the new code doesn't break anything existing and (b) the tests do check that parameters are validated and that some bad settings are picked up. S3AFileSystem p140: what if a field like password or domain is not supplied. Is setting them to null acceptable? I presume so, based on the testsl p145, SLF4Jj handles the stringification of vararg elements, even null ones —so you can cut te String.valueOf. TestS3AConfiguration . Can you rethrow invalid exceptions, such as if (!e.getMessage().contains(proxy + " refused" )) { throw e; } This will help debug why a test is failing.
          Hide
          thodemoor Thomas Demoor added a comment -

          Patch 8 addresses Steve's 3 remarks.

          p140: what if a field like password or domain is not supplied. Is setting them to null acceptable? I presume so, based on the testsl

          In the aws-sdk, they all default to null (com/amazonaws/ClientConfiguration.java) and are used as follows:

          com/amazonaws/http/HttpClientFactory.java
          if (proxyUsername != null && proxyPassword != null) {
              httpClient.getCredentialsProvider().setCredentials(
                  new AuthScope(proxyHost, proxyPort),
                  new NTCredentials(proxyUsername, proxyPassword,
                      proxyWorkstation,proxyDomain));
          }
          

          Further investigation of NTCredentials and NTUserPrincipal reveal that workstation and domain are allowed to be null.

          However, if username or password is set while the other is null the aws-sdk skips this code block. This patch adds code to throw an exception in this case (+ testcase).

          Show
          thodemoor Thomas Demoor added a comment - Patch 8 addresses Steve's 3 remarks. p140: what if a field like password or domain is not supplied. Is setting them to null acceptable? I presume so, based on the testsl In the aws-sdk, they all default to null (com/amazonaws/ClientConfiguration.java) and are used as follows: com/amazonaws/http/HttpClientFactory.java if (proxyUsername != null && proxyPassword != null ) { httpClient.getCredentialsProvider().setCredentials( new AuthScope(proxyHost, proxyPort), new NTCredentials(proxyUsername, proxyPassword, proxyWorkstation,proxyDomain)); } Further investigation of NTCredentials and NTUserPrincipal reveal that workstation and domain are allowed to be null. However, if username or password is set while the other is null the aws-sdk skips this code block. This patch adds code to throw an exception in this case (+ testcase).
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Thomas, this doesn't apply any more. Can you sync it up with trunk/branch-2 and I'll test & apply it

          Show
          stevel@apache.org Steve Loughran added a comment - Thomas, this doesn't apply any more. Can you sync it up with trunk/branch-2 and I'll test & apply it
          Hide
          thodemoor Thomas Demoor added a comment -

          Rebased on trunk and reran the tests.

          Didn't apply as this shares the testclass with my endpoint feature that you merged in earlier today.

          Show
          thodemoor Thomas Demoor added a comment - Rebased on trunk and reran the tests. Didn't apply as this shares the testclass with my endpoint feature that you merged in earlier today.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          well, the tests pass, but they take ~5 minutes

          Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 295.421 sec - in org.apache.hadoop.fs.s3a.TestS3AConfiguration
          
          Results :
          
          Tests run: 5, Failures: 0, Errors: 0, Skipped: 0
          
          [INFO] ------------------------------------------------------------------------
          [INFO] BUILD SUCCESS
          [INFO] ------------------------------------------------------------------------
          [INFO] Total time: 4:57.925s
          

          Cause? Connection retries:

             java.lang.Thread.State: TIMED_WAITING (sleeping)
          	at java.lang.Thread.sleep(Native Method)
          	at com.amazonaws.http.AmazonHttpClient.pauseBeforeNextRetry(AmazonHttpClient.java:864)
          	at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:353)
          	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232)
          	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3528)
          	at com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1031)
          	at com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:994)
          	at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:282)
          	at org.apache.hadoop.fs.s3a.S3ATestUtils.createTestFileSystem(S3ATestUtils.java:48)
          	at org.apache.hadoop.fs.s3a.TestS3AConfiguration.TestAutomaticProxyPortSelection(TestS3AConfiguration.java:126)
          	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          	at java.lang.reflect.Method.invoke(Method.java:606)
          	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
          	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
          	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
          	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
          	at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
          

          Is there a way to tune the failure policy of the AWS lib so it doesn't retry on these test runs?

          Show
          stevel@apache.org Steve Loughran added a comment - well, the tests pass, but they take ~5 minutes Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 295.421 sec - in org.apache.hadoop.fs.s3a.TestS3AConfiguration Results : Tests run: 5, Failures: 0, Errors: 0, Skipped: 0 [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 4:57.925s Cause? Connection retries: java.lang. Thread .State: TIMED_WAITING (sleeping) at java.lang. Thread .sleep(Native Method) at com.amazonaws.http.AmazonHttpClient.pauseBeforeNextRetry(AmazonHttpClient.java:864) at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:353) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3528) at com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1031) at com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:994) at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:282) at org.apache.hadoop.fs.s3a.S3ATestUtils.createTestFileSystem(S3ATestUtils.java:48) at org.apache.hadoop.fs.s3a.TestS3AConfiguration.TestAutomaticProxyPortSelection(TestS3AConfiguration.java:126) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) Is there a way to tune the failure policy of the AWS lib so it doesn't retry on these test runs?
          Hide
          thodemoor Thomas Demoor added a comment -

          By default failed connections are retried 10 times. Thi spatch reduces this to 2 for the proxy tests causing the runtime to decrease from 5 mins to 8 secs. The huge difference in runtime is caused by the exponential backoff strategy used by the underlying aws-sdk.

          Show
          thodemoor Thomas Demoor added a comment - By default failed connections are retried 10 times. Thi spatch reduces this to 2 for the proxy tests causing the runtime to decrease from 5 mins to 8 secs. The huge difference in runtime is caused by the exponential backoff strategy used by the underlying aws-sdk.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          now takes ~3s for me, dropping to ~ 2.7s on Java 8.

          +1, committing.

          Show
          stevel@apache.org Steve Loughran added a comment - now takes ~3s for me, dropping to ~ 2.7s on Java 8. +1, committing.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          committed. You are building up more things to document through

          Show
          stevel@apache.org Steve Loughran added a comment - committed. You are building up more things to document through
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #6883 (See https://builds.apache.org/job/Hadoop-trunk-Commit/6883/)
          HADOOP-11171 Enable using a proxy server to connect to S3a. (Thomas Demoor via stevel) (stevel: rev 2908fe4ec52f78d74e4207274a34d88d54cd468f)

          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
          • hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3AConfiguration.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #6883 (See https://builds.apache.org/job/Hadoop-trunk-Commit/6883/ ) HADOOP-11171 Enable using a proxy server to connect to S3a. (Thomas Demoor via stevel) (stevel: rev 2908fe4ec52f78d74e4207274a34d88d54cd468f) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3AConfiguration.java hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #77 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/77/)
          HADOOP-11171 Enable using a proxy server to connect to S3a. (Thomas Demoor via stevel) (stevel: rev 2908fe4ec52f78d74e4207274a34d88d54cd468f)

          • hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3AConfiguration.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #77 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/77/ ) HADOOP-11171 Enable using a proxy server to connect to S3a. (Thomas Demoor via stevel) (stevel: rev 2908fe4ec52f78d74e4207274a34d88d54cd468f) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3AConfiguration.java hadoop-common-project/hadoop-common/CHANGES.txt hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #811 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/811/)
          HADOOP-11171 Enable using a proxy server to connect to S3a. (Thomas Demoor via stevel) (stevel: rev 2908fe4ec52f78d74e4207274a34d88d54cd468f)

          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
          • hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3AConfiguration.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #811 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/811/ ) HADOOP-11171 Enable using a proxy server to connect to S3a. (Thomas Demoor via stevel) (stevel: rev 2908fe4ec52f78d74e4207274a34d88d54cd468f) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3AConfiguration.java hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2009 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2009/)
          HADOOP-11171 Enable using a proxy server to connect to S3a. (Thomas Demoor via stevel) (stevel: rev 2908fe4ec52f78d74e4207274a34d88d54cd468f)

          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3AConfiguration.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2009 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2009/ ) HADOOP-11171 Enable using a proxy server to connect to S3a. (Thomas Demoor via stevel) (stevel: rev 2908fe4ec52f78d74e4207274a34d88d54cd468f) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java hadoop-common-project/hadoop-common/CHANGES.txt hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3AConfiguration.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #74 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/74/)
          HADOOP-11171 Enable using a proxy server to connect to S3a. (Thomas Demoor via stevel) (stevel: rev 2908fe4ec52f78d74e4207274a34d88d54cd468f)

          • hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3AConfiguration.java
          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #74 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/74/ ) HADOOP-11171 Enable using a proxy server to connect to S3a. (Thomas Demoor via stevel) (stevel: rev 2908fe4ec52f78d74e4207274a34d88d54cd468f) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3AConfiguration.java hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #78 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/78/)
          HADOOP-11171 Enable using a proxy server to connect to S3a. (Thomas Demoor via stevel) (stevel: rev 2908fe4ec52f78d74e4207274a34d88d54cd468f)

          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
          • hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3AConfiguration.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #78 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/78/ ) HADOOP-11171 Enable using a proxy server to connect to S3a. (Thomas Demoor via stevel) (stevel: rev 2908fe4ec52f78d74e4207274a34d88d54cd468f) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java hadoop-common-project/hadoop-common/CHANGES.txt hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3AConfiguration.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2028 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2028/)
          HADOOP-11171 Enable using a proxy server to connect to S3a. (Thomas Demoor via stevel) (stevel: rev 2908fe4ec52f78d74e4207274a34d88d54cd468f)

          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
          • hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3AConfiguration.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2028 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2028/ ) HADOOP-11171 Enable using a proxy server to connect to S3a. (Thomas Demoor via stevel) (stevel: rev 2908fe4ec52f78d74e4207274a34d88d54cd468f) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3AConfiguration.java hadoop-common-project/hadoop-common/CHANGES.txt hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java

            People

            • Assignee:
              thodemoor Thomas Demoor
              Reporter:
              thodemoor Thomas Demoor
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development