Details

    • Type: Improvement
    • Status: Open
    • Priority: Blocker
    • Resolution: Unresolved
    • Affects Version/s: 2.8.1
    • Fix Version/s: None
    • Component/s: fs/s3
    • Labels:
      None
    • Target Version/s:

      Description

      Improve S3a error handling and reporting

      this includes

      1. looking at error codes and translating to more specific exceptions
      2. better retry logic where present
      3. adding retry logic where not present
      4. more diagnostics in exceptions
      5. docs

      Overall goals

      • things that can be retried and will go away are retried for a bit
      • things that don't go away when retried failfast (302, no auth, unknown host, connection refused)
      • meaningful exceptions are built in translate exception
      • diagnostics are included, where possible
      • our troubleshooting docs are expanded with new failures we encounter

      AWS S3 error codes: http://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html

        Issue Links

          Activity

          Hide
          stevel@apache.org Steve Loughran added a comment -

          Note that HADOOP-13786 has done some java-8-friendly work to wrap aws client IO via closures and translation; this could be used here too

          Show
          stevel@apache.org Steve Loughran added a comment - Note that HADOOP-13786 has done some java-8-friendly work to wrap aws client IO via closures and translation; this could be used here too
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Also: we can extend the InconsistentClient to throw new exceptions on various operations. In particular, random raising of throttle exceptions can be used to guarantee that all code is resilient to transient request throttling

          Show
          stevel@apache.org Steve Loughran added a comment - Also: we can extend the InconsistentClient to throw new exceptions on various operations. In particular, random raising of throttle exceptions can be used to guarantee that all code is resilient to transient request throttling
          Hide
          stevel@apache.org Steve Loughran added a comment -

          DDB, STS, KMS all have their own meanings for 404; mapping to FNFE isn't appropriate for them.

          Proposed: use service information in the AWS exception fields to forward translation to a service-specific translator, if present. If this returns an explicit exception, throw it. Otherwise fall back to the main one.

          Testing easiest if we can just create the exceptions from stack traces in JIRAs

          Show
          stevel@apache.org Steve Loughran added a comment - DDB, STS, KMS all have their own meanings for 404; mapping to FNFE isn't appropriate for them. Proposed: use service information in the AWS exception fields to forward translation to a service-specific translator, if present. If this returns an explicit exception, throw it. Otherwise fall back to the main one. Testing easiest if we can just create the exceptions from stack traces in JIRAs
          Hide
          stevel@apache.org Steve Loughran added a comment -

          403 from a put request which didn't include everything it should. This should translate to an auth failure (as it is)

          testPutObjectDirect(org.apache.hadoop.fs.s3a.ITestS3AMiscOperations)  Time elapsed: 1.232 sec  <<< ERROR!
          com.amazonaws.services.s3.model.AmazonS3Exception: The request signature we calculated does not match the signature you provided. Check your key and signing method. (Service: Amazon S3; Status Code: 403; Error Code: SignatureDoesNotMatch; Request ID: C2B015BE1F6ED102)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1588)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1258)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1030)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:742)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:716)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)
          	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)
          	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4221)
          	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4168)
          	at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1718)
          	at org.apache.hadoop.fs.s3a.S3AFileSystem.putObjectDirect(S3AFileSystem.java:1253)
          	at org.apache.hadoop.fs.s3a.ITestS3AMiscOperations.lambda$testPutObjectDirect$0(ITestS3AMiscOperations.java:74)
          
          Show
          stevel@apache.org Steve Loughran added a comment - 403 from a put request which didn't include everything it should. This should translate to an auth failure (as it is) testPutObjectDirect(org.apache.hadoop.fs.s3a.ITestS3AMiscOperations) Time elapsed: 1.232 sec <<< ERROR! com.amazonaws.services.s3.model.AmazonS3Exception: The request signature we calculated does not match the signature you provided. Check your key and signing method. (Service: Amazon S3; Status Code: 403; Error Code: SignatureDoesNotMatch; Request ID: C2B015BE1F6ED102) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1588) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1258) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1030) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:742) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:716) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667) at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4221) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4168) at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1718) at org.apache.hadoop.fs.s3a.S3AFileSystem.putObjectDirect(S3AFileSystem.java:1253) at org.apache.hadoop.fs.s3a.ITestS3AMiscOperations.lambda$testPutObjectDirect$0(ITestS3AMiscOperations.java:74)
          Hide
          stevel@apache.org Steve Loughran added a comment - - edited

          connection timeout lost deep in things

          testCreateFile(org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextURI)  Time elapsed: 69.244 sec  <<< ERROR!
          java.io.InterruptedIOException: getFileStatus on s3a://hwdev-steve-frankfurt-new/Users/stevel/Projects/hadoop-trunk/hadoop-tools/hadoop-aws/target/test-dir/4/testContextURI/test#File: com.amazonaws.SdkClientException: Unable to execute HTTP request: Connect to hwdev-steve-frankfurt-new.s3.eu-central-1.amazonaws.com:443 [hwdev-steve-frankfurt-new.s3.eu-central-1.amazonaws.com/52.219.73.5] failed: connect timed out
          	at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:141)
          	at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:117)
          	at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:1883)
          	at org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:1850)
          	at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:1793)
          	at org.apache.hadoop.fs.DelegateToFileSystem.getFileStatus(DelegateToFileSystem.java:126)
          	at org.apache.hadoop.fs.FileContext$15.next(FileContext.java:1176)
          	at org.apache.hadoop.fs.FileContext$15.next(FileContext.java:1172)
          	at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90)
          	at org.apache.hadoop.fs.FileContext.getFileStatus(FileContext.java:1178)
          	at org.apache.hadoop.fs.FileContext$Util.exists(FileContext.java:1637)
          	at org.apache.hadoop.fs.FileContextTestHelper.exists(FileContextTestHelper.java:157)
          	at org.apache.hadoop.fs.FileContextURIBase.testCreateFile(FileContextURIBase.java:108)
          	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          	at java.lang.reflect.Method.invoke(Method.java:498)
          	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
          	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
          	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
          	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
          	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
          	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
          	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
          	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
          	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
          	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
          	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
          	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
          	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
          	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
          	at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
          	at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264)
          	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
          	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
          	at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
          	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
          	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
          Caused by: com.amazonaws.SdkClientException: Unable to execute HTTP request: Connect to hwdev-steve-frankfurt-new.s3.eu-central-1.amazonaws.com:443 [hwdev-steve-frankfurt-new.s3.eu-central-1.amazonaws.com/52.219.73.5] failed: connect timed out
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1069)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1035)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:742)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:716)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)
          	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)
          	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4221)
          	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4168)
          	at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1249)
          	at org.apache.hadoop.fs.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:1039)
          	at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:1865)
          	at org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:1850)
          	at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:1793)
          	at org.apache.hadoop.fs.DelegateToFileSystem.getFileStatus(DelegateToFileSystem.java:126)
          	at org.apache.hadoop.fs.FileContext$15.next(FileContext.java:1176)
          	at org.apache.hadoop.fs.FileContext$15.next(FileContext.java:1172)
          	at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90)
          	at org.apache.hadoop.fs.FileContext.getFileStatus(FileContext.java:1178)
          	at org.apache.hadoop.fs.FileContext$Util.exists(FileContext.java:1637)
          	at org.apache.hadoop.fs.FileContextTestHelper.exists(FileContextTestHelper.java:157)
          	at org.apache.hadoop.fs.FileContextURIBase.testCreateFile(FileContextURIBase.java:108)
          	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          	at java.lang.reflect.Method.invoke(Method.java:498)
          	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
          	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
          	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
          	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
          	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
          	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
          	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
          	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
          	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
          	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
          	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
          	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
          	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
          	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
          	at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
          	at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264)
          	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
          	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
          	at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
          	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
          	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
          Caused by: com.amazonaws.thirdparty.apache.http.conn.ConnectTimeoutException: Connect to hwdev-steve-frankfurt-new.s3.eu-central-1.amazonaws.com:443 [hwdev-steve-frankfurt-new.s3.eu-central-1.amazonaws.com/52.219.73.5] failed: connect timed out
          	at com.amazonaws.thirdparty.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:150)
          	at com.amazonaws.thirdparty.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353)
          	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          	at java.lang.reflect.Method.invoke(Method.java:498)
          	at com.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76)
          	at com.amazonaws.http.conn.$Proxy10.connect(Unknown Source)
          	at com.amazonaws.thirdparty.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380)
          	at com.amazonaws.thirdparty.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
          	at com.amazonaws.thirdparty.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
          	at com.amazonaws.thirdparty.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
          	at com.amazonaws.thirdparty.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
          	at com.amazonaws.thirdparty.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
          	at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1190)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1030)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:742)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:716)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)
          	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)
          	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4221)
          	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4168)
          	at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1249)
          	at org.apache.hadoop.fs.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:1039)
          	at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:1865)
          	at org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:1850)
          	at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:1793)
          	at org.apache.hadoop.fs.DelegateToFileSystem.getFileStatus(DelegateToFileSystem.java:126)
          	at org.apache.hadoop.fs.FileContext$15.next(FileContext.java:1176)
          	at org.apache.hadoop.fs.FileContext$15.next(FileContext.java:1172)
          	at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90)
          	at org.apache.hadoop.fs.FileContext.getFileStatus(FileContext.java:1178)
          	at org.apache.hadoop.fs.FileContext$Util.exists(FileContext.java:1637)
          	at org.apache.hadoop.fs.FileContextTestHelper.exists(FileContextTestHelper.java:157)
          	at org.apache.hadoop.fs.FileContextURIBase.testCreateFile(FileContextURIBase.java:108)
          	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          	at java.lang.reflect.Method.invoke(Method.java:498)
          	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
          	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
          	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
          	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
          	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
          	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
          	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
          	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
          	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
          	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
          	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
          	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
          	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
          	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
          	at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
          	at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264)
          	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
          	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
          	at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
          	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
          	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
          Caused by: java.net.SocketTimeoutException: connect timed out
          	at java.net.PlainSocketImpl.socketConnect(Native Method)
          	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
          	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
          	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
          	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
          	at java.net.Socket.connect(Socket.java:589)
          	at com.amazonaws.thirdparty.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:337)
          	at com.amazonaws.http.conn.ssl.SdkTLSSocketFactory.connectSocket(SdkTLSSocketFactory.java:132)
          	at com.amazonaws.thirdparty.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:141)
          	at com.amazonaws.thirdparty.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353)
          	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          	at java.lang.reflect.Method.invoke(Method.java:498)
          	at com.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76)
          	at com.amazonaws.http.conn.$Proxy10.connect(Unknown Source)
          	at com.amazonaws.thirdparty.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380)
          	at com.amazonaws.thirdparty.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
          	at com.amazonaws.thirdparty.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
          	at com.amazonaws.thirdparty.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
          	at com.amazonaws.thirdparty.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
          	at com.amazonaws.thirdparty.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
          	at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1190)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1030)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:742)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:716)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)
          	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)
          	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4221)
          	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4168)
          	at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1249)
          	at org.apache.hadoop.fs.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:1039)
          	at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:1865)
          	at org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:1850)
          	at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:1793)
          	at org.apache.hadoop.fs.DelegateToFileSystem.getFileStatus(DelegateToFileSystem.java:126)
          	at org.apache.hadoop.fs.FileContext$15.next(FileContext.java:1176)
          	at org.apache.hadoop.fs.FileContext$15.next(FileContext.java:1172)
          	at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90)
          	at org.apache.hadoop.fs.FileContext.getFileStatus(FileContext.java:1178)
          	at org.apache.hadoop.fs.FileContext$Util.exists(FileContext.java:1637)
          	at org.apache.hadoop.fs.FileContextTestHelper.exists(FileContextTestHelper.java:157)
          	at org.apache.hadoop.fs.FileContextURIBase.testCreateFile(FileContextURIBase.java:108)
          	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          	at java.lang.reflect.Method.invoke(Method.java:498)
          	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
          	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
          	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
          	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
          	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
          	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
          	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
          	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
          	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
          	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
          	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
          	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
          	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
          	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
          	at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
          	at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264)
          	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
          	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
          	at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
          	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
          	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
          
          Show
          stevel@apache.org Steve Loughran added a comment - - edited connection timeout lost deep in things testCreateFile(org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextURI) Time elapsed: 69.244 sec <<< ERROR! java.io.InterruptedIOException: getFileStatus on s3a: //hwdev-steve-frankfurt- new /Users/stevel/Projects/hadoop-trunk/hadoop-tools/hadoop-aws/target/test-dir/4/testContextURI/test#File: com.amazonaws.SdkClientException: Unable to execute HTTP request: Connect to hwdev-steve-frankfurt- new .s3.eu-central-1.amazonaws.com:443 [hwdev-steve-frankfurt- new .s3.eu-central-1.amazonaws.com/52.219.73.5] failed: connect timed out at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:141) at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:117) at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:1883) at org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:1850) at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:1793) at org.apache.hadoop.fs.DelegateToFileSystem.getFileStatus(DelegateToFileSystem.java:126) at org.apache.hadoop.fs.FileContext$15.next(FileContext.java:1176) at org.apache.hadoop.fs.FileContext$15.next(FileContext.java:1172) at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90) at org.apache.hadoop.fs.FileContext.getFileStatus(FileContext.java:1178) at org.apache.hadoop.fs.FileContext$Util.exists(FileContext.java:1637) at org.apache.hadoop.fs.FileContextTestHelper.exists(FileContextTestHelper.java:157) at org.apache.hadoop.fs.FileContextURIBase.testCreateFile(FileContextURIBase.java:108) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) at org.junit.runners.ParentRunner.run(ParentRunner.java:309) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124) at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103) Caused by: com.amazonaws.SdkClientException: Unable to execute HTTP request: Connect to hwdev-steve-frankfurt- new .s3.eu-central-1.amazonaws.com:443 [hwdev-steve-frankfurt- new .s3.eu-central-1.amazonaws.com/52.219.73.5] failed: connect timed out at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1069) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1035) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:742) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:716) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667) at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4221) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4168) at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1249) at org.apache.hadoop.fs.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:1039) at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:1865) at org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:1850) at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:1793) at org.apache.hadoop.fs.DelegateToFileSystem.getFileStatus(DelegateToFileSystem.java:126) at org.apache.hadoop.fs.FileContext$15.next(FileContext.java:1176) at org.apache.hadoop.fs.FileContext$15.next(FileContext.java:1172) at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90) at org.apache.hadoop.fs.FileContext.getFileStatus(FileContext.java:1178) at org.apache.hadoop.fs.FileContext$Util.exists(FileContext.java:1637) at org.apache.hadoop.fs.FileContextTestHelper.exists(FileContextTestHelper.java:157) at org.apache.hadoop.fs.FileContextURIBase.testCreateFile(FileContextURIBase.java:108) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) at org.junit.runners.ParentRunner.run(ParentRunner.java:309) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124) at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103) Caused by: com.amazonaws.thirdparty.apache.http.conn.ConnectTimeoutException: Connect to hwdev-steve-frankfurt- new .s3.eu-central-1.amazonaws.com:443 [hwdev-steve-frankfurt- new .s3.eu-central-1.amazonaws.com/52.219.73.5] failed: connect timed out at com.amazonaws.thirdparty.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:150) at com.amazonaws.thirdparty.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353) at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76) at com.amazonaws.http.conn.$Proxy10.connect(Unknown Source) at com.amazonaws.thirdparty.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380) at com.amazonaws.thirdparty.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at com.amazonaws.thirdparty.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184) at com.amazonaws.thirdparty.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) at com.amazonaws.thirdparty.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) at com.amazonaws.thirdparty.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55) at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1190) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1030) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:742) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:716) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667) at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4221) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4168) at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1249) at org.apache.hadoop.fs.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:1039) at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:1865) at org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:1850) at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:1793) at org.apache.hadoop.fs.DelegateToFileSystem.getFileStatus(DelegateToFileSystem.java:126) at org.apache.hadoop.fs.FileContext$15.next(FileContext.java:1176) at org.apache.hadoop.fs.FileContext$15.next(FileContext.java:1172) at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90) at org.apache.hadoop.fs.FileContext.getFileStatus(FileContext.java:1178) at org.apache.hadoop.fs.FileContext$Util.exists(FileContext.java:1637) at org.apache.hadoop.fs.FileContextTestHelper.exists(FileContextTestHelper.java:157) at org.apache.hadoop.fs.FileContextURIBase.testCreateFile(FileContextURIBase.java:108) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) at org.junit.runners.ParentRunner.run(ParentRunner.java:309) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124) at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103) Caused by: java.net.SocketTimeoutException: connect timed out at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:589) at com.amazonaws.thirdparty.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:337) at com.amazonaws.http.conn.ssl.SdkTLSSocketFactory.connectSocket(SdkTLSSocketFactory.java:132) at com.amazonaws.thirdparty.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:141) at com.amazonaws.thirdparty.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353) at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76) at com.amazonaws.http.conn.$Proxy10.connect(Unknown Source) at com.amazonaws.thirdparty.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380) at com.amazonaws.thirdparty.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at com.amazonaws.thirdparty.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184) at com.amazonaws.thirdparty.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) at com.amazonaws.thirdparty.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) at com.amazonaws.thirdparty.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55) at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1190) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1030) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:742) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:716) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667) at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4221) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4168) at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1249) at org.apache.hadoop.fs.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:1039) at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:1865) at org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:1850) at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:1793) at org.apache.hadoop.fs.DelegateToFileSystem.getFileStatus(DelegateToFileSystem.java:126) at org.apache.hadoop.fs.FileContext$15.next(FileContext.java:1176) at org.apache.hadoop.fs.FileContext$15.next(FileContext.java:1172) at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90) at org.apache.hadoop.fs.FileContext.getFileStatus(FileContext.java:1178) at org.apache.hadoop.fs.FileContext$Util.exists(FileContext.java:1637) at org.apache.hadoop.fs.FileContextTestHelper.exists(FileContextTestHelper.java:157) at org.apache.hadoop.fs.FileContextURIBase.testCreateFile(FileContextURIBase.java:108) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) at org.junit.runners.ParentRunner.run(ParentRunner.java:309) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124) at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
          Hide
          stevel@apache.org Steve Loughran added a comment -
          testCreateDirectory(org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextURI)  Time elapsed: 13.42 sec  <<< ERROR!
          java.lang.RuntimeException: java.io.InterruptedIOException: doesBucketExist on hwdev-steve-frankfurt-new: com.amazonaws.SdkClientException: Unable to execute HTTP request: Connect to hwdev-steve-frankfurt-new.s3.eu-central-1.amazonaws.com:443 [hwdev-steve-frankfurt-new.s3.eu-central-1.amazonaws.com/52.219.73.21] failed: connect timed out
          	at java.net.PlainSocketImpl.socketConnect(Native Method)
          	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
          	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
          	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
          	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
          	at java.net.Socket.connect(Socket.java:589)
          	at com.amazonaws.thirdparty.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:337)
          	at com.amazonaws.http.conn.ssl.SdkTLSSocketFactory.connectSocket(SdkTLSSocketFactory.java:132)
          	at com.amazonaws.thirdparty.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:141)
          	at com.amazonaws.thirdparty.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353)
          	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          	at java.lang.reflect.Method.invoke(Method.java:498)
          	at com.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76)
          	at com.amazonaws.http.conn.$Proxy10.connect(Unknown Source)
          	at com.amazonaws.thirdparty.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380)
          	at com.amazonaws.thirdparty.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
          	at com.amazonaws.thirdparty.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
          	at com.amazonaws.thirdparty.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
          	at com.amazonaws.thirdparty.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
          	at com.amazonaws.thirdparty.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
          	at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1190)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1030)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:742)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:716)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)
          	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)
          	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4221)
          	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4168)
          	at com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1306)
          	at com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:1263)
          	at org.apache.hadoop.fs.s3a.S3AFileSystem.verifyBucketExists(S3AFileSystem.java:309)
          	at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:268)
          	at org.apache.hadoop.fs.DelegateToFileSystem.<init>(DelegateToFileSystem.java:52)
          	at org.apache.hadoop.fs.s3a.S3A.<init>(S3A.java:40)
          	at sun.reflect.GeneratedConstructorAccessor39.newInstance(Unknown Source)
          	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
          	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
          	at org.apache.hadoop.fs.AbstractFileSystem.newInstance(AbstractFileSystem.java:135)
          	at org.apache.hadoop.fs.AbstractFileSystem.createFileSystem(AbstractFileSystem.java:173)
          	at org.apache.hadoop.fs.AbstractFileSystem.get(AbstractFileSystem.java:258)
          	at org.apache.hadoop.fs.FileContext$2.run(FileContext.java:331)
          	at org.apache.hadoop.fs.FileContext$2.run(FileContext.java:328)
          	at java.security.AccessController.doPrivileged(Native Method)
          	at javax.security.auth.Subject.doAs(Subject.java:422)
          	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1965)
          	at org.apache.hadoop.fs.FileContext.getAbstractFileSystem(FileContext.java:328)
          	at org.apache.hadoop.fs.FileContext.getFileContext(FileContext.java:454)
          	at org.apache.hadoop.fs.s3a.S3ATestUtils.createTestFileContext(S3ATestUtils.java:162)
          	at org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextURI.setUp(ITestS3AFileContextURI.java:43)
          
          
          Show
          stevel@apache.org Steve Loughran added a comment - testCreateDirectory(org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextURI) Time elapsed: 13.42 sec <<< ERROR! java.lang.RuntimeException: java.io.InterruptedIOException: doesBucketExist on hwdev-steve-frankfurt- new : com.amazonaws.SdkClientException: Unable to execute HTTP request: Connect to hwdev-steve-frankfurt- new .s3.eu-central-1.amazonaws.com:443 [hwdev-steve-frankfurt- new .s3.eu-central-1.amazonaws.com/52.219.73.21] failed: connect timed out at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:589) at com.amazonaws.thirdparty.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:337) at com.amazonaws.http.conn.ssl.SdkTLSSocketFactory.connectSocket(SdkTLSSocketFactory.java:132) at com.amazonaws.thirdparty.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:141) at com.amazonaws.thirdparty.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353) at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76) at com.amazonaws.http.conn.$Proxy10.connect(Unknown Source) at com.amazonaws.thirdparty.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380) at com.amazonaws.thirdparty.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at com.amazonaws.thirdparty.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184) at com.amazonaws.thirdparty.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) at com.amazonaws.thirdparty.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) at com.amazonaws.thirdparty.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55) at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1190) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1030) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:742) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:716) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667) at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4221) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4168) at com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1306) at com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:1263) at org.apache.hadoop.fs.s3a.S3AFileSystem.verifyBucketExists(S3AFileSystem.java:309) at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:268) at org.apache.hadoop.fs.DelegateToFileSystem.<init>(DelegateToFileSystem.java:52) at org.apache.hadoop.fs.s3a.S3A.<init>(S3A.java:40) at sun.reflect.GeneratedConstructorAccessor39.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.fs.AbstractFileSystem.newInstance(AbstractFileSystem.java:135) at org.apache.hadoop.fs.AbstractFileSystem.createFileSystem(AbstractFileSystem.java:173) at org.apache.hadoop.fs.AbstractFileSystem.get(AbstractFileSystem.java:258) at org.apache.hadoop.fs.FileContext$2.run(FileContext.java:331) at org.apache.hadoop.fs.FileContext$2.run(FileContext.java:328) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1965) at org.apache.hadoop.fs.FileContext.getAbstractFileSystem(FileContext.java:328) at org.apache.hadoop.fs.FileContext.getFileContext(FileContext.java:454) at org.apache.hadoop.fs.s3a.S3ATestUtils.createTestFileContext(S3ATestUtils.java:162) at org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextURI.setUp(ITestS3AFileContextURI.java:43)
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Implementing the core of this in HADOOP-13786, but not retrofitting it to operations inside S3AFS/ block output stream.

          1. new exceptions for specific failures which merit special retry policy (bad request => fail, throttle => retry with throttle policy, redirect => fail).
          2. new retry policy which sets that up, also implements failfast for unrecoverable network errors
          3. AwsLambda adds a retry() call which retries around the executed operation

          I'm using this as appropriate in commit/abort calls, all of which I'm considering repeatable other than delete calls

          Show
          stevel@apache.org Steve Loughran added a comment - Implementing the core of this in HADOOP-13786 , but not retrofitting it to operations inside S3AFS/ block output stream. new exceptions for specific failures which merit special retry policy (bad request => fail, throttle => retry with throttle policy, redirect => fail). new retry policy which sets that up, also implements failfast for unrecoverable network errors AwsLambda adds a retry() call which retries around the executed operation I'm using this as appropriate in commit/abort calls, all of which I'm considering repeatable other than delete calls
          Hide
          stevel@apache.org Steve Loughran added a comment -

          This actually means you tried a PUT to the key ""

            Cause: org.apache.hadoop.fs.s3a.AWSBadRequestException: put on : com.amazonaws.services.s3.model.AmazonS3Exception: The unspecified location constraint is incompatible for the region specific endpoint this request was sent to. (Service: Amazon S3; Status Code: 400; Error Code: IllegalLocationConstraintException; Request ID: EEBC5A08BCB3A645), S3 Extended Request ID: TWhryc4S5sFe1fwmhKCEuIHXnBZcoRvbGhf+u2UUKw3vcu2Zro25NGAd9Xo91WyPE7aoD1v8bj8=:IllegalLocationConstraintException: The unspecified location constraint is incompatible for the region specific endpoint this request was sent to. (Service: Amazon S3; Status Code: 400; Error Code: IllegalLocationConstraintException; Request ID: EEBC5A08BCB3A645)
            at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:178)
            at org.apache.hadoop.fs.s3a.S3ALambda.execute(S3ALambda.java:64)
            at org.apache.hadoop.fs.s3a.WriteOperationHelper.uploadObject(WriteOperationHelper.java:451)
            at org.apache.hadoop.fs.s3a.commit.magic.MagicCommitTracker.aboutToComplete(MagicCommitTracker.java:128)
            at org.apache.hadoop.fs.s3a.S3ABlockOutputStream.close(S3ABlockOutputStream.java:373)
            at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
            at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101)
            at org.apache.hadoop.hive.ql.io.orc.WriterImpl.close(WriterImpl.java:2429)
            at org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat$OrcRecordWriter.close(OrcOutputFormat.java:106)
            at org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat$OrcRecordWriter.close(OrcOutputFormat.java:91)
            ...
            Cause: com.amazonaws.services.s3.model.AmazonS3Exception: The unspecified location constraint is incompatible for the region specific endpoint this request was sent to. (Service: Amazon S3; Status Code: 400; Error Code: IllegalLocationConstraintException; Request ID: EEBC5A08BCB3A645)
            at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1588)
            at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1258)
            at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1030)
            at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:742)
            at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:716)
            at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)
            at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)
            at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)
            at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)
            at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4221)
            ...
          
          Show
          stevel@apache.org Steve Loughran added a comment - This actually means you tried a PUT to the key "" Cause: org.apache.hadoop.fs.s3a.AWSBadRequestException: put on : com.amazonaws.services.s3.model.AmazonS3Exception: The unspecified location constraint is incompatible for the region specific endpoint this request was sent to. (Service: Amazon S3; Status Code: 400; Error Code: IllegalLocationConstraintException; Request ID: EEBC5A08BCB3A645), S3 Extended Request ID: TWhryc4S5sFe1fwmhKCEuIHXnBZcoRvbGhf+u2UUKw3vcu2Zro25NGAd9Xo91WyPE7aoD1v8bj8=:IllegalLocationConstraintException: The unspecified location constraint is incompatible for the region specific endpoint this request was sent to. (Service: Amazon S3; Status Code: 400; Error Code: IllegalLocationConstraintException; Request ID: EEBC5A08BCB3A645) at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:178) at org.apache.hadoop.fs.s3a.S3ALambda.execute(S3ALambda.java:64) at org.apache.hadoop.fs.s3a.WriteOperationHelper.uploadObject(WriteOperationHelper.java:451) at org.apache.hadoop.fs.s3a.commit.magic.MagicCommitTracker.aboutToComplete(MagicCommitTracker.java:128) at org.apache.hadoop.fs.s3a.S3ABlockOutputStream.close(S3ABlockOutputStream.java:373) at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72) at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101) at org.apache.hadoop.hive.ql.io.orc.WriterImpl.close(WriterImpl.java:2429) at org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat$OrcRecordWriter.close(OrcOutputFormat.java:106) at org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat$OrcRecordWriter.close(OrcOutputFormat.java:91) ... Cause: com.amazonaws.services.s3.model.AmazonS3Exception: The unspecified location constraint is incompatible for the region specific endpoint this request was sent to. (Service: Amazon S3; Status Code: 400; Error Code: IllegalLocationConstraintException; Request ID: EEBC5A08BCB3A645) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1588) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1258) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1030) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:742) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:716) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667) at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4221) ...
          Hide
          stevel@apache.org Steve Loughran added a comment -

          you can get a 443 sometimes; no response. Retryable on idempotent calls only

          2017-02-07 10:00:44,200 INFO [main] org.apache.hadoop.mapred.Task: Task attempt_1486454881801_0009_m_000024_0 is allowed to commit now
          2017-02-07 10:01:07,950 INFO [s3a-transfer-shared-pool1-t7] com.amazonaws.http.AmazonHttpClient: Unable to execute HTTP request: hwdev-rajesh-new2.s3.amazonaws.com:443 failed to respond
          org.apache.http.NoHttpResponseException: hwdev-rajesh-new2.s3.amazonaws.com:443 failed to respond
                  at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:143)
                  at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
                  at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
                  at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
                  at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:259)
                  at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:209)
                  at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
                  at com.amazonaws.http.protocol.SdkHttpRequestExecutor.doReceiveResponse(SdkHttpRequestExecutor.java:66)
                  at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
                  at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:686)
                  at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:488)
                  at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:884)
                  at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
                  at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
                  at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:728)
                  at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:489)
                  at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:310)
                  at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3785)
                  at com.amazonaws.services.s3.AmazonS3Client.copyPart(AmazonS3Client.java:1731)
                  at com.amazonaws.services.s3.transfer.internal.CopyPartCallable.call(CopyPartCallable.java:41)
                  at com.amazonaws.services.s3.transfer.internal.CopyPartCallable.call(CopyPartCallable.java:28)
                  at org.apache.hadoop.fs.s3a.SemaphoredDelegatingExecutor$CallableWithPermitRelease.call(SemaphoredDelegatingExecutor.java:222)
                  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
                  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
                  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
                  at java.lang.Thread.run(Thread.java:745)
          
          Show
          stevel@apache.org Steve Loughran added a comment - you can get a 443 sometimes; no response. Retryable on idempotent calls only 2017-02-07 10:00:44,200 INFO [main] org.apache.hadoop.mapred.Task: Task attempt_1486454881801_0009_m_000024_0 is allowed to commit now 2017-02-07 10:01:07,950 INFO [s3a-transfer-shared-pool1-t7] com.amazonaws.http.AmazonHttpClient: Unable to execute HTTP request: hwdev-rajesh-new2.s3.amazonaws.com:443 failed to respond org.apache.http.NoHttpResponseException: hwdev-rajesh-new2.s3.amazonaws.com:443 failed to respond at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:143) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261) at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283) at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:259) at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:209) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272) at com.amazonaws.http.protocol.SdkHttpRequestExecutor.doReceiveResponse(SdkHttpRequestExecutor.java:66) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124) at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:686) at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:488) at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:884) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55) at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:728) at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:489) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:310) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3785) at com.amazonaws.services.s3.AmazonS3Client.copyPart(AmazonS3Client.java:1731) at com.amazonaws.services.s3.transfer.internal.CopyPartCallable.call(CopyPartCallable.java:41) at com.amazonaws.services.s3.transfer.internal.CopyPartCallable.call(CopyPartCallable.java:28) at org.apache.hadoop.fs.s3a.SemaphoredDelegatingExecutor$CallableWithPermitRelease.call(SemaphoredDelegatingExecutor.java:222) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang. Thread .run( Thread .java:745)
          Hide
          stevel@apache.org Steve Loughran added a comment -

          stack on an init with a bucket which doesn't exist, maps to FNFE

          bin/hadoop s3guard bucket-info s3a://stevel45r56666
          java.io.FileNotFoundException: Bucket stevel45r56666 does not exist
          	at org.apache.hadoop.fs.s3a.S3AFileSystem.verifyBucketExists(S3AFileSystem.java:361)
          	at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:293)
          	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3288)
          	at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:123)
          	at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3337)
          	at org.apache.hadoop.fs.FileSystem$Cache.getUnique(FileSystem.java:3311)
          	at org.apache.hadoop.fs.FileSystem.newInstance(FileSystem.java:529)
          	at org.apache.hadoop.fs.s3a.s3guard.S3GuardTool$BucketInfo.run(S3GuardTool.java:997)
          	at org.apache.hadoop.fs.s3a.s3guard.S3GuardTool.run(S3GuardTool.java:309)
          	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
          	at org.apache.hadoop.fs.s3a.s3guard.S3GuardTool.run(S3GuardTool.java:1218)
          	at org.apache.hadoop.fs.s3a.s3guard.S3GuardTool.main(S3GuardTool.java:1227)
          2017-09-26 13:58:40,024 INFO util.ExitUtil: Exiting with status -1: java.io.FileNotFoundException: Bucket stevel45r56666 does not exist
          
          Show
          stevel@apache.org Steve Loughran added a comment - stack on an init with a bucket which doesn't exist, maps to FNFE bin/hadoop s3guard bucket-info s3a: //stevel45r56666 java.io.FileNotFoundException: Bucket stevel45r56666 does not exist at org.apache.hadoop.fs.s3a.S3AFileSystem.verifyBucketExists(S3AFileSystem.java:361) at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:293) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3288) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:123) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3337) at org.apache.hadoop.fs.FileSystem$Cache.getUnique(FileSystem.java:3311) at org.apache.hadoop.fs.FileSystem.newInstance(FileSystem.java:529) at org.apache.hadoop.fs.s3a.s3guard.S3GuardTool$BucketInfo.run(S3GuardTool.java:997) at org.apache.hadoop.fs.s3a.s3guard.S3GuardTool.run(S3GuardTool.java:309) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.hadoop.fs.s3a.s3guard.S3GuardTool.run(S3GuardTool.java:1218) at org.apache.hadoop.fs.s3a.s3guard.S3GuardTool.main(S3GuardTool.java:1227) 2017-09-26 13:58:40,024 INFO util.ExitUtil: Exiting with status -1: java.io.FileNotFoundException: Bucket stevel45r56666 does not exist
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Bucket exists, but doesn't grant access (FWIW, it's a bucket of mine since ~2010, just on a personal a/c}: Result AccessDeniedException

          bin/hadoop s3guard bucket-info s3a://stevel
          2017-09-26 14:00:33,944 INFO Configuration.deprecation: fs.s3a.server-side-encryption-key is deprecated. Instead, use fs.s3a.server-side-encryption.key
          Filesystem s3a://stevel
          2017-09-26 14:00:34,132 INFO s3a.S3ALambda: getBucketLocation() on stevel: java.nio.file.AccessDeniedException: stevel: getBucketLocation() on stevel: com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 3AF0ECF22A60DD1D; S3 Extended Request ID: 1ljfM4kRdqobBBDKMVbRqeguhfn4vTH3uPjhNiU0VR5+GdP8ArUB89Qp3XY5gjUajeBwo2YUJLk=), S3 Extended Request ID: 1ljfM4kRdqobBBDKMVbRqeguhfn4vTH3uPjhNiU0VR5+GdP8ArUB89Qp3XY5gjUajeBwo2YUJLk=:AccessDenied
          2017-09-26 14:00:34,133 WARN s3a.S3ALambda: getBucketLocation() on stevel failing after 1 attempts: java.nio.file.AccessDeniedException: stevel: getBucketLocation() on stevel: com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 3AF0ECF22A60DD1D; S3 Extended Request ID: 1ljfM4kRdqobBBDKMVbRqeguhfn4vTH3uPjhNiU0VR5+GdP8ArUB89Qp3XY5gjUajeBwo2YUJLk=), S3 Extended Request ID: 1ljfM4kRdqobBBDKMVbRqeguhfn4vTH3uPjhNiU0VR5+GdP8ArUB89Qp3XY5gjUajeBwo2YUJLk=:AccessDenied
          java.nio.file.AccessDeniedException: stevel: getBucketLocation() on stevel: com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 3AF0ECF22A60DD1D; S3 Extended Request ID: 1ljfM4kRdqobBBDKMVbRqeguhfn4vTH3uPjhNiU0VR5+GdP8ArUB89Qp3XY5gjUajeBwo2YUJLk=), S3 Extended Request ID: 1ljfM4kRdqobBBDKMVbRqeguhfn4vTH3uPjhNiU0VR5+GdP8ArUB89Qp3XY5gjUajeBwo2YUJLk=:AccessDenied
          	at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:205)
          	at org.apache.hadoop.fs.s3a.S3ALambda.once(S3ALambda.java:122)
          	at org.apache.hadoop.fs.s3a.S3ALambda.lambda$retry$2(S3ALambda.java:233)
          	at org.apache.hadoop.fs.s3a.S3ALambda.retryUntranslated(S3ALambda.java:288)
          	at org.apache.hadoop.fs.s3a.S3ALambda.retry(S3ALambda.java:228)
          	at org.apache.hadoop.fs.s3a.S3ALambda.retry(S3ALambda.java:203)
          	at org.apache.hadoop.fs.s3a.S3AFileSystem.getBucketLocation(S3AFileSystem.java:513)
          	at org.apache.hadoop.fs.s3a.S3AFileSystem.getBucketLocation(S3AFileSystem.java:501)
          	at org.apache.hadoop.fs.s3a.s3guard.S3GuardTool$BucketInfo.run(S3GuardTool.java:1004)
          	at org.apache.hadoop.fs.s3a.s3guard.S3GuardTool.run(S3GuardTool.java:309)
          	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
          	at org.apache.hadoop.fs.s3a.s3guard.S3GuardTool.run(S3GuardTool.java:1218)
          	at org.apache.hadoop.fs.s3a.s3guard.S3GuardTool.main(S3GuardTool.java:1227)
          Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 3AF0ECF22A60DD1D; S3 Extended Request ID: 1ljfM4kRdqobBBDKMVbRqeguhfn4vTH3uPjhNiU0VR5+GdP8ArUB89Qp3XY5gjUajeBwo2YUJLk=), S3 Extended Request ID: 1ljfM4kRdqobBBDKMVbRqeguhfn4vTH3uPjhNiU0VR5+GdP8ArUB89Qp3XY5gjUajeBwo2YUJLk=
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1638)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1303)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1055)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)
          	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)
          	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4229)
          	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4176)
          	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4170)
          	at com.amazonaws.services.s3.AmazonS3Client.getBucketLocation(AmazonS3Client.java:938)
          	at com.amazonaws.services.s3.AmazonS3Client.getBucketLocation(AmazonS3Client.java:944)
          	at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$getBucketLocation$3(S3AFileSystem.java:516)
          	at org.apache.hadoop.fs.s3a.S3ALambda.once(S3ALambda.java:120)
          	... 11 more
          2017-09-26 14:00:34,136 INFO util.ExitUtil: Exiting with status -1: java.nio.file.AccessDeniedException: stevel: getBucketLocation() on stevel: com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 3AF0ECF22A60DD1D; S3 Extended Request ID: 1ljfM4kRdqobBBDKMVbRqeguhfn4vTH3uPjhNiU0VR5+GdP8ArUB89Qp3XY5gjUajeBwo2YUJLk=), S3 Extended Request ID: 1ljfM4kRdqobBBDKMVbRqeguhfn4vTH3uPjhNiU0VR5+GdP8ArUB89Qp3XY5gjUajeBwo2YUJLk=:AccessDenied
          
          Show
          stevel@apache.org Steve Loughran added a comment - Bucket exists, but doesn't grant access (FWIW, it's a bucket of mine since ~2010, just on a personal a/c}: Result AccessDeniedException bin/hadoop s3guard bucket-info s3a: //stevel 2017-09-26 14:00:33,944 INFO Configuration.deprecation: fs.s3a.server-side-encryption-key is deprecated. Instead, use fs.s3a.server-side-encryption.key Filesystem s3a: //stevel 2017-09-26 14:00:34,132 INFO s3a.S3ALambda: getBucketLocation() on stevel: java.nio.file.AccessDeniedException: stevel: getBucketLocation() on stevel: com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 3AF0ECF22A60DD1D; S3 Extended Request ID: 1ljfM4kRdqobBBDKMVbRqeguhfn4vTH3uPjhNiU0VR5+GdP8ArUB89Qp3XY5gjUajeBwo2YUJLk=), S3 Extended Request ID: 1ljfM4kRdqobBBDKMVbRqeguhfn4vTH3uPjhNiU0VR5+GdP8ArUB89Qp3XY5gjUajeBwo2YUJLk=:AccessDenied 2017-09-26 14:00:34,133 WARN s3a.S3ALambda: getBucketLocation() on stevel failing after 1 attempts: java.nio.file.AccessDeniedException: stevel: getBucketLocation() on stevel: com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 3AF0ECF22A60DD1D; S3 Extended Request ID: 1ljfM4kRdqobBBDKMVbRqeguhfn4vTH3uPjhNiU0VR5+GdP8ArUB89Qp3XY5gjUajeBwo2YUJLk=), S3 Extended Request ID: 1ljfM4kRdqobBBDKMVbRqeguhfn4vTH3uPjhNiU0VR5+GdP8ArUB89Qp3XY5gjUajeBwo2YUJLk=:AccessDenied java.nio.file.AccessDeniedException: stevel: getBucketLocation() on stevel: com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 3AF0ECF22A60DD1D; S3 Extended Request ID: 1ljfM4kRdqobBBDKMVbRqeguhfn4vTH3uPjhNiU0VR5+GdP8ArUB89Qp3XY5gjUajeBwo2YUJLk=), S3 Extended Request ID: 1ljfM4kRdqobBBDKMVbRqeguhfn4vTH3uPjhNiU0VR5+GdP8ArUB89Qp3XY5gjUajeBwo2YUJLk=:AccessDenied at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:205) at org.apache.hadoop.fs.s3a.S3ALambda.once(S3ALambda.java:122) at org.apache.hadoop.fs.s3a.S3ALambda.lambda$retry$2(S3ALambda.java:233) at org.apache.hadoop.fs.s3a.S3ALambda.retryUntranslated(S3ALambda.java:288) at org.apache.hadoop.fs.s3a.S3ALambda.retry(S3ALambda.java:228) at org.apache.hadoop.fs.s3a.S3ALambda.retry(S3ALambda.java:203) at org.apache.hadoop.fs.s3a.S3AFileSystem.getBucketLocation(S3AFileSystem.java:513) at org.apache.hadoop.fs.s3a.S3AFileSystem.getBucketLocation(S3AFileSystem.java:501) at org.apache.hadoop.fs.s3a.s3guard.S3GuardTool$BucketInfo.run(S3GuardTool.java:1004) at org.apache.hadoop.fs.s3a.s3guard.S3GuardTool.run(S3GuardTool.java:309) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.hadoop.fs.s3a.s3guard.S3GuardTool.run(S3GuardTool.java:1218) at org.apache.hadoop.fs.s3a.s3guard.S3GuardTool.main(S3GuardTool.java:1227) Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 3AF0ECF22A60DD1D; S3 Extended Request ID: 1ljfM4kRdqobBBDKMVbRqeguhfn4vTH3uPjhNiU0VR5+GdP8ArUB89Qp3XY5gjUajeBwo2YUJLk=), S3 Extended Request ID: 1ljfM4kRdqobBBDKMVbRqeguhfn4vTH3uPjhNiU0VR5+GdP8ArUB89Qp3XY5gjUajeBwo2YUJLk= at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1638) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1303) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1055) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667) at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4229) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4176) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4170) at com.amazonaws.services.s3.AmazonS3Client.getBucketLocation(AmazonS3Client.java:938) at com.amazonaws.services.s3.AmazonS3Client.getBucketLocation(AmazonS3Client.java:944) at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$getBucketLocation$3(S3AFileSystem.java:516) at org.apache.hadoop.fs.s3a.S3ALambda.once(S3ALambda.java:120) ... 11 more 2017-09-26 14:00:34,136 INFO util.ExitUtil: Exiting with status -1: java.nio.file.AccessDeniedException: stevel: getBucketLocation() on stevel: com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 3AF0ECF22A60DD1D; S3 Extended Request ID: 1ljfM4kRdqobBBDKMVbRqeguhfn4vTH3uPjhNiU0VR5+GdP8ArUB89Qp3XY5gjUajeBwo2YUJLk=), S3 Extended Request ID: 1ljfM4kRdqobBBDKMVbRqeguhfn4vTH3uPjhNiU0VR5+GdP8ArUB89Qp3XY5gjUajeBwo2YUJLk=:AccessDenied
          Hide
          stevel@apache.org Steve Loughran added a comment -

          And a randomly picked key sequence "fsfd" maps to a bucket which appears to exist but has access disabled, raises Access Denied. That's a slightly different text message than before

           bin/hadoop s3guard bucket-info s3a://fdsd
          2017-09-26 13:57:56,458 INFO s3a.S3ALambda: doesBucketExist on fdsd: java.nio.file.AccessDeniedException: fdsd: doesBucketExist on fdsd: com.amazonaws.services.s3.model.AmazonS3Exception: All access to this object has been disabled (Service: Amazon S3; Status Code: 403; Error Code: AllAccessDisabled; Request ID: E6229D7F8134E64F; S3 Extended Request ID: 6SzVz2t4qa8J2Wxo/oc8yBuB13Mgrn9uMKnxVY0hsBd2kU/YdHzW1IaujpJdDXRDCQRX3f1RYn0=), S3 Extended Request ID: 6SzVz2t4qa8J2Wxo/oc8yBuB13Mgrn9uMKnxVY0hsBd2kU/YdHzW1IaujpJdDXRDCQRX3f1RYn0=:AllAccessDisabled
          2017-09-26 13:57:56,459 WARN s3a.S3ALambda: doesBucketExist on fdsd failing after 1 attempts: java.nio.file.AccessDeniedException: fdsd: doesBucketExist on fdsd: com.amazonaws.services.s3.model.AmazonS3Exception: All access to this object has been disabled (Service: Amazon S3; Status Code: 403; Error Code: AllAccessDisabled; Request ID: E6229D7F8134E64F; S3 Extended Request ID: 6SzVz2t4qa8J2Wxo/oc8yBuB13Mgrn9uMKnxVY0hsBd2kU/YdHzW1IaujpJdDXRDCQRX3f1RYn0=), S3 Extended Request ID: 6SzVz2t4qa8J2Wxo/oc8yBuB13Mgrn9uMKnxVY0hsBd2kU/YdHzW1IaujpJdDXRDCQRX3f1RYn0=:AllAccessDisabled
          java.nio.file.AccessDeniedException: fdsd: doesBucketExist on fdsd: com.amazonaws.services.s3.model.AmazonS3Exception: All access to this object has been disabled (Service: Amazon S3; Status Code: 403; Error Code: AllAccessDisabled; Request ID: E6229D7F8134E64F; S3 Extended Request ID: 6SzVz2t4qa8J2Wxo/oc8yBuB13Mgrn9uMKnxVY0hsBd2kU/YdHzW1IaujpJdDXRDCQRX3f1RYn0=), S3 Extended Request ID: 6SzVz2t4qa8J2Wxo/oc8yBuB13Mgrn9uMKnxVY0hsBd2kU/YdHzW1IaujpJdDXRDCQRX3f1RYn0=:AllAccessDisabled
          	at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:205)
          	at org.apache.hadoop.fs.s3a.S3ALambda.once(S3ALambda.java:122)
          	at org.apache.hadoop.fs.s3a.S3ALambda.lambda$retry$2(S3ALambda.java:233)
          	at org.apache.hadoop.fs.s3a.S3ALambda.retryUntranslated(S3ALambda.java:288)
          	at org.apache.hadoop.fs.s3a.S3ALambda.retry(S3ALambda.java:228)
          	at org.apache.hadoop.fs.s3a.S3ALambda.retry(S3ALambda.java:203)
          	at org.apache.hadoop.fs.s3a.S3AFileSystem.verifyBucketExists(S3AFileSystem.java:357)
          	at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:293)
          	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3288)
          	at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:123)
          	at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3337)
          	at org.apache.hadoop.fs.FileSystem$Cache.getUnique(FileSystem.java:3311)
          	at org.apache.hadoop.fs.FileSystem.newInstance(FileSystem.java:529)
          	at org.apache.hadoop.fs.s3a.s3guard.S3GuardTool$BucketInfo.run(S3GuardTool.java:997)
          	at org.apache.hadoop.fs.s3a.s3guard.S3GuardTool.run(S3GuardTool.java:309)
          	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
          	at org.apache.hadoop.fs.s3a.s3guard.S3GuardTool.run(S3GuardTool.java:1218)
          	at org.apache.hadoop.fs.s3a.s3guard.S3GuardTool.main(S3GuardTool.java:1227)
          Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: All access to this object has been disabled (Service: Amazon S3; Status Code: 403; Error Code: AllAccessDisabled; Request ID: E6229D7F8134E64F; S3 Extended Request ID: 6SzVz2t4qa8J2Wxo/oc8yBuB13Mgrn9uMKnxVY0hsBd2kU/YdHzW1IaujpJdDXRDCQRX3f1RYn0=), S3 Extended Request ID: 6SzVz2t4qa8J2Wxo/oc8yBuB13Mgrn9uMKnxVY0hsBd2kU/YdHzW1IaujpJdDXRDCQRX3f1RYn0=
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1638)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1303)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1055)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)
          	at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)
          	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)
          	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4229)
          	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4176)
          	at com.amazonaws.services.s3.AmazonS3Client.getAcl(AmazonS3Client.java:3381)
          	at com.amazonaws.services.s3.AmazonS3Client.getBucketAcl(AmazonS3Client.java:1160)
          	at com.amazonaws.services.s3.AmazonS3Client.getBucketAcl(AmazonS3Client.java:1150)
          	at com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:1266)
          	at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$verifyBucketExists$1(S3AFileSystem.java:360)
          	at org.apache.hadoop.fs.s3a.S3ALambda.once(S3ALambda.java:120)
          	... 16 more
          
          Show
          stevel@apache.org Steve Loughran added a comment - And a randomly picked key sequence "fsfd" maps to a bucket which appears to exist but has access disabled, raises Access Denied. That's a slightly different text message than before bin/hadoop s3guard bucket-info s3a: //fdsd 2017-09-26 13:57:56,458 INFO s3a.S3ALambda: doesBucketExist on fdsd: java.nio.file.AccessDeniedException: fdsd: doesBucketExist on fdsd: com.amazonaws.services.s3.model.AmazonS3Exception: All access to this object has been disabled (Service: Amazon S3; Status Code: 403; Error Code: AllAccessDisabled; Request ID: E6229D7F8134E64F; S3 Extended Request ID: 6SzVz2t4qa8J2Wxo/oc8yBuB13Mgrn9uMKnxVY0hsBd2kU/YdHzW1IaujpJdDXRDCQRX3f1RYn0=), S3 Extended Request ID: 6SzVz2t4qa8J2Wxo/oc8yBuB13Mgrn9uMKnxVY0hsBd2kU/YdHzW1IaujpJdDXRDCQRX3f1RYn0=:AllAccessDisabled 2017-09-26 13:57:56,459 WARN s3a.S3ALambda: doesBucketExist on fdsd failing after 1 attempts: java.nio.file.AccessDeniedException: fdsd: doesBucketExist on fdsd: com.amazonaws.services.s3.model.AmazonS3Exception: All access to this object has been disabled (Service: Amazon S3; Status Code: 403; Error Code: AllAccessDisabled; Request ID: E6229D7F8134E64F; S3 Extended Request ID: 6SzVz2t4qa8J2Wxo/oc8yBuB13Mgrn9uMKnxVY0hsBd2kU/YdHzW1IaujpJdDXRDCQRX3f1RYn0=), S3 Extended Request ID: 6SzVz2t4qa8J2Wxo/oc8yBuB13Mgrn9uMKnxVY0hsBd2kU/YdHzW1IaujpJdDXRDCQRX3f1RYn0=:AllAccessDisabled java.nio.file.AccessDeniedException: fdsd: doesBucketExist on fdsd: com.amazonaws.services.s3.model.AmazonS3Exception: All access to this object has been disabled (Service: Amazon S3; Status Code: 403; Error Code: AllAccessDisabled; Request ID: E6229D7F8134E64F; S3 Extended Request ID: 6SzVz2t4qa8J2Wxo/oc8yBuB13Mgrn9uMKnxVY0hsBd2kU/YdHzW1IaujpJdDXRDCQRX3f1RYn0=), S3 Extended Request ID: 6SzVz2t4qa8J2Wxo/oc8yBuB13Mgrn9uMKnxVY0hsBd2kU/YdHzW1IaujpJdDXRDCQRX3f1RYn0=:AllAccessDisabled at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:205) at org.apache.hadoop.fs.s3a.S3ALambda.once(S3ALambda.java:122) at org.apache.hadoop.fs.s3a.S3ALambda.lambda$retry$2(S3ALambda.java:233) at org.apache.hadoop.fs.s3a.S3ALambda.retryUntranslated(S3ALambda.java:288) at org.apache.hadoop.fs.s3a.S3ALambda.retry(S3ALambda.java:228) at org.apache.hadoop.fs.s3a.S3ALambda.retry(S3ALambda.java:203) at org.apache.hadoop.fs.s3a.S3AFileSystem.verifyBucketExists(S3AFileSystem.java:357) at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:293) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3288) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:123) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3337) at org.apache.hadoop.fs.FileSystem$Cache.getUnique(FileSystem.java:3311) at org.apache.hadoop.fs.FileSystem.newInstance(FileSystem.java:529) at org.apache.hadoop.fs.s3a.s3guard.S3GuardTool$BucketInfo.run(S3GuardTool.java:997) at org.apache.hadoop.fs.s3a.s3guard.S3GuardTool.run(S3GuardTool.java:309) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.hadoop.fs.s3a.s3guard.S3GuardTool.run(S3GuardTool.java:1218) at org.apache.hadoop.fs.s3a.s3guard.S3GuardTool.main(S3GuardTool.java:1227) Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: All access to this object has been disabled (Service: Amazon S3; Status Code: 403; Error Code: AllAccessDisabled; Request ID: E6229D7F8134E64F; S3 Extended Request ID: 6SzVz2t4qa8J2Wxo/oc8yBuB13Mgrn9uMKnxVY0hsBd2kU/YdHzW1IaujpJdDXRDCQRX3f1RYn0=), S3 Extended Request ID: 6SzVz2t4qa8J2Wxo/oc8yBuB13Mgrn9uMKnxVY0hsBd2kU/YdHzW1IaujpJdDXRDCQRX3f1RYn0= at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1638) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1303) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1055) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667) at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4229) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4176) at com.amazonaws.services.s3.AmazonS3Client.getAcl(AmazonS3Client.java:3381) at com.amazonaws.services.s3.AmazonS3Client.getBucketAcl(AmazonS3Client.java:1160) at com.amazonaws.services.s3.AmazonS3Client.getBucketAcl(AmazonS3Client.java:1150) at com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:1266) at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$verifyBucketExists$1(S3AFileSystem.java:360) at org.apache.hadoop.fs.s3a.S3ALambda.once(S3ALambda.java:120) ... 16 more

            People

            • Assignee:
              stevel@apache.org Steve Loughran
              Reporter:
              stevel@apache.org Steve Loughran
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:

                Development