Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-11488

Difference in default connection timeout for S3A FS

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.6.0
    • Fix Version/s: 2.7.0
    • Component/s: fs/s3
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The core-default.xml defines fs.s3a.connection.timeout as 5000, and the code under hadoop-tools/hadoop-aws defines it as 50000.

      We should update the former to 50s so it gets taken properly, as we're also noticing that 5s is often too low, especially in cases such as large DistCp operations (which fail with Read timed out errors from the S3 service).

      1. HADOOP-11488.patch
        1 kB
        Harsh J
      2. HADOOP-11488.patch
        1 kB
        Daisuke Kobayashi

        Issue Links

          Activity

          Hide
          daisuke.kobayashi Daisuke Kobayashi added a comment -

          Thanks for filing the jira, Harsh! Please review the patch I've attached here.

          Show
          daisuke.kobayashi Daisuke Kobayashi added a comment - Thanks for filing the jira, Harsh! Please review the patch I've attached here.
          Hide
          qwertymaniac Harsh J added a comment -

          Thanks! LGTM. +1 pending jenkins.

          Show
          qwertymaniac Harsh J added a comment - Thanks! LGTM. +1 pending jenkins.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12692998/HADOOP-11488.patch
          against trunk revision 24315e7.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws:

          org.apache.hadoop.ha.TestZKFailoverControllerStress

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5425//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5425//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12692998/HADOOP-11488.patch against trunk revision 24315e7. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws: org.apache.hadoop.ha.TestZKFailoverControllerStress Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5425//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5425//console This message is automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          -1

          1. this patch adds some needless changes to the tail of the file...please can you cut that bit from a resubmitted patch
          2. Jenkins doesn't test s3a...can someone state that they have manually run the entire hadoop-aws test suite with this option? That includes the tests that went in this weekend which are designed to run fast even though they expect connection failures.
          Show
          stevel@apache.org Steve Loughran added a comment - -1 this patch adds some needless changes to the tail of the file...please can you cut that bit from a resubmitted patch Jenkins doesn't test s3a...can someone state that they have manually run the entire hadoop-aws test suite with this option? That includes the tests that went in this weekend which are designed to run fast even though they expect connection failures.
          Hide
          qwertymaniac Harsh J added a comment -

          Thanks for taking a look Steve.

          For (1) the patch is just adding newline to EOF which was amiss in the document - I don't think thats a bad thing, although yes it is unrelated. I'm willing to let that pass, as it is not intrusive (like say, reordering of import statements).

          For (2), I wasn't aware we didn't run them by default. Can you kindly also explain how to run the tests manually so I may do so? I'll also file a separate JIRA to drop a README under the hadoop-aws directory if it helps, as it is not clear how to otherwise.

          Is your -1 just for these two points or do you also have a concern w.r.t. fixing the discrepancy of timeouts in the code (Constants.java) vs. the preceding XML (core-default.xml)?

          Show
          qwertymaniac Harsh J added a comment - Thanks for taking a look Steve. For (1) the patch is just adding newline to EOF which was amiss in the document - I don't think thats a bad thing, although yes it is unrelated. I'm willing to let that pass, as it is not intrusive (like say, reordering of import statements). For (2), I wasn't aware we didn't run them by default. Can you kindly also explain how to run the tests manually so I may do so? I'll also file a separate JIRA to drop a README under the hadoop-aws directory if it helps, as it is not clear how to otherwise. Is your -1 just for these two points or do you also have a concern w.r.t. fixing the discrepancy of timeouts in the code (Constants.java) vs. the preceding XML (core-default.xml)?
          Hide
          qwertymaniac Harsh J added a comment -

          I see the index.md carrying some instructions. I'll check them out and try running the tests.

          Show
          qwertymaniac Harsh J added a comment - I see the index.md carrying some instructions. I'll check them out and try running the tests.
          Hide
          qwertymaniac Harsh J added a comment -

          OK, I don't have access to an S3 key, so I cannot run the tests. I've uploaded a new patch to satisfy (1). If you need anything else changed, let me know.

          Show
          qwertymaniac Harsh J added a comment - OK, I don't have access to an S3 key, so I cannot run the tests. I've uploaded a new patch to satisfy (1). If you need anything else changed, let me know.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          if you have an AWS account you can run the test, it only costs a few cents

          Show
          stevel@apache.org Steve Loughran added a comment - if you have an AWS account you can run the test, it only costs a few cents
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12693121/HADOOP-11488.patch
          against trunk revision 4a5c3a4.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5428//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5428//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12693121/HADOOP-11488.patch against trunk revision 4a5c3a4. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5428//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5428//console This message is automatically generated.
          Hide
          daisuke.kobayashi Daisuke Kobayashi added a comment -

          Steve Loughran, I've ran the test for hadoop-aws locally and here's the results. Please review and let me know if anything is needed from my side. Thanks!

          daisuke:hadoop-aws/ (HADOOP-11488✗) $ mvn test -Ptests-on
          [INFO] Scanning for projects...
          [INFO]
          [INFO] ------------------------------------------------------------------------
          [INFO] Building Apache Hadoop Amazon Web Services support 3.0.0-SNAPSHOT
          [INFO] ------------------------------------------------------------------------
          [INFO]
          [INFO] --- maven-antrun-plugin:1.7:run (create-testdirs) @ hadoop-aws ---
          [INFO] Executing tasks
          
          main:
          [INFO] Executed tasks
          [INFO]
          [INFO] --- maven-resources-plugin:2.2:resources (default-resources) @ hadoop-aws ---
          [INFO] Using default encoding to copy filtered resources.
          [INFO]
          [INFO] --- maven-compiler-plugin:2.5.1:compile (default-compile) @ hadoop-aws ---
          [INFO] Nothing to compile - all classes are up to date
          [INFO]
          [INFO] --- maven-resources-plugin:2.2:testResources (default-testResources) @ hadoop-aws ---
          [INFO] Using default encoding to copy filtered resources.
          [INFO]
          [INFO] --- maven-compiler-plugin:2.5.1:testCompile (default-testCompile) @ hadoop-aws ---
          [INFO] Nothing to compile - all classes are up to date
          [INFO]
          [INFO] --- maven-surefire-plugin:2.17:test (default-test) @ hadoop-aws ---
          [INFO] Surefire report directory: /src/apache/hadoop/hadoop-tools/hadoop-aws/target/surefire-reports
          
          -------------------------------------------------------
           T E S T S
          -------------------------------------------------------
          
          -------------------------------------------------------
           T E S T S
          -------------------------------------------------------
          Running org.apache.hadoop.fs.contract.s3a.TestS3AContractCreate
          Tests run: 6, Failures: 0, Errors: 0, Skipped: 3, Time elapsed: 53.492 sec - in org.apache.hadoop.fs.contract.s3a.TestS3AContractCreate
          Running org.apache.hadoop.fs.contract.s3a.TestS3AContractDelete
          Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 90.338 sec - in org.apache.hadoop.fs.contract.s3a.TestS3AContractDelete
          Running org.apache.hadoop.fs.contract.s3a.TestS3AContractMkdir
          Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 57.846 sec - in org.apache.hadoop.fs.contract.s3a.TestS3AContractMkdir
          Running org.apache.hadoop.fs.contract.s3a.TestS3AContractOpen
          Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 50.969 sec - in org.apache.hadoop.fs.contract.s3a.TestS3AContractOpen
          Running org.apache.hadoop.fs.contract.s3a.TestS3AContractRename
          Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 93.803 sec - in org.apache.hadoop.fs.contract.s3a.TestS3AContractRename
          Running org.apache.hadoop.fs.contract.s3a.TestS3AContractRootDir
          Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 32.372 sec - in org.apache.hadoop.fs.contract.s3a.TestS3AContractRootDir
          Running org.apache.hadoop.fs.contract.s3a.TestS3AContractSeek
          Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 115.805 sec - in org.apache.hadoop.fs.contract.s3a.TestS3AContractSeek
          Running org.apache.hadoop.fs.contract.s3n.TestS3NContractCreate
          Tests run: 6, Failures: 0, Errors: 0, Skipped: 3, Time elapsed: 37.581 sec - in org.apache.hadoop.fs.contract.s3n.TestS3NContractCreate
          Running org.apache.hadoop.fs.contract.s3n.TestS3NContractDelete
          Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 71.566 sec - in org.apache.hadoop.fs.contract.s3n.TestS3NContractDelete
          Running org.apache.hadoop.fs.contract.s3n.TestS3NContractMkdir
          Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 48.025 sec - in org.apache.hadoop.fs.contract.s3n.TestS3NContractMkdir
          Running org.apache.hadoop.fs.contract.s3n.TestS3NContractOpen
          Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 37.374 sec - in org.apache.hadoop.fs.contract.s3n.TestS3NContractOpen
          Running org.apache.hadoop.fs.contract.s3n.TestS3NContractRename
          Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 65.436 sec - in org.apache.hadoop.fs.contract.s3n.TestS3NContractRename
          Running org.apache.hadoop.fs.contract.s3n.TestS3NContractRootDir
          Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 25.648 sec - in org.apache.hadoop.fs.contract.s3n.TestS3NContractRootDir
          Running org.apache.hadoop.fs.contract.s3n.TestS3NContractSeek
          Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 78.111 sec - in org.apache.hadoop.fs.contract.s3n.TestS3NContractSeek
          Running org.apache.hadoop.fs.s3.TestInMemoryS3FileSystemContract
          Tests run: 44, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.918 sec - in org.apache.hadoop.fs.s3.TestInMemoryS3FileSystemContract
          Running org.apache.hadoop.fs.s3.TestINode
          Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.086 sec - in org.apache.hadoop.fs.s3.TestINode
          Running org.apache.hadoop.fs.s3.TestS3Credentials
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.098 sec - in org.apache.hadoop.fs.s3.TestS3Credentials
          Running org.apache.hadoop.fs.s3.TestS3FileSystem
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.441 sec - in org.apache.hadoop.fs.s3.TestS3FileSystem
          Running org.apache.hadoop.fs.s3.TestS3InMemoryFileSystem
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.27 sec - in org.apache.hadoop.fs.s3.TestS3InMemoryFileSystem
          Running org.apache.hadoop.fs.s3a.scale.TestS3ADeleteManyFiles
          Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1,904.4 sec <<< FAILURE! - in org.apache.hadoop.fs.s3a.scale.TestS3ADeleteManyFiles
          testBulkRenameAndDelete(org.apache.hadoop.fs.s3a.scale.TestS3ADeleteManyFiles)  Time elapsed: 1,800.007 sec  <<< ERROR!
          java.lang.Exception: test timed out after 1800000 milliseconds
          	at java.net.SocketInputStream.socketRead0(Native Method)
          	at java.net.SocketInputStream.read(SocketInputStream.java:152)
          	at java.net.SocketInputStream.read(SocketInputStream.java:122)
          	at sun.security.ssl.InputRecord.readFully(InputRecord.java:442)
          	at sun.security.ssl.InputRecord.read(InputRecord.java:480)
          	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:927)
          	at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:884)
          	at sun.security.ssl.AppInputStream.read(AppInputStream.java:102)
          	at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166)
          	at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90)
          	at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:281)
          	at org.apache.http.impl.conn.LoggingSessionInputBuffer.readLine(LoggingSessionInputBuffer.java:115)
          	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:92)
          	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:62)
          	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:254)
          	at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:289)
          	at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:252)
          	at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:191)
          	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:300)
          	at com.amazonaws.http.protocol.SdkHttpRequestExecutor.doReceiveResponse(SdkHttpRequestExecutor.java:66)
          	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:127)
          	at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:715)
          	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:520)
          	at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
          	at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
          	at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:384)
          	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232)
          	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3528)
          	at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:976)
          	at com.amazonaws.services.s3.transfer.TransferManager.copy(TransferManager.java:1285)
          	at com.amazonaws.services.s3.transfer.TransferManager.copy(TransferManager.java:1223)
          	at org.apache.hadoop.fs.s3a.S3AFileSystem.copyFile(S3AFileSystem.java:1085)
          	at org.apache.hadoop.fs.s3a.S3AFileSystem.rename(S3AFileSystem.java:542)
          	at org.apache.hadoop.fs.s3a.scale.TestS3ADeleteManyFiles.testBulkRenameAndDelete(TestS3ADeleteManyFiles.java:95)
          
          Running org.apache.hadoop.fs.s3a.TestS3AConfiguration
          Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.635 sec - in org.apache.hadoop.fs.s3a.TestS3AConfiguration
          Running org.apache.hadoop.fs.s3a.TestS3AFileSystemContract
          Tests run: 43, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1,010.135 sec - in org.apache.hadoop.fs.s3a.TestS3AFileSystemContract
          Running org.apache.hadoop.fs.s3native.TestInMemoryNativeS3FileSystemContract
          Tests run: 52, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.543 sec - in org.apache.hadoop.fs.s3native.TestInMemoryNativeS3FileSystemContract
          Running org.apache.hadoop.fs.s3native.TestJets3tNativeFileSystemStore
          Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 757.153 sec - in org.apache.hadoop.fs.s3native.TestJets3tNativeFileSystemStore
          Running org.apache.hadoop.fs.s3native.TestS3NInMemoryFileSystem
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.753 sec - in org.apache.hadoop.fs.s3native.TestS3NInMemoryFileSystem
          
          Results :
          
          Tests in error:
            TestS3ADeleteManyFiles.testBulkRenameAndDelete:95 ?  test timed out after 1800...
          
          Tests run: 247, Failures: 0, Errors: 1, Skipped: 6
          
          [INFO] ------------------------------------------------------------------------
          [INFO] BUILD FAILURE
          [INFO] ------------------------------------------------------------------------
          [INFO] Total time: 1:15:46.776s
          [INFO] Finished at: Tue Jan 20 17:45:28 JST 2015
          [INFO] Final Memory: 10M/310M
          [INFO] ------------------------------------------------------------------------
          [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.17:test (default-test) on project hadoop-aws: There are test failures.
          [ERROR]
          [ERROR] Please refer to /src/apache/hadoop/hadoop-tools/hadoop-aws/target/surefire-reports for the individual test results.
          [ERROR] -> [Help 1]
          [ERROR]
          [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
          [ERROR] Re-run Maven using the -X switch to enable full debug logging.
          [ERROR]
          [ERROR] For more information about the errors and possible solutions, please read the following articles:
          [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
          
          Show
          daisuke.kobayashi Daisuke Kobayashi added a comment - Steve Loughran , I've ran the test for hadoop-aws locally and here's the results. Please review and let me know if anything is needed from my side. Thanks! daisuke:hadoop-aws/ (HADOOP-11488✗) $ mvn test -Ptests-on [INFO] Scanning for projects... [INFO] [INFO] ------------------------------------------------------------------------ [INFO] Building Apache Hadoop Amazon Web Services support 3.0.0-SNAPSHOT [INFO] ------------------------------------------------------------------------ [INFO] [INFO] --- maven-antrun-plugin:1.7:run (create-testdirs) @ hadoop-aws --- [INFO] Executing tasks main: [INFO] Executed tasks [INFO] [INFO] --- maven-resources-plugin:2.2:resources (default-resources) @ hadoop-aws --- [INFO] Using default encoding to copy filtered resources. [INFO] [INFO] --- maven-compiler-plugin:2.5.1:compile (default-compile) @ hadoop-aws --- [INFO] Nothing to compile - all classes are up to date [INFO] [INFO] --- maven-resources-plugin:2.2:testResources (default-testResources) @ hadoop-aws --- [INFO] Using default encoding to copy filtered resources. [INFO] [INFO] --- maven-compiler-plugin:2.5.1:testCompile (default-testCompile) @ hadoop-aws --- [INFO] Nothing to compile - all classes are up to date [INFO] [INFO] --- maven-surefire-plugin:2.17:test (default-test) @ hadoop-aws --- [INFO] Surefire report directory: /src/apache/hadoop/hadoop-tools/hadoop-aws/target/surefire-reports ------------------------------------------------------- T E S T S ------------------------------------------------------- ------------------------------------------------------- T E S T S ------------------------------------------------------- Running org.apache.hadoop.fs.contract.s3a.TestS3AContractCreate Tests run: 6, Failures: 0, Errors: 0, Skipped: 3, Time elapsed: 53.492 sec - in org.apache.hadoop.fs.contract.s3a.TestS3AContractCreate Running org.apache.hadoop.fs.contract.s3a.TestS3AContractDelete Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 90.338 sec - in org.apache.hadoop.fs.contract.s3a.TestS3AContractDelete Running org.apache.hadoop.fs.contract.s3a.TestS3AContractMkdir Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 57.846 sec - in org.apache.hadoop.fs.contract.s3a.TestS3AContractMkdir Running org.apache.hadoop.fs.contract.s3a.TestS3AContractOpen Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 50.969 sec - in org.apache.hadoop.fs.contract.s3a.TestS3AContractOpen Running org.apache.hadoop.fs.contract.s3a.TestS3AContractRename Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 93.803 sec - in org.apache.hadoop.fs.contract.s3a.TestS3AContractRename Running org.apache.hadoop.fs.contract.s3a.TestS3AContractRootDir Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 32.372 sec - in org.apache.hadoop.fs.contract.s3a.TestS3AContractRootDir Running org.apache.hadoop.fs.contract.s3a.TestS3AContractSeek Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 115.805 sec - in org.apache.hadoop.fs.contract.s3a.TestS3AContractSeek Running org.apache.hadoop.fs.contract.s3n.TestS3NContractCreate Tests run: 6, Failures: 0, Errors: 0, Skipped: 3, Time elapsed: 37.581 sec - in org.apache.hadoop.fs.contract.s3n.TestS3NContractCreate Running org.apache.hadoop.fs.contract.s3n.TestS3NContractDelete Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 71.566 sec - in org.apache.hadoop.fs.contract.s3n.TestS3NContractDelete Running org.apache.hadoop.fs.contract.s3n.TestS3NContractMkdir Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 48.025 sec - in org.apache.hadoop.fs.contract.s3n.TestS3NContractMkdir Running org.apache.hadoop.fs.contract.s3n.TestS3NContractOpen Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 37.374 sec - in org.apache.hadoop.fs.contract.s3n.TestS3NContractOpen Running org.apache.hadoop.fs.contract.s3n.TestS3NContractRename Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 65.436 sec - in org.apache.hadoop.fs.contract.s3n.TestS3NContractRename Running org.apache.hadoop.fs.contract.s3n.TestS3NContractRootDir Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 25.648 sec - in org.apache.hadoop.fs.contract.s3n.TestS3NContractRootDir Running org.apache.hadoop.fs.contract.s3n.TestS3NContractSeek Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 78.111 sec - in org.apache.hadoop.fs.contract.s3n.TestS3NContractSeek Running org.apache.hadoop.fs.s3.TestInMemoryS3FileSystemContract Tests run: 44, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.918 sec - in org.apache.hadoop.fs.s3.TestInMemoryS3FileSystemContract Running org.apache.hadoop.fs.s3.TestINode Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.086 sec - in org.apache.hadoop.fs.s3.TestINode Running org.apache.hadoop.fs.s3.TestS3Credentials Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.098 sec - in org.apache.hadoop.fs.s3.TestS3Credentials Running org.apache.hadoop.fs.s3.TestS3FileSystem Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.441 sec - in org.apache.hadoop.fs.s3.TestS3FileSystem Running org.apache.hadoop.fs.s3.TestS3InMemoryFileSystem Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.27 sec - in org.apache.hadoop.fs.s3.TestS3InMemoryFileSystem Running org.apache.hadoop.fs.s3a.scale.TestS3ADeleteManyFiles Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1,904.4 sec <<< FAILURE! - in org.apache.hadoop.fs.s3a.scale.TestS3ADeleteManyFiles testBulkRenameAndDelete(org.apache.hadoop.fs.s3a.scale.TestS3ADeleteManyFiles) Time elapsed: 1,800.007 sec <<< ERROR! java.lang.Exception: test timed out after 1800000 milliseconds at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:152) at java.net.SocketInputStream.read(SocketInputStream.java:122) at sun.security.ssl.InputRecord.readFully(InputRecord.java:442) at sun.security.ssl.InputRecord.read(InputRecord.java:480) at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:927) at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:884) at sun.security.ssl.AppInputStream.read(AppInputStream.java:102) at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166) at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90) at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:281) at org.apache.http.impl.conn.LoggingSessionInputBuffer.readLine(LoggingSessionInputBuffer.java:115) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:92) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:62) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:254) at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:289) at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:252) at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:191) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:300) at com.amazonaws.http.protocol.SdkHttpRequestExecutor.doReceiveResponse(SdkHttpRequestExecutor.java:66) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:127) at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:715) at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:520) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805) at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:384) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3528) at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:976) at com.amazonaws.services.s3.transfer.TransferManager.copy(TransferManager.java:1285) at com.amazonaws.services.s3.transfer.TransferManager.copy(TransferManager.java:1223) at org.apache.hadoop.fs.s3a.S3AFileSystem.copyFile(S3AFileSystem.java:1085) at org.apache.hadoop.fs.s3a.S3AFileSystem.rename(S3AFileSystem.java:542) at org.apache.hadoop.fs.s3a.scale.TestS3ADeleteManyFiles.testBulkRenameAndDelete(TestS3ADeleteManyFiles.java:95) Running org.apache.hadoop.fs.s3a.TestS3AConfiguration Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.635 sec - in org.apache.hadoop.fs.s3a.TestS3AConfiguration Running org.apache.hadoop.fs.s3a.TestS3AFileSystemContract Tests run: 43, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1,010.135 sec - in org.apache.hadoop.fs.s3a.TestS3AFileSystemContract Running org.apache.hadoop.fs.s3native.TestInMemoryNativeS3FileSystemContract Tests run: 52, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.543 sec - in org.apache.hadoop.fs.s3native.TestInMemoryNativeS3FileSystemContract Running org.apache.hadoop.fs.s3native.TestJets3tNativeFileSystemStore Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 757.153 sec - in org.apache.hadoop.fs.s3native.TestJets3tNativeFileSystemStore Running org.apache.hadoop.fs.s3native.TestS3NInMemoryFileSystem Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.753 sec - in org.apache.hadoop.fs.s3native.TestS3NInMemoryFileSystem Results : Tests in error: TestS3ADeleteManyFiles.testBulkRenameAndDelete:95 ? test timed out after 1800... Tests run: 247, Failures: 0, Errors: 1, Skipped: 6 [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 1:15:46.776s [INFO] Finished at: Tue Jan 20 17:45:28 JST 2015 [INFO] Final Memory: 10M/310M [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.17:test (default-test) on project hadoop-aws: There are test failures. [ERROR] [ERROR] Please refer to /src/apache/hadoop/hadoop-tools/hadoop-aws/target/surefire-reports for the individual test results. [ERROR] -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
          Hide
          stevel@apache.org Steve Loughran added a comment -

          -1 until testBulkRenameAndDelete passes. It worked when I last ran the test suite, ~8-9 days ago.

          1. Does it pass if you retry it?
          2. Does it pass if you target a different s3 instance? I test against s3 EU.
          3. Does it pass if you revert the patch?
          Show
          stevel@apache.org Steve Loughran added a comment - -1 until testBulkRenameAndDelete passes. It worked when I last ran the test suite, ~8-9 days ago. Does it pass if you retry it? Does it pass if you target a different s3 instance? I test against s3 EU. Does it pass if you revert the patch?
          Hide
          thodemoor Thomas Demoor added a comment -

          Steve Loughran Tests pass against bucket in eu-west1 (Ireland).

          -------------------------------------------------------
           T E S T S
          -------------------------------------------------------
          Running org.apache.hadoop.fs.s3a.TestS3AFileSystemContract
          Tests run: 43, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 186.434 sec - in org.apache.hadoop.fs.s3a.TestS3AFileSystemContract
          Running org.apache.hadoop.fs.s3a.scale.TestS3ADeleteManyFiles
          Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 366.926 sec - in org.apache.hadoop.fs.s3a.scale.TestS3ADeleteManyFiles
          Running org.apache.hadoop.fs.s3a.TestS3AConfiguration
          Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.551 sec - in org.apache.hadoop.fs.s3a.TestS3AConfiguration
          Running org.apache.hadoop.fs.contract.s3a.TestS3AContractRootDir
          Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.301 sec - in org.apache.hadoop.fs.contract.s3a.TestS3AContractRootDir
          Running org.apache.hadoop.fs.contract.s3a.TestS3AContractDelete
          Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 24.643 sec - in org.apache.hadoop.fs.contract.s3a.TestS3AContractDelete
          Running org.apache.hadoop.fs.contract.s3a.TestS3AContractRename
          Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.497 sec - in org.apache.hadoop.fs.contract.s3a.TestS3AContractRename
          Running org.apache.hadoop.fs.contract.s3a.TestS3AContractMkdir
          Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.127 sec - in org.apache.hadoop.fs.contract.s3a.TestS3AContractMkdir
          Running org.apache.hadoop.fs.contract.s3a.TestS3AContractOpen
          Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.894 sec - in org.apache.hadoop.fs.contract.s3a.TestS3AContractOpen
          Running org.apache.hadoop.fs.contract.s3a.TestS3AContractCreate
          Tests run: 6, Failures: 0, Errors: 0, Skipped: 3, Time elapsed: 9.769 sec - in org.apache.hadoop.fs.contract.s3a.TestS3AContractCreate
          Running org.apache.hadoop.fs.contract.s3a.TestS3AContractSeek
          Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 20.816 sec - in org.apache.hadoop.fs.contract.s3a.TestS3AContractSeek
          Running org.apache.hadoop.fs.contract.s3n.TestS3NContractMkdir
          Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.431 sec - in org.apache.hadoop.fs.contract.s3n.TestS3NContractMkdir
          Running org.apache.hadoop.fs.contract.s3n.TestS3NContractRename
          Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.882 sec - in org.apache.hadoop.fs.contract.s3n.TestS3NContractRename
          Running org.apache.hadoop.fs.contract.s3n.TestS3NContractSeek
          Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.157 sec - in org.apache.hadoop.fs.contract.s3n.TestS3NContractSeek
          Running org.apache.hadoop.fs.contract.s3n.TestS3NContractCreate
          Tests run: 6, Failures: 0, Errors: 0, Skipped: 3, Time elapsed: 7.307 sec - in org.apache.hadoop.fs.contract.s3n.TestS3NContractCreate
          Running org.apache.hadoop.fs.contract.s3n.TestS3NContractDelete
          Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.792 sec - in org.apache.hadoop.fs.contract.s3n.TestS3NContractDelete
          Running org.apache.hadoop.fs.contract.s3n.TestS3NContractRootDir
          Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.093 sec - in org.apache.hadoop.fs.contract.s3n.TestS3NContractRootDir
          Running org.apache.hadoop.fs.contract.s3n.TestS3NContractOpen
          Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.693 sec - in org.apache.hadoop.fs.contract.s3n.TestS3NContractOpen
          Running org.apache.hadoop.fs.s3.TestS3Credentials
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.13 sec - in org.apache.hadoop.fs.s3.TestS3Credentials
          Running org.apache.hadoop.fs.s3.TestS3InMemoryFileSystem
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.427 sec - in org.apache.hadoop.fs.s3.TestS3InMemoryFileSystem
          Running org.apache.hadoop.fs.s3.TestInMemoryS3FileSystemContract
          Tests run: 44, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.068 sec - in org.apache.hadoop.fs.s3.TestInMemoryS3FileSystemContract
          Running org.apache.hadoop.fs.s3.TestINode
          Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.145 sec - in org.apache.hadoop.fs.s3.TestINode
          Running org.apache.hadoop.fs.s3.TestS3FileSystem
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.475 sec - in org.apache.hadoop.fs.s3.TestS3FileSystem
          Running org.apache.hadoop.fs.s3native.TestJets3tNativeFileSystemStore
          Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 17.004 sec - in org.apache.hadoop.fs.s3native.TestJets3tNativeFileSystemStore
          Running org.apache.hadoop.fs.s3native.TestInMemoryNativeS3FileSystemContract
          Tests run: 52, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.512 sec - in org.apache.hadoop.fs.s3native.TestInMemoryNativeS3FileSystemContract
          Running org.apache.hadoop.fs.s3native.TestS3NInMemoryFileSystem
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.634 sec - in org.apache.hadoop.fs.s3native.TestS3NInMemoryFileSystem
          
          Results :
          
          Tests run: 247, Failures: 0, Errors: 0, Skipped: 6
          
          [INFO] ------------------------------------------------------------------------
          [INFO] BUILD SUCCESS
          [INFO] ------------------------------------------------------------------------
          [INFO] Total time: 12:31.091s
          [INFO] Finished at: Tue Jan 20 17:30:50 CET 2015
          [INFO] Final Memory: 12M/327M
          [INFO] ------------------------------------------------------------------------
          

          Daisuke Kobayashi your test durations are quite long (compared to mine, see above) so I assume you're connection is too slow to meaningfully run the tests (do you run against a bucket in a region nearby?). You encounter the test timeout before possibly hitting the (elongated by this patch) connection timeout.

          As this issue looks like it has some people fairly new to testing S3A, a small side note: I will extend the testing guidelines in the documentation while I pass by to document my recently accepted patches. A quick preview:

          • When testing against Amazon S3, tests can sometimes randomly fail due to eventual consistency. I frequently get this on TestS3AFileSystemContract.testRenameRootDirForbidden. Retry and it should run fine (or use a strong consistent object store )
          • Beware test failures during multi-part uploads. Completed parts of a failed multi-part upload do not show up in "ls" but linger (and are billed) eternally. One can remove them manually or through purging: setting fs.s3a.multipart.purge to true.
          Show
          thodemoor Thomas Demoor added a comment - Steve Loughran Tests pass against bucket in eu-west1 (Ireland). ------------------------------------------------------- T E S T S ------------------------------------------------------- Running org.apache.hadoop.fs.s3a.TestS3AFileSystemContract Tests run: 43, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 186.434 sec - in org.apache.hadoop.fs.s3a.TestS3AFileSystemContract Running org.apache.hadoop.fs.s3a.scale.TestS3ADeleteManyFiles Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 366.926 sec - in org.apache.hadoop.fs.s3a.scale.TestS3ADeleteManyFiles Running org.apache.hadoop.fs.s3a.TestS3AConfiguration Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.551 sec - in org.apache.hadoop.fs.s3a.TestS3AConfiguration Running org.apache.hadoop.fs.contract.s3a.TestS3AContractRootDir Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.301 sec - in org.apache.hadoop.fs.contract.s3a.TestS3AContractRootDir Running org.apache.hadoop.fs.contract.s3a.TestS3AContractDelete Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 24.643 sec - in org.apache.hadoop.fs.contract.s3a.TestS3AContractDelete Running org.apache.hadoop.fs.contract.s3a.TestS3AContractRename Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.497 sec - in org.apache.hadoop.fs.contract.s3a.TestS3AContractRename Running org.apache.hadoop.fs.contract.s3a.TestS3AContractMkdir Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.127 sec - in org.apache.hadoop.fs.contract.s3a.TestS3AContractMkdir Running org.apache.hadoop.fs.contract.s3a.TestS3AContractOpen Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.894 sec - in org.apache.hadoop.fs.contract.s3a.TestS3AContractOpen Running org.apache.hadoop.fs.contract.s3a.TestS3AContractCreate Tests run: 6, Failures: 0, Errors: 0, Skipped: 3, Time elapsed: 9.769 sec - in org.apache.hadoop.fs.contract.s3a.TestS3AContractCreate Running org.apache.hadoop.fs.contract.s3a.TestS3AContractSeek Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 20.816 sec - in org.apache.hadoop.fs.contract.s3a.TestS3AContractSeek Running org.apache.hadoop.fs.contract.s3n.TestS3NContractMkdir Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.431 sec - in org.apache.hadoop.fs.contract.s3n.TestS3NContractMkdir Running org.apache.hadoop.fs.contract.s3n.TestS3NContractRename Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.882 sec - in org.apache.hadoop.fs.contract.s3n.TestS3NContractRename Running org.apache.hadoop.fs.contract.s3n.TestS3NContractSeek Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.157 sec - in org.apache.hadoop.fs.contract.s3n.TestS3NContractSeek Running org.apache.hadoop.fs.contract.s3n.TestS3NContractCreate Tests run: 6, Failures: 0, Errors: 0, Skipped: 3, Time elapsed: 7.307 sec - in org.apache.hadoop.fs.contract.s3n.TestS3NContractCreate Running org.apache.hadoop.fs.contract.s3n.TestS3NContractDelete Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.792 sec - in org.apache.hadoop.fs.contract.s3n.TestS3NContractDelete Running org.apache.hadoop.fs.contract.s3n.TestS3NContractRootDir Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.093 sec - in org.apache.hadoop.fs.contract.s3n.TestS3NContractRootDir Running org.apache.hadoop.fs.contract.s3n.TestS3NContractOpen Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.693 sec - in org.apache.hadoop.fs.contract.s3n.TestS3NContractOpen Running org.apache.hadoop.fs.s3.TestS3Credentials Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.13 sec - in org.apache.hadoop.fs.s3.TestS3Credentials Running org.apache.hadoop.fs.s3.TestS3InMemoryFileSystem Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.427 sec - in org.apache.hadoop.fs.s3.TestS3InMemoryFileSystem Running org.apache.hadoop.fs.s3.TestInMemoryS3FileSystemContract Tests run: 44, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.068 sec - in org.apache.hadoop.fs.s3.TestInMemoryS3FileSystemContract Running org.apache.hadoop.fs.s3.TestINode Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.145 sec - in org.apache.hadoop.fs.s3.TestINode Running org.apache.hadoop.fs.s3.TestS3FileSystem Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.475 sec - in org.apache.hadoop.fs.s3.TestS3FileSystem Running org.apache.hadoop.fs.s3native.TestJets3tNativeFileSystemStore Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 17.004 sec - in org.apache.hadoop.fs.s3native.TestJets3tNativeFileSystemStore Running org.apache.hadoop.fs.s3native.TestInMemoryNativeS3FileSystemContract Tests run: 52, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.512 sec - in org.apache.hadoop.fs.s3native.TestInMemoryNativeS3FileSystemContract Running org.apache.hadoop.fs.s3native.TestS3NInMemoryFileSystem Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.634 sec - in org.apache.hadoop.fs.s3native.TestS3NInMemoryFileSystem Results : Tests run: 247, Failures: 0, Errors: 0, Skipped: 6 [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 12:31.091s [INFO] Finished at: Tue Jan 20 17:30:50 CET 2015 [INFO] Final Memory: 12M/327M [INFO] ------------------------------------------------------------------------ Daisuke Kobayashi your test durations are quite long (compared to mine, see above) so I assume you're connection is too slow to meaningfully run the tests (do you run against a bucket in a region nearby?). You encounter the test timeout before possibly hitting the (elongated by this patch) connection timeout. As this issue looks like it has some people fairly new to testing S3A, a small side note: I will extend the testing guidelines in the documentation while I pass by to document my recently accepted patches. A quick preview: When testing against Amazon S3, tests can sometimes randomly fail due to eventual consistency. I frequently get this on TestS3AFileSystemContract.testRenameRootDirForbidden. Retry and it should run fine (or use a strong consistent object store ) Beware test failures during multi-part uploads. Completed parts of a failed multi-part upload do not show up in "ls" but linger (and are billed) eternally. One can remove them manually or through purging: setting fs.s3a.multipart.purge to true.
          Hide
          daisuke.kobayashi Daisuke Kobayashi added a comment -

          Thomas Demoor, thanks for the confirmation! Am located in Tokyo and using a bucket on Tokyo region, but it's still slow...
          Steve Loughran, can I get +1 from you per the result of test by Thomas?

          Show
          daisuke.kobayashi Daisuke Kobayashi added a comment - Thomas Demoor , thanks for the confirmation! Am located in Tokyo and using a bucket on Tokyo region, but it's still slow... Steve Loughran , can I get +1 from you per the result of test by Thomas?
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Based on Thomas's results, +1.

          tests against the object stores are always a bit inconsistent, with more variance across the swift implementations, and given that when you run them from your desk you are more likely to hit connectivity problems, potentially brittle.

          Yet those same problems will arise in the real worlds, so it's good to encounter them early.

          Thomas: should we be setting fs.s3a.multipart.purge by default in test runs?

          Show
          stevel@apache.org Steve Loughran added a comment - Based on Thomas's results, +1. tests against the object stores are always a bit inconsistent, with more variance across the swift implementations, and given that when you run them from your desk you are more likely to hit connectivity problems, potentially brittle. Yet those same problems will arise in the real worlds, so it's good to encounter them early. Thomas: should we be setting fs.s3a.multipart.purge by default in test runs?
          Hide
          thodemoor Thomas Demoor added a comment -

          My pleasure, Daisuke.

          Steve, that seems sensible.

          Purging uses abortMultiPartUploads, which is a list operation followed by a delete operation per listed upload. It happens synchronously at the end of fs.initialize(). The test might timeout if the bucket has too much (thousands?) in-progress uploads that need to be aborted. However, the s3a docs explicitly state that a dedicated test bucket should be used and the current test classes do not perform multiple concurrent multiPartUploads so typically 0 and at most 1 upload will be purged. Consequently, we can safely assume "too much in-progress uploads" to be a "should not happen" during testing.

          I see 2 ways to implement this:
          1. add purge = true, purgetime = 0 secs to the conf in S3ATestUtils.createTestFileSystem
          +: 2 lines of code
          -: cleaning up only happens on the next test run

          2. in tearDown() of each test class : run S3ATestUtils.createTestFileSystem with a config that has purge=true, purgetime = 0 secs
          +: immediate cleanup at end of test (except if test is interrupted, f.i. ctrl-c)
          -: each test class has to explicitly include the code path

          Let me know what you prefer and I'll create an issue addressing this.

          Show
          thodemoor Thomas Demoor added a comment - My pleasure, Daisuke. Steve, that seems sensible. Purging uses abortMultiPartUploads, which is a list operation followed by a delete operation per listed upload. It happens synchronously at the end of fs.initialize(). The test might timeout if the bucket has too much (thousands?) in-progress uploads that need to be aborted. However, the s3a docs explicitly state that a dedicated test bucket should be used and the current test classes do not perform multiple concurrent multiPartUploads so typically 0 and at most 1 upload will be purged. Consequently, we can safely assume "too much in-progress uploads" to be a "should not happen" during testing. I see 2 ways to implement this: 1. add purge = true, purgetime = 0 secs to the conf in S3ATestUtils.createTestFileSystem +: 2 lines of code -: cleaning up only happens on the next test run 2. in tearDown() of each test class : run S3ATestUtils.createTestFileSystem with a config that has purge=true, purgetime = 0 secs +: immediate cleanup at end of test (except if test is interrupted, f.i. ctrl-c) -: each test class has to explicitly include the code path Let me know what you prefer and I'll create an issue addressing this.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          although option #2 is "purest", I'm going to recommend #1. Why? if creating the test FS fails, then trying to do it again in the teardown is simply going to create more problems

          -steve

          Show
          stevel@apache.org Steve Loughran added a comment - although option #2 is "purest", I'm going to recommend #1. Why? if creating the test FS fails, then trying to do it again in the teardown is simply going to create more problems -steve
          Hide
          thodemoor Thomas Demoor added a comment -

          Implementing in HADOOP-11520

          Show
          thodemoor Thomas Demoor added a comment - Implementing in HADOOP-11520
          Hide
          qwertymaniac Harsh J added a comment -

          Thanks for all the work here Thomas, Steve and Daisuke! Committing to branch-2 and trunk shortly.

          Show
          qwertymaniac Harsh J added a comment - Thanks for all the work here Thomas, Steve and Daisuke! Committing to branch-2 and trunk shortly.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #6976 (See https://builds.apache.org/job/Hadoop-trunk-Commit/6976/)
          HADOOP-11488. Difference in default connection timeout for S3A FS. Contributed by Daisuke Kobayashi. (harsh: rev ffc75d6ebe4912f20f4f4870d2a50abbe4557bfa)

          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md
          • hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #6976 (See https://builds.apache.org/job/Hadoop-trunk-Commit/6976/ ) HADOOP-11488 . Difference in default connection timeout for S3A FS. Contributed by Daisuke Kobayashi. (harsh: rev ffc75d6ebe4912f20f4f4870d2a50abbe4557bfa) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #91 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/91/)
          HADOOP-11488. Difference in default connection timeout for S3A FS. Contributed by Daisuke Kobayashi. (harsh: rev ffc75d6ebe4912f20f4f4870d2a50abbe4557bfa)

          • hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md
          • hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #91 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/91/ ) HADOOP-11488 . Difference in default connection timeout for S3A FS. Contributed by Daisuke Kobayashi. (harsh: rev ffc75d6ebe4912f20f4f4870d2a50abbe4557bfa) hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md hadoop-common-project/hadoop-common/src/main/resources/core-default.xml hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk #825 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/825/)
          HADOOP-11488. Difference in default connection timeout for S3A FS. Contributed by Daisuke Kobayashi. (harsh: rev ffc75d6ebe4912f20f4f4870d2a50abbe4557bfa)

          • hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md
          • hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #825 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/825/ ) HADOOP-11488 . Difference in default connection timeout for S3A FS. Contributed by Daisuke Kobayashi. (harsh: rev ffc75d6ebe4912f20f4f4870d2a50abbe4557bfa) hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md hadoop-common-project/hadoop-common/src/main/resources/core-default.xml hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2023 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2023/)
          HADOOP-11488. Difference in default connection timeout for S3A FS. Contributed by Daisuke Kobayashi. (harsh: rev ffc75d6ebe4912f20f4f4870d2a50abbe4557bfa)

          • hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2023 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2023/ ) HADOOP-11488 . Difference in default connection timeout for S3A FS. Contributed by Daisuke Kobayashi. (harsh: rev ffc75d6ebe4912f20f4f4870d2a50abbe4557bfa) hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #88 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/88/)
          HADOOP-11488. Difference in default connection timeout for S3A FS. Contributed by Daisuke Kobayashi. (harsh: rev ffc75d6ebe4912f20f4f4870d2a50abbe4557bfa)

          • hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #88 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/88/ ) HADOOP-11488 . Difference in default connection timeout for S3A FS. Contributed by Daisuke Kobayashi. (harsh: rev ffc75d6ebe4912f20f4f4870d2a50abbe4557bfa) hadoop-common-project/hadoop-common/src/main/resources/core-default.xml hadoop-common-project/hadoop-common/CHANGES.txt hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #92 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/92/)
          HADOOP-11488. Difference in default connection timeout for S3A FS. Contributed by Daisuke Kobayashi. (harsh: rev ffc75d6ebe4912f20f4f4870d2a50abbe4557bfa)

          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md
          • hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #92 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/92/ ) HADOOP-11488 . Difference in default connection timeout for S3A FS. Contributed by Daisuke Kobayashi. (harsh: rev ffc75d6ebe4912f20f4f4870d2a50abbe4557bfa) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2042 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2042/)
          HADOOP-11488. Difference in default connection timeout for S3A FS. Contributed by Daisuke Kobayashi. (harsh: rev ffc75d6ebe4912f20f4f4870d2a50abbe4557bfa)

          • hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2042 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2042/ ) HADOOP-11488 . Difference in default connection timeout for S3A FS. Contributed by Daisuke Kobayashi. (harsh: rev ffc75d6ebe4912f20f4f4870d2a50abbe4557bfa) hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/main/resources/core-default.xml

            People

            • Assignee:
              daisuke.kobayashi Daisuke Kobayashi
              Reporter:
              qwertymaniac Harsh J
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development