Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.8.0, 2.9.0, 3.0.0-alpha2
    • Fix Version/s: 2.8.0, 3.0.0-alpha2
    • Component/s: fs/s3
    • Labels:
      None
    • Environment:

      landsat bucket from the UK

    • Hadoop Flags:
      Reviewed

      Description

      The ITestS3AInputStreamPerformance.testTimeToOpenAndReadWholeFileBlocks takes 15s on branch-2, but is now taking minutes.

      This is a regression, and it's surfacing on some internal branches too. Even where the code hasn't changed. It does not happen on branch-2, which has a later SDK.

      1. HADOOP-13871-001.patch
        2 kB
        Steve Loughran
      2. HADOOP-13871-002.patch
        11 kB
        Steve Loughran
      3. HADOOP-13871-003.patch
        12 kB
        Steve Loughran
      4. org.apache.hadoop.fs.s3a.scale.ITestS3AInputStreamPerformance-output.txt
        1.10 MB
        Steve Loughran

        Issue Links

          Activity

          Hide
          stevel@apache.org Steve Loughran added a comment -
          2016-12-07 15:14:06,958 [Thread-1] INFO  contract.AbstractFSContractTestBase (AbstractFSContractTestBase.java:setup(184)) - Test filesystem = s3a://hwdev-steve-ireland-new implemented by S3AFileSystem{uri=s3a://hwdev-steve-ireland-new, workingDir=s3a://hwdev-steve-ireland-new/user/stevel, inputPolicy=normal, partSize=8000000, enableMultiObjectsDelete=true, maxKeys=5000, readAhead=65536, blockSize=33554432, multiPartThreshold=2147483647, blockFactory=org.apache.hadoop.fs.s3a.S3ADataBlocks$DiskBlockFactory@2c5b7418, executor=BlockingThreadPoolExecutorService{SemaphoredDelegatingExecutor{permitCount=30, available=30, waiting=0}, activeCount=0}, statistics {0 bytes read, 0 bytes written, 0 read ops, 0 large read ops, 0 write ops}, metrics {{Context=S3AFileSystem} {FileSystemId=b86cdd0f-4b13-4a66-b368-6eb2624d62b0-hwdev-steve-ireland-new} {fsURI=s3a://hwdev-steve-ireland-new/} {files_created=0} {files_copied=0} {files_copied_bytes=0} {files_deleted=0} {fake_directories_deleted=0} {directories_created=0} {directories_deleted=0} {ignored_errors=0} {op_copy_from_local_file=0} {op_exists=0} {op_get_file_status=0} {op_glob_status=0} {op_is_directory=0} {op_is_file=0} {op_list_files=0} {op_list_located_status=0} {op_list_status=0} {op_mkdirs=0} {op_rename=0} {object_copy_requests=0} {object_delete_requests=0} {object_list_requests=0} {object_continue_list_requests=0} {object_metadata_requests=0} {object_multipart_aborted=0} {object_put_bytes=0} {object_put_requests=0} {object_put_requests_completed=0} {stream_write_failures=0} {stream_write_block_uploads=0} {stream_write_block_uploads_committed=0} {stream_write_block_uploads_aborted=0} {stream_write_total_time=0} {stream_write_total_data=0} {object_put_requests_active=0} {object_put_bytes_pending=0} {stream_write_block_uploads_active=0} {stream_write_block_uploads_pending=0} {stream_write_block_uploads_data_pending=0} {stream_read_fully_operations=0} {stream_opened=0} {stream_bytes_skipped_on_seek=0} {stream_closed=0} {stream_bytes_backwards_on_seek=0} {stream_bytes_read=0} {stream_read_operations_incomplete=0} {stream_bytes_discarded_in_abort=0} {stream_close_operations=0} {stream_read_operations=0} {stream_aborted=0} {stream_forward_seek_operations=0} {stream_backward_seek_operations=0} {stream_seek_operations=0} {stream_bytes_read_in_close=0} {stream_read_exceptions=0} }}
          2016-12-07 15:14:08,423 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  s3a.AbstractS3ATestBase (AbstractS3ATestBase.java:describe(87)) - 
          
          testTimeToOpenAndReadWholeFileBlocks: Open the test file s3a://landsat-pds/scene_list.gz and read it in blocks of size 1048576
          
          2016-12-07 15:15:12,083 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 0 in 121 reads: 63,467,576,563 nS
          2016-12-07 15:16:11,111 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 1 in 121 reads: 59,028,100,447 nS
          2016-12-07 15:17:37,074 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 2 in 121 reads: 85,963,412,735 nS
          2016-12-07 15:19:25,300 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 3 in 122 reads: 108,226,732,007 nS
          
          Show
          stevel@apache.org Steve Loughran added a comment - 2016-12-07 15:14:06,958 [ Thread -1] INFO contract.AbstractFSContractTestBase (AbstractFSContractTestBase.java:setup(184)) - Test filesystem = s3a: //hwdev-steve-ireland- new implemented by S3AFileSystem{uri=s3a://hwdev-steve-ireland- new , workingDir=s3a://hwdev-steve-ireland- new /user/stevel, inputPolicy=normal, partSize=8000000, enableMultiObjectsDelete= true , maxKeys=5000, readAhead=65536, blockSize=33554432, multiPartThreshold=2147483647, blockFactory=org.apache.hadoop.fs.s3a.S3ADataBlocks$DiskBlockFactory@2c5b7418, executor=BlockingThreadPoolExecutorService{SemaphoredDelegatingExecutor{permitCount=30, available=30, waiting=0}, activeCount=0}, statistics {0 bytes read, 0 bytes written, 0 read ops, 0 large read ops, 0 write ops}, metrics {{Context=S3AFileSystem} {FileSystemId=b86cdd0f-4b13-4a66-b368-6eb2624d62b0-hwdev-steve-ireland- new } {fsURI=s3a://hwdev-steve-ireland- new /} {files_created=0} {files_copied=0} {files_copied_bytes=0} {files_deleted=0} {fake_directories_deleted=0} {directories_created=0} {directories_deleted=0} {ignored_errors=0} {op_copy_from_local_file=0} {op_exists=0} {op_get_file_status=0} {op_glob_status=0} {op_is_directory=0} {op_is_file=0} {op_list_files=0} {op_list_located_status=0} {op_list_status=0} {op_mkdirs=0} {op_rename=0} {object_copy_requests=0} {object_delete_requests=0} {object_list_requests=0} {object_continue_list_requests=0} {object_metadata_requests=0} {object_multipart_aborted=0} {object_put_bytes=0} {object_put_requests=0} {object_put_requests_completed=0} {stream_write_failures=0} {stream_write_block_uploads=0} {stream_write_block_uploads_committed=0} {stream_write_block_uploads_aborted=0} {stream_write_total_time=0} {stream_write_total_data=0} {object_put_requests_active=0} {object_put_bytes_pending=0} {stream_write_block_uploads_active=0} {stream_write_block_uploads_pending=0} {stream_write_block_uploads_data_pending=0} {stream_read_fully_operations=0} {stream_opened=0} {stream_bytes_skipped_on_seek=0} {stream_closed=0} {stream_bytes_backwards_on_seek=0} {stream_bytes_read=0} {stream_read_operations_incomplete=0} {stream_bytes_discarded_in_abort=0} {stream_close_operations=0} {stream_read_operations=0} {stream_aborted=0} {stream_forward_seek_operations=0} {stream_backward_seek_operations=0} {stream_seek_operations=0} {stream_bytes_read_in_close=0} {stream_read_exceptions=0} }} 2016-12-07 15:14:08,423 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO s3a.AbstractS3ATestBase (AbstractS3ATestBase.java:describe(87)) - testTimeToOpenAndReadWholeFileBlocks: Open the test file s3a: //landsat-pds/scene_list.gz and read it in blocks of size 1048576 2016-12-07 15:15:12,083 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 0 in 121 reads: 63,467,576,563 nS 2016-12-07 15:16:11,111 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 1 in 121 reads: 59,028,100,447 nS 2016-12-07 15:17:37,074 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 2 in 121 reads: 85,963,412,735 nS 2016-12-07 15:19:25,300 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 3 in 122 reads: 108,226,732,007 nS
          Hide
          stevel@apache.org Steve Loughran added a comment -

          stack

          
          "JUnit-testTimeToOpenAndReadWholeFileBlocks" #11 prio=5 os_prio=31 tid=0x00007fccd7ba3800 nid=0x5703 runnable [0x000070000c4dd000]
             java.lang.Thread.State: RUNNABLE
          	at java.net.SocketInputStream.socketRead0(Native Method)
          	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
          	at java.net.SocketInputStream.read(SocketInputStream.java:170)
          	at java.net.SocketInputStream.read(SocketInputStream.java:141)
          	at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
          	at sun.security.ssl.InputRecord.readV3Record(InputRecord.java:593)
          	at sun.security.ssl.InputRecord.read(InputRecord.java:532)
          	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
          	- locked <0x000000076c2764f8> (a java.lang.Object)
          	at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:930)
          	at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
          	- locked <0x000000076c2942a8> (a sun.security.ssl.AppInputStream)
          	at org.apache.http.impl.io.AbstractSessionInputBuffer.read(AbstractSessionInputBuffer.java:198)
          	at org.apache.http.impl.io.ContentLengthInputStream.read(ContentLengthInputStream.java:178)
          	at org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:137)
          	at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:72)
          	at com.amazonaws.event.ProgressInputStream.read(ProgressInputStream.java:151)
          	at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:72)
          	at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:72)
          	at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:72)
          	at com.amazonaws.event.ProgressInputStream.read(ProgressInputStream.java:151)
          	at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:72)
          	at com.amazonaws.util.LengthCheckInputStream.read(LengthCheckInputStream.java:108)
          	at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:72)
          	at org.apache.hadoop.fs.s3a.S3AInputStream.read(S3AInputStream.java:373)
          	- locked <0x000000076c3e24f8> (a org.apache.hadoop.fs.s3a.S3AInputStream)
          	at java.io.DataInputStream.read(DataInputStream.java:149)
          	at org.apache.hadoop.fs.s3a.scale.ITestS3AInputStreamPerformance.testTimeToOpenAndReadWholeFileBlocks(ITestS3AInputStreamPerformance.java:225)
          	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          	at java.lang.reflect.Method.invoke(Method.java:498)
          	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
          	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
          	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
          	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
          	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
          	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
          	at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
          	at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
          
          Show
          stevel@apache.org Steve Loughran added a comment - stack "JUnit-testTimeToOpenAndReadWholeFileBlocks" #11 prio=5 os_prio=31 tid=0x00007fccd7ba3800 nid=0x5703 runnable [0x000070000c4dd000] java.lang. Thread .State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) at java.net.SocketInputStream.read(SocketInputStream.java:170) at java.net.SocketInputStream.read(SocketInputStream.java:141) at sun.security.ssl.InputRecord.readFully(InputRecord.java:465) at sun.security.ssl.InputRecord.readV3Record(InputRecord.java:593) at sun.security.ssl.InputRecord.read(InputRecord.java:532) at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973) - locked <0x000000076c2764f8> (a java.lang. Object ) at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:930) at sun.security.ssl.AppInputStream.read(AppInputStream.java:105) - locked <0x000000076c2942a8> (a sun.security.ssl.AppInputStream) at org.apache.http.impl.io.AbstractSessionInputBuffer.read(AbstractSessionInputBuffer.java:198) at org.apache.http.impl.io.ContentLengthInputStream.read(ContentLengthInputStream.java:178) at org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:137) at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:72) at com.amazonaws.event.ProgressInputStream.read(ProgressInputStream.java:151) at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:72) at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:72) at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:72) at com.amazonaws.event.ProgressInputStream.read(ProgressInputStream.java:151) at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:72) at com.amazonaws.util.LengthCheckInputStream.read(LengthCheckInputStream.java:108) at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:72) at org.apache.hadoop.fs.s3a.S3AInputStream.read(S3AInputStream.java:373) - locked <0x000000076c3e24f8> (a org.apache.hadoop.fs.s3a.S3AInputStream) at java.io.DataInputStream.read(DataInputStream.java:149) at org.apache.hadoop.fs.s3a.scale.ITestS3AInputStreamPerformance.testTimeToOpenAndReadWholeFileBlocks(ITestS3AInputStreamPerformance.java:225) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
          Hide
          stevel@apache.org Steve Loughran added a comment -

          updating AWS jackson and SDK makes this problem go away: -Djackson2.version=2.7.8 -Daws-java-sdk.version=1.11.45

          2016-12-07 15:29:42,315 [Thread-1] INFO  contract.AbstractFSContractTestBase (AbstractFSContractTestBase.java:setup(184)) - Test filesystem = s3a://hwdev-steve-ireland-new implemented by S3AFileSystem{uri=s3a://hwdev-steve-ireland-new, workingDir=s3a://hwdev-steve-ireland-new/user/stevel, inputPolicy=normal, partSize=8000000, enableMultiObjectsDelete=true, maxKeys=5000, readAhead=65536, blockSize=33554432, multiPartThreshold=2147483647, blockFactory=org.apache.hadoop.fs.s3a.S3ADataBlocks$DiskBlockFactory@53488388, executor=BlockingThreadPoolExecutorService{SemaphoredDelegatingExecutor{permitCount=30, available=30, waiting=0}, activeCount=0}, statistics {0 bytes read, 0 bytes written, 0 read ops, 0 large read ops, 0 write ops}, metrics {{Context=S3AFileSystem} {FileSystemId=9adef667-fa4c-407a-b41f-bdeae12f1b8a-hwdev-steve-ireland-new} {fsURI=s3a://hwdev-steve-ireland-new/} {files_created=0} {files_copied=0} {files_copied_bytes=0} {files_deleted=0} {fake_directories_deleted=0} {directories_created=0} {directories_deleted=0} {ignored_errors=0} {op_copy_from_local_file=0} {op_exists=0} {op_get_file_status=0} {op_glob_status=0} {op_is_directory=0} {op_is_file=0} {op_list_files=0} {op_list_located_status=0} {op_list_status=0} {op_mkdirs=0} {op_rename=0} {object_copy_requests=0} {object_delete_requests=0} {object_list_requests=0} {object_continue_list_requests=0} {object_metadata_requests=0} {object_multipart_aborted=0} {object_put_bytes=0} {object_put_requests=0} {object_put_requests_completed=0} {stream_write_failures=0} {stream_write_block_uploads=0} {stream_write_block_uploads_committed=0} {stream_write_block_uploads_aborted=0} {stream_write_total_time=0} {stream_write_total_data=0} {object_put_requests_active=0} {object_put_bytes_pending=0} {stream_write_block_uploads_active=0} {stream_write_block_uploads_pending=0} {stream_write_block_uploads_data_pending=0} {stream_read_fully_operations=0} {stream_opened=0} {stream_bytes_skipped_on_seek=0} {stream_closed=0} {stream_bytes_backwards_on_seek=0} {stream_bytes_read=0} {stream_read_operations_incomplete=0} {stream_bytes_discarded_in_abort=0} {stream_close_operations=0} {stream_read_operations=0} {stream_aborted=0} {stream_forward_seek_operations=0} {stream_backward_seek_operations=0} {stream_seek_operations=0} {stream_bytes_read_in_close=0} {stream_read_exceptions=0} }}
          2016-12-07 15:29:44,321 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  s3a.AbstractS3ATestBase (AbstractS3ATestBase.java:describe(87)) - 
          
          testTimeToOpenAndReadWholeFileBlocks: Open the test file s3a://landsat-pds/scene_list.gz and read it in blocks of size 1048576
          
          2016-12-07 15:29:46,091 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 0 in 121 reads: 1,515,218,115 nS
          2016-12-07 15:29:46,541 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 1 in 121 reads: 449,303,312 nS
          2016-12-07 15:29:46,810 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 2 in 121 reads: 269,565,006 nS
          2016-12-07 15:29:47,065 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 3 in 122 reads: 254,014,989 nS
          2016-12-07 15:29:47,269 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 4 in 96 reads: 203,917,979 nS
          2016-12-07 15:29:47,453 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 5 in 69 reads: 183,067,748 nS
          2016-12-07 15:29:47,626 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 6 in 69 reads: 173,406,636 nS
          2016-12-07 15:29:47,788 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 7 in 69 reads: 160,965,161 nS
          2016-12-07 15:29:47,952 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 8 in 69 reads: 163,948,695 nS
          2016-12-07 15:29:48,114 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 9 in 69 reads: 161,570,313 nS
          2016-12-07 15:29:48,277 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 10 in 69 reads: 162,656,405 nS
          2016-12-07 15:29:48,441 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 11 in 69 reads: 163,710,666 nS
          2016-12-07 15:29:48,603 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 12 in 69 reads: 161,123,556 nS
          2016-12-07 15:29:48,775 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 13 in 69 reads: 172,327,505 nS
          2016-12-07 15:29:48,938 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 14 in 69 reads: 161,634,225 nS
          2016-12-07 15:29:49,102 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 15 in 69 reads: 163,570,995 nS
          2016-12-07 15:29:49,268 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 16 in 69 reads: 165,127,762 nS
          2016-12-07 15:29:49,432 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 17 in 69 reads: 163,958,777 nS
          2016-12-07 15:29:49,592 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 18 in 69 reads: 159,689,844 nS
          2016-12-07 15:29:49,754 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 19 in 69 reads: 161,307,026 nS
          2016-12-07 15:29:49,916 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 20 in 69 reads: 161,726,575 nS
          2016-12-07 15:29:50,080 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 21 in 69 reads: 163,586,849 nS
          2016-12-07 15:29:50,242 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 22 in 69 reads: 161,619,820 nS
          2016-12-07 15:29:50,402 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 23 in 69 reads: 159,661,876 nS
          2016-12-07 15:29:50,566 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 24 in 69 reads: 163,744,412 nS
          2016-12-07 15:29:50,733 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 25 in 69 reads: 165,948,045 nS
          2016-12-07 15:29:50,893 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 26 in 69 reads: 160,269,836 nS
          2016-12-07 15:29:51,055 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 27 in 69 reads: 161,523,271 nS
          2016-12-07 15:29:51,216 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 28 in 69 reads: 160,873,600 nS
          2016-12-07 15:29:51,380 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 29 in 69 reads: 163,739,121 nS
          2016-12-07 15:29:51,544 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 30 in 69 reads: 163,212,814 nS
          2016-12-07 15:29:51,705 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 31 in 69 reads: 160,553,601 nS
          2016-12-07 15:29:51,887 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 32 in 69 reads: 181,887,655 nS
          2016-12-07 15:29:53,611 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 33 in 69 reads: 1,723,811,824 nS
          2016-12-07 15:29:53,872 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 34 in 69 reads: 260,553,127 nS
          2016-12-07 15:29:54,131 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 35 in 69 reads: 258,802,994 nS
          2016-12-07 15:29:54,328 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 36 in 69 reads: 196,090,847 nS
          2016-12-07 15:29:54,328 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Time to read 38820130 bytes in 37 blocks: 9,754,980,049 nS
          2016-12-07 15:29:54,329 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:bandwidth(963)) - Bandwidth = 3.792934  MB/S
          2016-12-07 15:29:54,329 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  scale.ITestS3AInputStreamPerformance (ITestS3AInputStreamPerformance.java:logStreamStatistics(324)) - Stream Statistics
          StreamStatistics{OpenOperations=1, CloseOperations=0, Closed=0, Aborted=0, SeekOperations=0, ReadExceptions=0, ForwardSeekOperations=0, BackwardSeekOperations=0, BytesSkippedOnSeek=0, BytesBackwardsOnSeek=0, BytesRead=38797312, BytesRead excluding skipped=38797312, ReadOperations=2789, ReadFullyOperations=0, ReadsIncomplete=2752, BytesReadInClose=0, BytesDiscardedInAbort=0}
          2016-12-07 15:29:54,329 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.AbstractFSContractTestBase (AbstractFSContractTestBase.java:describe(255)) - cleanup
          2016-12-07 15:29:54,602 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.AbstractFSContractTestBase (AbstractFSContractTestBase.java:describe(255)) - closing file system
          
          Show
          stevel@apache.org Steve Loughran added a comment - updating AWS jackson and SDK makes this problem go away: -Djackson2.version=2.7.8 -Daws-java-sdk.version=1.11.45 2016-12-07 15:29:42,315 [ Thread -1] INFO contract.AbstractFSContractTestBase (AbstractFSContractTestBase.java:setup(184)) - Test filesystem = s3a: //hwdev-steve-ireland- new implemented by S3AFileSystem{uri=s3a://hwdev-steve-ireland- new , workingDir=s3a://hwdev-steve-ireland- new /user/stevel, inputPolicy=normal, partSize=8000000, enableMultiObjectsDelete= true , maxKeys=5000, readAhead=65536, blockSize=33554432, multiPartThreshold=2147483647, blockFactory=org.apache.hadoop.fs.s3a.S3ADataBlocks$DiskBlockFactory@53488388, executor=BlockingThreadPoolExecutorService{SemaphoredDelegatingExecutor{permitCount=30, available=30, waiting=0}, activeCount=0}, statistics {0 bytes read, 0 bytes written, 0 read ops, 0 large read ops, 0 write ops}, metrics {{Context=S3AFileSystem} {FileSystemId=9adef667-fa4c-407a-b41f-bdeae12f1b8a-hwdev-steve-ireland- new } {fsURI=s3a://hwdev-steve-ireland- new /} {files_created=0} {files_copied=0} {files_copied_bytes=0} {files_deleted=0} {fake_directories_deleted=0} {directories_created=0} {directories_deleted=0} {ignored_errors=0} {op_copy_from_local_file=0} {op_exists=0} {op_get_file_status=0} {op_glob_status=0} {op_is_directory=0} {op_is_file=0} {op_list_files=0} {op_list_located_status=0} {op_list_status=0} {op_mkdirs=0} {op_rename=0} {object_copy_requests=0} {object_delete_requests=0} {object_list_requests=0} {object_continue_list_requests=0} {object_metadata_requests=0} {object_multipart_aborted=0} {object_put_bytes=0} {object_put_requests=0} {object_put_requests_completed=0} {stream_write_failures=0} {stream_write_block_uploads=0} {stream_write_block_uploads_committed=0} {stream_write_block_uploads_aborted=0} {stream_write_total_time=0} {stream_write_total_data=0} {object_put_requests_active=0} {object_put_bytes_pending=0} {stream_write_block_uploads_active=0} {stream_write_block_uploads_pending=0} {stream_write_block_uploads_data_pending=0} {stream_read_fully_operations=0} {stream_opened=0} {stream_bytes_skipped_on_seek=0} {stream_closed=0} {stream_bytes_backwards_on_seek=0} {stream_bytes_read=0} {stream_read_operations_incomplete=0} {stream_bytes_discarded_in_abort=0} {stream_close_operations=0} {stream_read_operations=0} {stream_aborted=0} {stream_forward_seek_operations=0} {stream_backward_seek_operations=0} {stream_seek_operations=0} {stream_bytes_read_in_close=0} {stream_read_exceptions=0} }} 2016-12-07 15:29:44,321 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO s3a.AbstractS3ATestBase (AbstractS3ATestBase.java:describe(87)) - testTimeToOpenAndReadWholeFileBlocks: Open the test file s3a: //landsat-pds/scene_list.gz and read it in blocks of size 1048576 2016-12-07 15:29:46,091 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 0 in 121 reads: 1,515,218,115 nS 2016-12-07 15:29:46,541 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 1 in 121 reads: 449,303,312 nS 2016-12-07 15:29:46,810 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 2 in 121 reads: 269,565,006 nS 2016-12-07 15:29:47,065 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 3 in 122 reads: 254,014,989 nS 2016-12-07 15:29:47,269 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 4 in 96 reads: 203,917,979 nS 2016-12-07 15:29:47,453 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 5 in 69 reads: 183,067,748 nS 2016-12-07 15:29:47,626 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 6 in 69 reads: 173,406,636 nS 2016-12-07 15:29:47,788 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 7 in 69 reads: 160,965,161 nS 2016-12-07 15:29:47,952 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 8 in 69 reads: 163,948,695 nS 2016-12-07 15:29:48,114 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 9 in 69 reads: 161,570,313 nS 2016-12-07 15:29:48,277 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 10 in 69 reads: 162,656,405 nS 2016-12-07 15:29:48,441 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 11 in 69 reads: 163,710,666 nS 2016-12-07 15:29:48,603 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 12 in 69 reads: 161,123,556 nS 2016-12-07 15:29:48,775 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 13 in 69 reads: 172,327,505 nS 2016-12-07 15:29:48,938 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 14 in 69 reads: 161,634,225 nS 2016-12-07 15:29:49,102 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 15 in 69 reads: 163,570,995 nS 2016-12-07 15:29:49,268 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 16 in 69 reads: 165,127,762 nS 2016-12-07 15:29:49,432 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 17 in 69 reads: 163,958,777 nS 2016-12-07 15:29:49,592 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 18 in 69 reads: 159,689,844 nS 2016-12-07 15:29:49,754 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 19 in 69 reads: 161,307,026 nS 2016-12-07 15:29:49,916 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 20 in 69 reads: 161,726,575 nS 2016-12-07 15:29:50,080 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 21 in 69 reads: 163,586,849 nS 2016-12-07 15:29:50,242 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 22 in 69 reads: 161,619,820 nS 2016-12-07 15:29:50,402 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 23 in 69 reads: 159,661,876 nS 2016-12-07 15:29:50,566 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 24 in 69 reads: 163,744,412 nS 2016-12-07 15:29:50,733 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 25 in 69 reads: 165,948,045 nS 2016-12-07 15:29:50,893 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 26 in 69 reads: 160,269,836 nS 2016-12-07 15:29:51,055 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 27 in 69 reads: 161,523,271 nS 2016-12-07 15:29:51,216 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 28 in 69 reads: 160,873,600 nS 2016-12-07 15:29:51,380 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 29 in 69 reads: 163,739,121 nS 2016-12-07 15:29:51,544 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 30 in 69 reads: 163,212,814 nS 2016-12-07 15:29:51,705 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 31 in 69 reads: 160,553,601 nS 2016-12-07 15:29:51,887 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 32 in 69 reads: 181,887,655 nS 2016-12-07 15:29:53,611 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 33 in 69 reads: 1,723,811,824 nS 2016-12-07 15:29:53,872 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 34 in 69 reads: 260,553,127 nS 2016-12-07 15:29:54,131 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 35 in 69 reads: 258,802,994 nS 2016-12-07 15:29:54,328 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 36 in 69 reads: 196,090,847 nS 2016-12-07 15:29:54,328 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Time to read 38820130 bytes in 37 blocks: 9,754,980,049 nS 2016-12-07 15:29:54,329 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:bandwidth(963)) - Bandwidth = 3.792934 MB/S 2016-12-07 15:29:54,329 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO scale.ITestS3AInputStreamPerformance (ITestS3AInputStreamPerformance.java:logStreamStatistics(324)) - Stream Statistics StreamStatistics{OpenOperations=1, CloseOperations=0, Closed=0, Aborted=0, SeekOperations=0, ReadExceptions=0, ForwardSeekOperations=0, BackwardSeekOperations=0, BytesSkippedOnSeek=0, BytesBackwardsOnSeek=0, BytesRead=38797312, BytesRead excluding skipped=38797312, ReadOperations=2789, ReadFullyOperations=0, ReadsIncomplete=2752, BytesReadInClose=0, BytesDiscardedInAbort=0} 2016-12-07 15:29:54,329 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.AbstractFSContractTestBase (AbstractFSContractTestBase.java:describe(255)) - cleanup 2016-12-07 15:29:54,602 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.AbstractFSContractTestBase (AbstractFSContractTestBase.java:describe(255)) - closing file system
          Hide
          stevel@apache.org Steve Loughran added a comment - - edited

          rerun on 2.8, logs this time show a SocketTimeout -> restart and new https link, with the next attempt having the same numbers as the new SDK

          12-07 16:09:18,492 [Thread-1] INFO  contract.AbstractFSContractTestBase (AbstractFSContractTestBase.java:setup(184)) - Test filesystem = s3a://hwdev-steve-ireland-new implemented by S3AFileSystem{uri=s3a://hwdev-steve-ireland-new, workingDir=s3a://hwdev-steve-ireland-new/user/stevel, inputPolicy=normal, partSize=8000000, enableMultiObjectsDelete=true, maxKeys=5000, readAhead=65536, blockSize=33554432, multiPartThreshold=2147483647, blockFactory=org.apache.hadoop.fs.s3a.S3ADataBlocks$DiskBlockFactory@2c5b7418, executor=BlockingThreadPoolExecutorService{SemaphoredDelegatingExecutor{permitCount=30, available=30, waiting=0}, activeCount=0}, statistics {0 bytes read, 0 bytes written, 0 read ops, 0 large read ops, 0 write ops}, metrics {{Context=S3AFileSystem} {FileSystemId=be5a5251-b83f-441b-b43c-774586f7bbde-hwdev-steve-ireland-new} {fsURI=s3a://hwdev-steve-ireland-new/} {files_created=0} {files_copied=0} {files_copied_bytes=0} {files_deleted=0} {fake_directories_deleted=0} {directories_created=0} {directories_deleted=0} {ignored_errors=0} {op_copy_from_local_file=0} {op_exists=0} {op_get_file_status=0} {op_glob_status=0} {op_is_directory=0} {op_is_file=0} {op_list_files=0} {op_list_located_status=0} {op_list_status=0} {op_mkdirs=0} {op_rename=0} {object_copy_requests=0} {object_delete_requests=0} {object_list_requests=0} {object_continue_list_requests=0} {object_metadata_requests=0} {object_multipart_aborted=0} {object_put_bytes=0} {object_put_requests=0} {object_put_requests_completed=0} {stream_write_failures=0} {stream_write_block_uploads=0} {stream_write_block_uploads_committed=0} {stream_write_block_uploads_aborted=0} {stream_write_total_time=0} {stream_write_total_data=0} {object_put_requests_active=0} {object_put_bytes_pending=0} {stream_write_block_uploads_active=0} {stream_write_block_uploads_pending=0} {stream_write_block_uploads_data_pending=0} {stream_read_fully_operations=0} {stream_opened=0} {stream_bytes_skipped_on_seek=0} {stream_closed=0} {stream_bytes_backwards_on_seek=0} {stream_bytes_read=0} {stream_read_operations_incomplete=0} {stream_bytes_discarded_in_abort=0} {stream_close_operations=0} {stream_read_operations=0} {stream_aborted=0} {stream_forward_seek_operations=0} {stream_backward_seek_operations=0} {stream_seek_operations=0} {stream_bytes_read_in_close=0} {stream_read_exceptions=0} }}
          2016-12-07 16:09:20,294 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  s3a.AbstractS3ATestBase (AbstractS3ATestBase.java:describe(87)) - 
          
          testTimeToOpenAndReadWholeFileBlocks: Open the test file s3a://landsat-pds/scene_list.gz and read it in blocks of size 1048576
          
          2016-12-07 16:10:13,635 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 0 in 121 reads: 53,171,733,017 nS
          2016-12-07 16:11:11,834 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 1 in 121 reads: 58,198,581,698 nS
          2016-12-07 16:12:07,894 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 2 in 121 reads: 56,060,315,378 nS
          2016-12-07 16:13:09,745 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 3 in 122 reads: 61,851,755,314 nS
          2016-12-07 16:14:00,233 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 4 in 122 reads: 50,487,446,260 nS
          2016-12-07 16:15:05,391 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 5 in 121 reads: 65,158,241,571 nS
          2016-12-07 16:16:03,961 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 6 in 121 reads: 58,570,460,343 nS
          2016-12-07 16:17:05,090 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 7 in 121 reads: 61,129,325,214 nS
          2016-12-07 16:17:58,719 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 8 in 123 reads: 53,628,801,572 nS
          2016-12-07 16:19:36,115 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 9 in 121 reads: 97,396,659,275 nS
          2016-12-07 16:24:23,031 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  s3a.S3AFileSystem (S3AInputStream.java:onReadFailure(334)) - Got exception while trying to read from stream s3a://landsat-pds/scene_list.gz trying to recover: java.net.SocketTimeoutException: Read timed out
          2016-12-07 16:24:24,569 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 10 in 120 reads: 288,457,023,727 nS
          2016-12-07 16:24:25,463 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 11 in 121 reads: 892,929,231 nS
          2016-12-07 16:24:25,965 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 12 in 123 reads: 501,503,328 nS
          2016-12-07 16:24:26,308 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 13 in 121 reads: 341,840,365 nS
          2016-12-07 16:24:26,556 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 14 in 121 reads: 247,359,257 nS
          2016-12-07 16:24:26,794 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 15 in 86 reads: 238,286,855 nS
          2016-12-07 16:24:27,001 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 16 in 69 reads: 206,087,428 nS
          2016-12-07 16:24:27,190 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 17 in 69 reads: 188,955,262 nS
          2016-12-07 16:24:27,385 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 18 in 69 reads: 194,649,152 nS
          2016-12-07 16:24:27,563 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 19 in 69 reads: 177,754,120 nS
          2016-12-07 16:24:27,738 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 20 in 69 reads: 173,799,688 nS
          2016-12-07 16:24:27,913 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 21 in 69 reads: 175,191,414 nS
          2016-12-07 16:24:28,087 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 22 in 69 reads: 173,933,288 nS
          2016-12-07 16:24:28,264 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 23 in 69 reads: 175,654,086 nS
          2016-12-07 16:24:28,437 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 24 in 69 reads: 173,505,198 nS
          2016-12-07 16:24:28,612 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 25 in 69 reads: 173,746,842 nS
          2016-12-07 16:24:28,785 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 26 in 69 reads: 173,089,830 nS
          2016-12-07 16:24:28,958 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 27 in 69 reads: 172,631,269 nS
          2016-12-07 16:24:29,133 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 28 in 69 reads: 174,324,505 nS
          2016-12-07 16:24:29,306 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 29 in 69 reads: 172,987,396 nS
          2016-12-07 16:24:29,482 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 30 in 69 reads: 175,804,772 nS
          2016-12-07 16:24:29,658 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 31 in 69 reads: 174,854,033 nS
          2016-12-07 16:24:29,835 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 32 in 69 reads: 177,342,092 nS
          2016-12-07 16:24:30,010 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 33 in 69 reads: 174,078,949 nS
          2016-12-07 16:24:30,184 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 34 in 69 reads: 173,946,606 nS
          2016-12-07 16:24:30,358 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 35 in 69 reads: 173,489,121 nS
          2016-12-07 16:24:30,533 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 36 in 69 reads: 174,367,264 nS
          2016-12-07 16:24:30,533 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Time to read 38820130 bytes in 37 blocks: 910,083,350,959 nS
          2016-12-07 16:24:30,535 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:bandwidth(963)) - Bandwidth = 0.040656  MB/S
          2016-12-07 16:24:30,535 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  scale.ITestS3AInputStreamPerformance (ITestS3AInputStreamPerformance.java:logStreamStatistics(324)) - Stream Statistics
          StreamStatistics{OpenOperations=2, CloseOperations=1, Closed=0, Aborted=1, SeekOperations=0, ReadExceptions=1, ForwardSeekOperations=0, BackwardSeekOperations=0, BytesSkippedOnSeek=0, BytesBackwardsOnSeek=0, BytesRead=38797312, BytesRead excluding skipped=38797312, ReadOperations=3355, ReadFullyOperations=0, ReadsIncomplete=3318, BytesReadInClose=0, BytesDiscardedInAbort=27401974}
          2016-12-07 16:24:30,535 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.AbstractFSContractTestBase (AbstractFSContractTestBase.java:describe(255)) - cleanup
          2016-12-07 16:24:30,864 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.AbstractFSContractTestBase (AbstractFSContractTestBase.java:describe(255)) - closing file system
          
          Show
          stevel@apache.org Steve Loughran added a comment - - edited rerun on 2.8, logs this time show a SocketTimeout -> restart and new https link, with the next attempt having the same numbers as the new SDK 12-07 16:09:18,492 [ Thread -1] INFO contract.AbstractFSContractTestBase (AbstractFSContractTestBase.java:setup(184)) - Test filesystem = s3a: //hwdev-steve-ireland- new implemented by S3AFileSystem{uri=s3a://hwdev-steve-ireland- new , workingDir=s3a://hwdev-steve-ireland- new /user/stevel, inputPolicy=normal, partSize=8000000, enableMultiObjectsDelete= true , maxKeys=5000, readAhead=65536, blockSize=33554432, multiPartThreshold=2147483647, blockFactory=org.apache.hadoop.fs.s3a.S3ADataBlocks$DiskBlockFactory@2c5b7418, executor=BlockingThreadPoolExecutorService{SemaphoredDelegatingExecutor{permitCount=30, available=30, waiting=0}, activeCount=0}, statistics {0 bytes read, 0 bytes written, 0 read ops, 0 large read ops, 0 write ops}, metrics {{Context=S3AFileSystem} {FileSystemId=be5a5251-b83f-441b-b43c-774586f7bbde-hwdev-steve-ireland- new } {fsURI=s3a://hwdev-steve-ireland- new /} {files_created=0} {files_copied=0} {files_copied_bytes=0} {files_deleted=0} {fake_directories_deleted=0} {directories_created=0} {directories_deleted=0} {ignored_errors=0} {op_copy_from_local_file=0} {op_exists=0} {op_get_file_status=0} {op_glob_status=0} {op_is_directory=0} {op_is_file=0} {op_list_files=0} {op_list_located_status=0} {op_list_status=0} {op_mkdirs=0} {op_rename=0} {object_copy_requests=0} {object_delete_requests=0} {object_list_requests=0} {object_continue_list_requests=0} {object_metadata_requests=0} {object_multipart_aborted=0} {object_put_bytes=0} {object_put_requests=0} {object_put_requests_completed=0} {stream_write_failures=0} {stream_write_block_uploads=0} {stream_write_block_uploads_committed=0} {stream_write_block_uploads_aborted=0} {stream_write_total_time=0} {stream_write_total_data=0} {object_put_requests_active=0} {object_put_bytes_pending=0} {stream_write_block_uploads_active=0} {stream_write_block_uploads_pending=0} {stream_write_block_uploads_data_pending=0} {stream_read_fully_operations=0} {stream_opened=0} {stream_bytes_skipped_on_seek=0} {stream_closed=0} {stream_bytes_backwards_on_seek=0} {stream_bytes_read=0} {stream_read_operations_incomplete=0} {stream_bytes_discarded_in_abort=0} {stream_close_operations=0} {stream_read_operations=0} {stream_aborted=0} {stream_forward_seek_operations=0} {stream_backward_seek_operations=0} {stream_seek_operations=0} {stream_bytes_read_in_close=0} {stream_read_exceptions=0} }} 2016-12-07 16:09:20,294 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO s3a.AbstractS3ATestBase (AbstractS3ATestBase.java:describe(87)) - testTimeToOpenAndReadWholeFileBlocks: Open the test file s3a: //landsat-pds/scene_list.gz and read it in blocks of size 1048576 2016-12-07 16:10:13,635 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 0 in 121 reads: 53,171,733,017 nS 2016-12-07 16:11:11,834 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 1 in 121 reads: 58,198,581,698 nS 2016-12-07 16:12:07,894 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 2 in 121 reads: 56,060,315,378 nS 2016-12-07 16:13:09,745 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 3 in 122 reads: 61,851,755,314 nS 2016-12-07 16:14:00,233 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 4 in 122 reads: 50,487,446,260 nS 2016-12-07 16:15:05,391 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 5 in 121 reads: 65,158,241,571 nS 2016-12-07 16:16:03,961 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 6 in 121 reads: 58,570,460,343 nS 2016-12-07 16:17:05,090 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 7 in 121 reads: 61,129,325,214 nS 2016-12-07 16:17:58,719 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 8 in 123 reads: 53,628,801,572 nS 2016-12-07 16:19:36,115 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 9 in 121 reads: 97,396,659,275 nS 2016-12-07 16:24:23,031 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO s3a.S3AFileSystem (S3AInputStream.java:onReadFailure(334)) - Got exception while trying to read from stream s3a: //landsat-pds/scene_list.gz trying to recover: java.net.SocketTimeoutException: Read timed out 2016-12-07 16:24:24,569 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 10 in 120 reads: 288,457,023,727 nS 2016-12-07 16:24:25,463 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 11 in 121 reads: 892,929,231 nS 2016-12-07 16:24:25,965 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 12 in 123 reads: 501,503,328 nS 2016-12-07 16:24:26,308 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 13 in 121 reads: 341,840,365 nS 2016-12-07 16:24:26,556 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 14 in 121 reads: 247,359,257 nS 2016-12-07 16:24:26,794 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 15 in 86 reads: 238,286,855 nS 2016-12-07 16:24:27,001 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 16 in 69 reads: 206,087,428 nS 2016-12-07 16:24:27,190 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 17 in 69 reads: 188,955,262 nS 2016-12-07 16:24:27,385 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 18 in 69 reads: 194,649,152 nS 2016-12-07 16:24:27,563 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 19 in 69 reads: 177,754,120 nS 2016-12-07 16:24:27,738 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 20 in 69 reads: 173,799,688 nS 2016-12-07 16:24:27,913 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 21 in 69 reads: 175,191,414 nS 2016-12-07 16:24:28,087 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 22 in 69 reads: 173,933,288 nS 2016-12-07 16:24:28,264 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 23 in 69 reads: 175,654,086 nS 2016-12-07 16:24:28,437 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 24 in 69 reads: 173,505,198 nS 2016-12-07 16:24:28,612 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 25 in 69 reads: 173,746,842 nS 2016-12-07 16:24:28,785 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 26 in 69 reads: 173,089,830 nS 2016-12-07 16:24:28,958 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 27 in 69 reads: 172,631,269 nS 2016-12-07 16:24:29,133 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 28 in 69 reads: 174,324,505 nS 2016-12-07 16:24:29,306 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 29 in 69 reads: 172,987,396 nS 2016-12-07 16:24:29,482 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 30 in 69 reads: 175,804,772 nS 2016-12-07 16:24:29,658 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 31 in 69 reads: 174,854,033 nS 2016-12-07 16:24:29,835 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 32 in 69 reads: 177,342,092 nS 2016-12-07 16:24:30,010 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 33 in 69 reads: 174,078,949 nS 2016-12-07 16:24:30,184 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 34 in 69 reads: 173,946,606 nS 2016-12-07 16:24:30,358 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 35 in 69 reads: 173,489,121 nS 2016-12-07 16:24:30,533 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 36 in 69 reads: 174,367,264 nS 2016-12-07 16:24:30,533 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Time to read 38820130 bytes in 37 blocks: 910,083,350,959 nS 2016-12-07 16:24:30,535 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:bandwidth(963)) - Bandwidth = 0.040656 MB/S 2016-12-07 16:24:30,535 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO scale.ITestS3AInputStreamPerformance (ITestS3AInputStreamPerformance.java:logStreamStatistics(324)) - Stream Statistics StreamStatistics{OpenOperations=2, CloseOperations=1, Closed=0, Aborted=1, SeekOperations=0, ReadExceptions=1, ForwardSeekOperations=0, BackwardSeekOperations=0, BytesSkippedOnSeek=0, BytesBackwardsOnSeek=0, BytesRead=38797312, BytesRead excluding skipped=38797312, ReadOperations=3355, ReadFullyOperations=0, ReadsIncomplete=3318, BytesReadInClose=0, BytesDiscardedInAbort=27401974} 2016-12-07 16:24:30,535 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.AbstractFSContractTestBase (AbstractFSContractTestBase.java:describe(255)) - cleanup 2016-12-07 16:24:30,864 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.AbstractFSContractTestBase (AbstractFSContractTestBase.java:describe(255)) - closing file system
          Hide
          stevel@apache.org Steve Loughran added a comment -

          For reference, curl

          curl -O https://landsat-pds.s3.amazonaws.com/scene_list.gz
            % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                           Dload  Upload   Total   Spent    Left  Speed
          100 37.1M  100 37.1M    0     0  4545k      0  0:00:08  0:00:08 --:--:-- 6303k
          
          Show
          stevel@apache.org Steve Loughran added a comment - For reference, curl curl -O https: //landsat-pds.s3.amazonaws.com/scene_list.gz % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 37.1M 100 37.1M 0 0 4545k 0 0:00:08 0:00:08 --:--:-- 6303k
          Hide
          stevel@apache.org Steve Loughran added a comment -

          adding debug logging in AWS shows a 206 coming back on the GET, but the latest SDK is doing this too

          testTimeToOpenAndReadWholeFileBlocks: Open the test file s3a://landsat-pds/scene_list.gz and read it in blocks of size 1048576
          
          2016-12-07 18:09:20,058 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG amazonaws.request (AmazonHttpClient.java:executeOneRequest(671)) - Sending Request: HEAD https://landsat-pds.s3.amazonaws.com /scene_list.gz Headers: (User-Agent: Hadoop 2.8.0-SNAPSHOT, aws-sdk-java/1.10.6 Mac_OS_X/10.12.1 Java_HotSpot(TM)_64-Bit_Server_VM/25.102-b14/1.8.0_102, Content-Type: application/x-www-form-urlencoded; charset=utf-8, ) 
          2016-12-07 18:09:20,225 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG amazonaws.request (AmazonHttpClient.java:logHeaderRequestId(836)) - x-amzn-RequestId: not available
          2016-12-07 18:09:20,226 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG amazonaws.request (AmazonHttpClient.java:handleResponse(1091)) - Received successful response: 200, AWS Request ID: EC0218D1AF8CF726
          2016-12-07 18:09:20,226 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG amazonaws.request (AmazonHttpClient.java:logResponseRequestId(856)) - AWS Request ID: EC0218D1AF8CF726
          2016-12-07 18:09:20,231 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG amazonaws.request (AmazonHttpClient.java:executeOneRequest(671)) - Sending Request: GET https://landsat-pds.s3.amazonaws.com /scene_list.gz Headers: (User-Agent: Hadoop 2.8.0-SNAPSHOT, aws-sdk-java/1.10.6 Mac_OS_X/10.12.1 Java_HotSpot(TM)_64-Bit_Server_VM/25.102-b14/1.8.0_102, Range: bytes=0-38939342, Content-Type: application/x-www-form-urlencoded; charset=utf-8, ) 
          2016-12-07 18:09:20,403 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG amazonaws.request (AmazonHttpClient.java:logHeaderRequestId(836)) - x-amzn-RequestId: not available
          2016-12-07 18:09:20,405 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG amazonaws.request (AmazonHttpClient.java:handleResponse(1091)) - Received successful response: 206, AWS Request ID: E179D5B84DAFE571
          2016-12-07 18:09:20,405 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG amazonaws.request (AmazonHttpClient.java:logResponseRequestId(856)) - AWS Request ID: E179D5B84DAFE571
          2016-12-07 18:10:42,604 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 0 in 121 reads: 82,365,859,396 nS
          2016-12-07 18:12:26,248 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 1 in 121 reads: 103,644,154,422 nS
          2016-12-07 18:14:09,399 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 2 in 121 reads: 103,151,904,180 nS
          
          Show
          stevel@apache.org Steve Loughran added a comment - adding debug logging in AWS shows a 206 coming back on the GET, but the latest SDK is doing this too testTimeToOpenAndReadWholeFileBlocks: Open the test file s3a: //landsat-pds/scene_list.gz and read it in blocks of size 1048576 2016-12-07 18:09:20,058 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG amazonaws.request (AmazonHttpClient.java:executeOneRequest(671)) - Sending Request: HEAD https: //landsat-pds.s3.amazonaws.com /scene_list.gz Headers: (User-Agent: Hadoop 2.8.0-SNAPSHOT, aws-sdk-java/1.10.6 Mac_OS_X/10.12.1 Java_HotSpot(TM)_64-Bit_Server_VM/25.102-b14/1.8.0_102, Content-Type: application/x-www-form-urlencoded; charset=utf-8, ) 2016-12-07 18:09:20,225 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG amazonaws.request (AmazonHttpClient.java:logHeaderRequestId(836)) - x-amzn-RequestId: not available 2016-12-07 18:09:20,226 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG amazonaws.request (AmazonHttpClient.java:handleResponse(1091)) - Received successful response: 200, AWS Request ID: EC0218D1AF8CF726 2016-12-07 18:09:20,226 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG amazonaws.request (AmazonHttpClient.java:logResponseRequestId(856)) - AWS Request ID: EC0218D1AF8CF726 2016-12-07 18:09:20,231 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG amazonaws.request (AmazonHttpClient.java:executeOneRequest(671)) - Sending Request: GET https: //landsat-pds.s3.amazonaws.com /scene_list.gz Headers: (User-Agent: Hadoop 2.8.0-SNAPSHOT, aws-sdk-java/1.10.6 Mac_OS_X/10.12.1 Java_HotSpot(TM)_64-Bit_Server_VM/25.102-b14/1.8.0_102, Range: bytes=0-38939342, Content-Type: application/x-www-form-urlencoded; charset=utf-8, ) 2016-12-07 18:09:20,403 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG amazonaws.request (AmazonHttpClient.java:logHeaderRequestId(836)) - x-amzn-RequestId: not available 2016-12-07 18:09:20,405 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG amazonaws.request (AmazonHttpClient.java:handleResponse(1091)) - Received successful response: 206, AWS Request ID: E179D5B84DAFE571 2016-12-07 18:09:20,405 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG amazonaws.request (AmazonHttpClient.java:logResponseRequestId(856)) - AWS Request ID: E179D5B84DAFE571 2016-12-07 18:10:42,604 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 0 in 121 reads: 82,365,859,396 nS 2016-12-07 18:12:26,248 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 1 in 121 reads: 103,644,154,422 nS 2016-12-07 18:14:09,399 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 2 in 121 reads: 103,151,904,180 nS
          Hide
          stevel@apache.org Steve Loughran added a comment - - edited

          good news, I've been able to replicate it on the new SDK too, meaning its a network/load balance problem, not SDK related

          2016-12-07 18:42:22,553 [Thread-1] DEBUG s3a.S3AFileSystem (S3ClientFactory.java:initUserAgent(187)) - Using User-Agent: Hadoop 2.8.0-SNAPSHOT
          2016-12-07 18:42:22,913 [Thread-1] DEBUG s3a.AWSCredentialProviderList (AWSCredentialProviderList.java:getCredentials(134)) - Using credentials from BasicAWSCredentialsProvider
          2016-12-07 18:42:22,921 [Thread-1] DEBUG amazonaws.request (AmazonHttpClient.java:executeOneRequest(1046)) - Sending Request: HEAD https://hwdev-steve-ireland-new.s3-us-west-1.amazonaws.com / Headers: (User-Agent: Hadoop 2.8.0-SNAPSHOT, aws-sdk-java/1.11.45 Mac_OS_X/10.12.1 Java_HotSpot(TM)_64-Bit_Server_VM/25.102-b14/1.8.0_102, amz-sdk-invocation-id: 787e1df3-9887-fd11-ec8d-801a7836e20d, Content-Type: application/octet-stream, ) 
          2016-12-07 18:42:23,927 [Thread-1] DEBUG amazonaws.request (AmazonHttpClient.java:handleErrorResponse(1497)) - Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Moved Permanently (Service: null; Status Code: 301; Error Code: 301 Moved Permanently; Request ID: 0D2990D655FE6CDD), S3 Extended Request ID: zam+lWQRapkEc17mnHEoLgeWBuFXGGLncQaxBi/a12Y3uohpoEAzRS1frsGb81AnU1HDJn2b7IU=
          2016-12-07 18:42:23,978 [Thread-1] DEBUG amazonaws.request (AmazonHttpClient.java:executeOneRequest(1046)) - Sending Request: HEAD https://hwdev-steve-ireland-new.s3-eu-west-1.amazonaws.com / Headers: (User-Agent: Hadoop 2.8.0-SNAPSHOT, aws-sdk-java/1.11.45 Mac_OS_X/10.12.1 Java_HotSpot(TM)_64-Bit_Server_VM/25.102-b14/1.8.0_102, amz-sdk-invocation-id: f9fabf3f-9809-ad08-7c2f-1683f6d593cf, Content-Type: application/octet-stream, ) 
          2016-12-07 18:42:24,163 [Thread-1] DEBUG amazonaws.request (AwsResponseHandlerAdapter.java:handle(87)) - Received successful response: 200, AWS Request ID: null
          2016-12-07 18:42:24,164 [Thread-1] DEBUG amazonaws.request (AwsResponseHandlerAdapter.java:logHeaderRequestId(138)) - x-amzn-RequestId: not available
          2016-12-07 18:42:24,164 [Thread-1] DEBUG amazonaws.request (AwsResponseHandlerAdapter.java:logResponseRequestId(158)) - AWS Request ID: not available
          2016-12-07 18:42:24,165 [Thread-1] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:initialize(225)) - Using encryption null
          2016-12-07 18:42:24,166 [Thread-1] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:initialize(238)) - Using S3ABlockOutputStream with buffer = disk; block=8000000; queue limit=4
          2016-12-07 18:42:24,169 [Thread-1] INFO  contract.AbstractFSContractTestBase (AbstractFSContractTestBase.java:setup(184)) - Test filesystem = s3a://hwdev-steve-ireland-new implemented by S3AFileSystem{uri=s3a://hwdev-steve-ireland-new, workingDir=s3a://hwdev-steve-ireland-new/user/stevel, inputPolicy=normal, partSize=8000000, enableMultiObjectsDelete=true, maxKeys=5000, readAhead=65536, blockSize=33554432, multiPartThreshold=2147483647, blockFactory=org.apache.hadoop.fs.s3a.S3ADataBlocks$DiskBlockFactory@51dffc97, executor=BlockingThreadPoolExecutorService{SemaphoredDelegatingExecutor{permitCount=30, available=30, waiting=0}, activeCount=0}, statistics {0 bytes read, 0 bytes written, 0 read ops, 0 large read ops, 0 write ops}, metrics {{Context=S3AFileSystem} {FileSystemId=461d5b68-e18f-4e9b-b88f-ee6cee6196ac-hwdev-steve-ireland-new} {fsURI=s3a://hwdev-steve-ireland-new/} {files_created=0} {files_copied=0} {files_copied_bytes=0} {files_deleted=0} {fake_directories_deleted=0} {directories_created=0} {directories_deleted=0} {ignored_errors=0} {op_copy_from_local_file=0} {op_exists=0} {op_get_file_status=0} {op_glob_status=0} {op_is_directory=0} {op_is_file=0} {op_list_files=0} {op_list_located_status=0} {op_list_status=0} {op_mkdirs=0} {op_rename=0} {object_copy_requests=0} {object_delete_requests=0} {object_list_requests=0} {object_continue_list_requests=0} {object_metadata_requests=0} {object_multipart_aborted=0} {object_put_bytes=0} {object_put_requests=0} {object_put_requests_completed=0} {stream_write_failures=0} {stream_write_block_uploads=0} {stream_write_block_uploads_committed=0} {stream_write_block_uploads_aborted=0} {stream_write_total_time=0} {stream_write_total_data=0} {object_put_requests_active=0} {object_put_bytes_pending=0} {stream_write_block_uploads_active=0} {stream_write_block_uploads_pending=0} {stream_write_block_uploads_data_pending=0} {stream_read_fully_operations=0} {stream_opened=0} {stream_bytes_skipped_on_seek=0} {stream_closed=0} {stream_bytes_backwards_on_seek=0} {stream_bytes_read=0} {stream_read_operations_incomplete=0} {stream_bytes_discarded_in_abort=0} {stream_close_operations=0} {stream_read_operations=0} {stream_aborted=0} {stream_forward_seek_operations=0} {stream_backward_seek_operations=0} {stream_seek_operations=0} {stream_bytes_read_in_close=0} {stream_read_exceptions=0} }}
          2016-12-07 18:42:24,171 [Thread-1] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerMkdirs(1478)) - Making directory: /test
          2016-12-07 18:42:24,171 [Thread-1] DEBUG s3a.S3AFileSystem (S3AStorageStatistics.java:incrementCounter(60)) - op_mkdirs += 1  ->  1
          2016-12-07 18:42:24,171 [Thread-1] DEBUG s3a.S3AFileSystem (S3AStorageStatistics.java:incrementCounter(60)) - op_get_file_status += 1  ->  1
          2016-12-07 18:42:24,172 [Thread-1] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:getFileStatus(1525)) - Getting path status for s3a://hwdev-steve-ireland-new/test  (test)
          2016-12-07 18:42:24,172 [Thread-1] DEBUG s3a.S3AFileSystem (S3AStorageStatistics.java:incrementCounter(60)) - object_metadata_requests += 1  ->  1
          2016-12-07 18:42:24,174 [Thread-1] DEBUG amazonaws.request (AmazonHttpClient.java:executeOneRequest(1046)) - Sending Request: HEAD https://hwdev-steve-ireland-new.s3-eu-west-1.amazonaws.com /test Headers: (User-Agent: Hadoop 2.8.0-SNAPSHOT, aws-sdk-java/1.11.45 Mac_OS_X/10.12.1 Java_HotSpot(TM)_64-Bit_Server_VM/25.102-b14/1.8.0_102, amz-sdk-invocation-id: 57239941-1812-a15f-736a-f3da7727b192, Content-Type: application/octet-stream, ) 
          2016-12-07 18:42:24,225 [Thread-1] DEBUG amazonaws.request (AmazonHttpClient.java:handleErrorResponse(1497)) - Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: null; Status Code: 404; Error Code: 404 Not Found; Request ID: 2B1E6AAB5BB13A50), S3 Extended Request ID: 37X2RRn7nQAt5RDTbO8jazu8ltPGdiUZ+OmvP0cPPtYaaoEGeI/qZ5sv4fEH3xFZOb+d5+wu/w4=
          2016-12-07 18:42:24,225 [Thread-1] DEBUG s3a.S3AFileSystem (S3AStorageStatistics.java:incrementCounter(60)) - object_metadata_requests += 1  ->  2
          2016-12-07 18:42:24,226 [Thread-1] DEBUG amazonaws.request (AmazonHttpClient.java:executeOneRequest(1046)) - Sending Request: HEAD https://hwdev-steve-ireland-new.s3-eu-west-1.amazonaws.com /test/ Headers: (User-Agent: Hadoop 2.8.0-SNAPSHOT, aws-sdk-java/1.11.45 Mac_OS_X/10.12.1 Java_HotSpot(TM)_64-Bit_Server_VM/25.102-b14/1.8.0_102, amz-sdk-invocation-id: 52c578b8-6d85-49e9-d148-2db3079cc239, Content-Type: application/octet-stream, ) 
          2016-12-07 18:42:24,274 [Thread-1] DEBUG amazonaws.request (AmazonHttpClient.java:handleErrorResponse(1497)) - Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: null; Status Code: 404; Error Code: 404 Not Found; Request ID: 4B7CD9CADDF086A7), S3 Extended Request ID: 9AR/mRdKgwAnb8SnWzLSqrM8K/DWFK1utzFqKXZihDtTN9UXPsxGFGN2GoBHgHNTqY41YBBptn8=
          2016-12-07 18:42:24,274 [Thread-1] DEBUG s3a.S3AFileSystem (S3AStorageStatistics.java:incrementCounter(60)) - object_list_requests += 1  ->  1
          2016-12-07 18:42:24,289 [Thread-1] DEBUG amazonaws.request (AmazonHttpClient.java:executeOneRequest(1046)) - Sending Request: GET https://hwdev-steve-ireland-new.s3-eu-west-1.amazonaws.com / Parameters: ({"prefix":["test/"],"delimiter":["/"],"max-keys":["1"],"encoding-type":["url"]}Headers: (User-Agent: Hadoop 2.8.0-SNAPSHOT, aws-sdk-java/1.11.45 Mac_OS_X/10.12.1 Java_HotSpot(TM)_64-Bit_Server_VM/25.102-b14/1.8.0_102, amz-sdk-invocation-id: 5cb39b7e-4fb2-5401-25d5-0fdc15db05ad, Content-Type: application/octet-stream, ) 
          2016-12-07 18:42:24,364 [Thread-1] DEBUG amazonaws.request (AwsResponseHandlerAdapter.java:handle(87)) - Received successful response: 200, AWS Request ID: A39D3AE5D27ABAC9
          2016-12-07 18:42:24,365 [Thread-1] DEBUG amazonaws.request (AwsResponseHandlerAdapter.java:logHeaderRequestId(138)) - x-amzn-RequestId: not available
          2016-12-07 18:42:24,365 [Thread-1] DEBUG amazonaws.request (AwsResponseHandlerAdapter.java:logResponseRequestId(158)) - AWS Request ID: A39D3AE5D27ABAC9
          2016-12-07 18:42:24,365 [Thread-1] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:getFileStatus(1616)) - Not Found: s3a://hwdev-steve-ireland-new/test
          2016-12-07 18:42:24,365 [Thread-1] DEBUG s3a.S3AFileSystem (S3AStorageStatistics.java:incrementCounter(60)) - op_get_file_status += 1  ->  2
          2016-12-07 18:42:24,365 [Thread-1] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:getFileStatus(1525)) - Getting path status for s3a://hwdev-steve-ireland-new/  ()
          2016-12-07 18:42:24,365 [Thread-1] DEBUG s3a.S3AFileSystem (S3AStorageStatistics.java:incrementCounter(60)) - object_list_requests += 1  ->  2
          2016-12-07 18:42:24,366 [Thread-1] DEBUG amazonaws.request (AmazonHttpClient.java:executeOneRequest(1046)) - Sending Request: GET https://hwdev-steve-ireland-new.s3-eu-west-1.amazonaws.com / Parameters: ({"prefix":[""],"delimiter":["/"],"max-keys":["1"],"encoding-type":["url"]}Headers: (User-Agent: Hadoop 2.8.0-SNAPSHOT, aws-sdk-java/1.11.45 Mac_OS_X/10.12.1 Java_HotSpot(TM)_64-Bit_Server_VM/25.102-b14/1.8.0_102, amz-sdk-invocation-id: d68ca495-6dee-8f17-0ff8-008f25a75702, Content-Type: application/octet-stream, ) 
          2016-12-07 18:42:24,426 [Thread-1] DEBUG amazonaws.request (AwsResponseHandlerAdapter.java:handle(87)) - Received successful response: 200, AWS Request ID: 7169BE12EAD059C9
          2016-12-07 18:42:24,426 [Thread-1] DEBUG amazonaws.request (AwsResponseHandlerAdapter.java:logHeaderRequestId(138)) - x-amzn-RequestId: not available
          2016-12-07 18:42:24,426 [Thread-1] DEBUG amazonaws.request (AwsResponseHandlerAdapter.java:logResponseRequestId(158)) - AWS Request ID: 7169BE12EAD059C9
          2016-12-07 18:42:24,427 [Thread-1] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:getFileStatus(1591)) - Found path as directory (with /): 1/0
          2016-12-07 18:42:24,427 [Thread-1] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:getFileStatus(1599)) - Prefix: Users/
          2016-12-07 18:42:24,429 [Thread-1] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:incrementPutStartStatistics(1116)) - PUT start 0 bytes
          2016-12-07 18:42:24,429 [Thread-1] DEBUG s3a.S3AFileSystem (S3AStorageStatistics.java:incrementCounter(60)) - object_put_requests += 1  ->  1
          2016-12-07 18:42:24,437 [Thread-1] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:incrementPutCompletedStatistics(1133)) - PUT completed success=true; 0 bytes
          2016-12-07 18:42:24,437 [Thread-1] DEBUG s3a.S3AFileSystem (S3AStorageStatistics.java:incrementCounter(60)) - object_put_requests_completed += 1  ->  1
          2016-12-07 18:42:24,440 [s3a-transfer-shared-pool1-t1] DEBUG amazonaws.request (AmazonHttpClient.java:executeOneRequest(1046)) - Sending Request: PUT https://hwdev-steve-ireland-new.s3-eu-west-1.amazonaws.com /test/ Headers: (User-Agent: Hadoop 2.8.0-SNAPSHOT, aws-sdk-java/1.11.45 Mac_OS_X/10.12.1 Java_HotSpot(TM)_64-Bit_Server_VM/25.102-b14/1.8.0_102 com.amazonaws.services.s3.transfer.TransferManager/1.11.45, amz-sdk-invocation-id: fad23d78-a104-156a-81ed-f3e29d668cd8, Content-Length: 0, Content-Type: application/octet-stream, ) 
          2016-12-07 18:42:24,509 [s3a-transfer-shared-pool1-t1] DEBUG amazonaws.request (AwsResponseHandlerAdapter.java:handle(87)) - Received successful response: 200, AWS Request ID: 1A25A7F6C78ED591
          2016-12-07 18:42:24,509 [s3a-transfer-shared-pool1-t1] DEBUG amazonaws.request (AwsResponseHandlerAdapter.java:logHeaderRequestId(138)) - x-amzn-RequestId: not available
          2016-12-07 18:42:24,509 [s3a-transfer-shared-pool1-t1] DEBUG amazonaws.request (AwsResponseHandlerAdapter.java:logResponseRequestId(158)) - AWS Request ID: 1A25A7F6C78ED591
          2016-12-07 18:42:24,514 [Thread-1] DEBUG S3AFileSystem.Progress (S3AFileSystem.java:incrementPutProgressStatistics(1151)) - PUT test/: 0 bytes
          2016-12-07 18:42:24,514 [Thread-1] DEBUG scale.S3AScaleTestBase (S3AScaleTestBase.java:setup(94)) - Scale test operation count = 1
          2016-12-07 18:42:24,516 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG s3a.S3AFileSystem (S3ClientFactory.java:initUserAgent(187)) - Using User-Agent: Hadoop 2.8.0-SNAPSHOT
          2016-12-07 18:42:24,519 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG s3a.AWSCredentialProviderList (AWSCredentialProviderList.java:getCredentials(134)) - Using credentials from BasicAWSCredentialsProvider
          2016-12-07 18:42:24,520 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG amazonaws.request (AmazonHttpClient.java:executeOneRequest(1046)) - Sending Request: HEAD https://landsat-pds.s3-us-west-1.amazonaws.com / Headers: (User-Agent: Hadoop 2.8.0-SNAPSHOT, aws-sdk-java/1.11.45 Mac_OS_X/10.12.1 Java_HotSpot(TM)_64-Bit_Server_VM/25.102-b14/1.8.0_102, amz-sdk-invocation-id: 40808c03-5cf7-c960-4c97-353313df657b, Content-Type: application/octet-stream, ) 
          2016-12-07 18:42:25,338 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG amazonaws.request (AmazonHttpClient.java:handleErrorResponse(1497)) - Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Moved Permanently (Service: null; Status Code: 301; Error Code: 301 Moved Permanently; Request ID: 58DECB0934A6428C), S3 Extended Request ID: QQxGx+1eyQEbD/skrLYpaXg+XuW8EvgwtCkpqqTJttl/ONjzD1xC1jVX6IqFCQZdfrSsup85pTM=
          2016-12-07 18:42:25,339 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG amazonaws.request (AmazonHttpClient.java:executeOneRequest(1046)) - Sending Request: HEAD https://landsat-pds.s3-us-west-2.amazonaws.com / Headers: (User-Agent: Hadoop 2.8.0-SNAPSHOT, aws-sdk-java/1.11.45 Mac_OS_X/10.12.1 Java_HotSpot(TM)_64-Bit_Server_VM/25.102-b14/1.8.0_102, amz-sdk-invocation-id: 5eb0baa1-d0f0-1825-27c1-4c122afb0d72, Content-Type: application/octet-stream, ) 
          2016-12-07 18:42:26,225 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG amazonaws.request (AwsResponseHandlerAdapter.java:handle(87)) - Received successful response: 200, AWS Request ID: null
          2016-12-07 18:42:26,225 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG amazonaws.request (AwsResponseHandlerAdapter.java:logHeaderRequestId(138)) - x-amzn-RequestId: not available
          2016-12-07 18:42:26,225 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG amazonaws.request (AwsResponseHandlerAdapter.java:logResponseRequestId(158)) - AWS Request ID: not available
          2016-12-07 18:42:26,226 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:initialize(225)) - Using encryption null
          2016-12-07 18:42:26,226 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:initialize(238)) - Using S3ABlockOutputStream with buffer = disk; block=8000000; queue limit=4
          2016-12-07 18:42:26,226 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG s3a.S3AFileSystem (S3AStorageStatistics.java:incrementCounter(60)) - op_get_file_status += 1  ->  3
          2016-12-07 18:42:26,226 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:getFileStatus(1525)) - Getting path status for s3a://landsat-pds/scene_list.gz  (scene_list.gz)
          2016-12-07 18:42:26,226 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG s3a.S3AFileSystem (S3AStorageStatistics.java:incrementCounter(60)) - object_metadata_requests += 1  ->  3
          2016-12-07 18:42:26,227 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG amazonaws.request (AmazonHttpClient.java:executeOneRequest(1046)) - Sending Request: HEAD https://landsat-pds.s3-us-west-2.amazonaws.com /scene_list.gz Headers: (User-Agent: Hadoop 2.8.0-SNAPSHOT, aws-sdk-java/1.11.45 Mac_OS_X/10.12.1 Java_HotSpot(TM)_64-Bit_Server_VM/25.102-b14/1.8.0_102, amz-sdk-invocation-id: 33d34c3d-ead9-809a-af05-257b905fede7, Content-Type: application/octet-stream, ) 
          2016-12-07 18:42:26,788 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG amazonaws.request (AwsResponseHandlerAdapter.java:handle(87)) - Received successful response: 200, AWS Request ID: C756074918F8BAB1
          2016-12-07 18:42:26,788 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG amazonaws.request (AwsResponseHandlerAdapter.java:logHeaderRequestId(138)) - x-amzn-RequestId: not available
          2016-12-07 18:42:26,788 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG amazonaws.request (AwsResponseHandlerAdapter.java:logResponseRequestId(158)) - AWS Request ID: C756074918F8BAB1
          2016-12-07 18:42:26,788 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:getFileStatus(1534)) - Found exact file: normal file
          2016-12-07 18:42:26,789 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  s3a.AbstractS3ATestBase (AbstractS3ATestBase.java:describe(87)) - 
          
          testTimeToOpenAndReadWholeFileBlocks: Open the test file s3a://landsat-pds/scene_list.gz and read it in blocks of size 1048576
          
          2016-12-07 18:42:26,789 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:setInputPolicy(413)) - Setting input strategy: normal
          2016-12-07 18:42:26,789 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:open(503)) - Opening 's3a://landsat-pds/scene_list.gz' for reading.
          2016-12-07 18:42:26,789 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG s3a.S3AFileSystem (S3AStorageStatistics.java:incrementCounter(60)) - op_get_file_status += 1  ->  4
          2016-12-07 18:42:26,789 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:getFileStatus(1525)) - Getting path status for s3a://landsat-pds/scene_list.gz  (scene_list.gz)
          2016-12-07 18:42:26,789 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG s3a.S3AFileSystem (S3AStorageStatistics.java:incrementCounter(60)) - object_metadata_requests += 1  ->  4
          2016-12-07 18:42:26,790 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG amazonaws.request (AmazonHttpClient.java:executeOneRequest(1046)) - Sending Request: HEAD https://landsat-pds.s3-us-west-2.amazonaws.com /scene_list.gz Headers: (User-Agent: Hadoop 2.8.0-SNAPSHOT, aws-sdk-java/1.11.45 Mac_OS_X/10.12.1 Java_HotSpot(TM)_64-Bit_Server_VM/25.102-b14/1.8.0_102, amz-sdk-invocation-id: 5d20098b-a1cc-a15d-7cf6-959355ceda8c, Content-Type: application/octet-stream, ) 
          2016-12-07 18:42:26,965 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG amazonaws.request (AwsResponseHandlerAdapter.java:handle(87)) - Received successful response: 200, AWS Request ID: 48BD932265D2E504
          2016-12-07 18:42:26,965 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG amazonaws.request (AwsResponseHandlerAdapter.java:logHeaderRequestId(138)) - x-amzn-RequestId: not available
          2016-12-07 18:42:26,965 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG amazonaws.request (AwsResponseHandlerAdapter.java:logResponseRequestId(158)) - AWS Request ID: 48BD932265D2E504
          2016-12-07 18:42:26,965 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:getFileStatus(1534)) - Found exact file: normal file
          2016-12-07 18:42:26,967 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:setInputPolicy(413)) - Setting input strategy: normal
          2016-12-07 18:42:26,967 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG s3a.S3AFileSystem (S3AInputStream.java:reopen(140)) - reopen(s3a://landsat-pds/scene_list.gz) for read from new offset range[0-38939342], length=1048576, streamPosition=0, nextReadPosition=0
          2016-12-07 18:42:26,969 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG amazonaws.request (AmazonHttpClient.java:executeOneRequest(1046)) - Sending Request: GET https://landsat-pds.s3-us-west-2.amazonaws.com /scene_list.gz Headers: (User-Agent: Hadoop 2.8.0-SNAPSHOT, aws-sdk-java/1.11.45 Mac_OS_X/10.12.1 Java_HotSpot(TM)_64-Bit_Server_VM/25.102-b14/1.8.0_102, amz-sdk-invocation-id: 6489ce83-1f56-b833-15dc-f78d8eed429f, Range: bytes=0-38939342, Content-Type: application/octet-stream, ) 
          2016-12-07 18:42:27,168 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG amazonaws.request (AwsResponseHandlerAdapter.java:handle(87)) - Received successful response: 206, AWS Request ID: 7BB5413C293FE587
          2016-12-07 18:42:27,168 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG amazonaws.request (AwsResponseHandlerAdapter.java:logHeaderRequestId(138)) - x-amzn-RequestId: not available
          2016-12-07 18:42:27,168 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG amazonaws.request (AwsResponseHandlerAdapter.java:logResponseRequestId(158)) - AWS Request ID: 7BB5413C293FE587
          2016-12-07 18:43:32,677 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 0 in 121 reads: 65,708,704,300 nS
          2016-12-07 18:44:50,868 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 1 in 121 reads: 78,191,068,156 nS
          2016-12-07 18:48:57,544 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  s3a.S3AFileSystem (S3AInputStream.java:onReadFailure(334)) - Got exception while trying to read from stream s3a://landsat-pds/scene_list.gz trying to recover: java.net.SocketTimeoutException: Read timed out
          2016-12-07 18:48:57,546 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG s3a.S3AFileSystem (S3AInputStream.java:onReadFailure(336)) - While trying to read from stream s3a://landsat-pds/scene_list.gz
          java.net.SocketTimeoutException: Read timed out
          	at java.net.SocketInputStream.socketRead0(Native Method)
          	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
          	at java.net.SocketInputStream.read(SocketInputStream.java:170)
          	at java.net.SocketInputStream.read(SocketInputStream.java:141)
          	at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
          	at sun.security.ssl.InputRecord.readV3Record(InputRecord.java:593)
          	at sun.security.ssl.InputRecord.read(InputRecord.java:532)
          	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
          	at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:930)
          	at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
          	at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139)
          	at org.apache.http.impl.io.SessionInputBufferImpl.read(SessionInputBufferImpl.java:200)
          	at org.apache.http.impl.io.ContentLengthInputStream.read(ContentLengthInputStream.java:178)
          	at org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:137)
          	at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:72)
          	at com.amazonaws.event.ProgressInputStream.read(ProgressInputStream.java:181)
          	at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:72)
          	at com.amazonaws.services.s3.model.S3ObjectInputStream.read(S3ObjectInputStream.java:155)
          	at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:72)
          	at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:72)
          	at com.amazonaws.event.ProgressInputStream.read(ProgressInputStream.java:181)
          	at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:72)
          	at com.amazonaws.util.LengthCheckInputStream.read(LengthCheckInputStream.java:108)
          	at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:72)
          	at com.amazonaws.services.s3.model.S3ObjectInputStream.read(S3ObjectInputStream.java:155)
          	at org.apache.hadoop.fs.s3a.S3AInputStream.read(S3AInputStream.java:373)
          	at java.io.DataInputStream.read(DataInputStream.java:149)
          	at org.apache.hadoop.fs.s3a.scale.ITestS3AInputStreamPerformance.testTimeToOpenAndReadWholeFileBlocks(ITestS3AInputStreamPerformance.java:225)
          	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          	at java.lang.reflect.Method.invoke(Method.java:498)
          	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
          	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
          	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
          	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
          	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
          	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
          	at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
          	at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
          2016-12-07 18:48:57,550 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG s3a.S3AFileSystem (S3AInputStream.java:closeStream(463)) - Stream s3a://landsat-pds/scene_list.gz aborted: reopen(failure recovery); streamPos=2575953, nextReadPos=2575953, request range 0-38939342 length=38939342
          2016-12-07 18:48:57,550 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG s3a.S3AFileSystem (S3AInputStream.java:reopen(140)) - reopen(s3a://landsat-pds/scene_list.gz) for failure recovery range[2575953-38939342], length=569775, streamPosition=2575953, nextReadPosition=2575953
          2016-12-07 18:48:57,554 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG amazonaws.request (AmazonHttpClient.java:executeOneRequest(1046)) - Sending Request: GET https://landsat-pds.s3-us-west-2.amazonaws.com /scene_list.gz Headers: (User-Agent: Hadoop 2.8.0-SNAPSHOT, aws-sdk-java/1.11.45 Mac_OS_X/10.12.1 Java_HotSpot(TM)_64-Bit_Server_VM/25.102-b14/1.8.0_102, amz-sdk-invocation-id: 4ae4b351-3374-49e5-fa73-81cff322f56c, Range: bytes=2575953-38939342, Content-Type: application/octet-stream, ) 
          2016-12-07 18:48:58,358 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG amazonaws.request (AwsResponseHandlerAdapter.java:handle(87)) - Received successful response: 206, AWS Request ID: 773084E647B9C370
          2016-12-07 18:48:58,359 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG amazonaws.request (AwsResponseHandlerAdapter.java:logHeaderRequestId(138)) - x-amzn-RequestId: not available
          2016-12-07 18:48:58,359 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG amazonaws.request (AwsResponseHandlerAdapter.java:logResponseRequestId(158)) - AWS Request ID: 773084E647B9C370
          2016-12-07 18:48:59,397 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 2 in 121 reads: 248,531,807,139 nS
          2016-12-07 18:48:59,988 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 3 in 122 reads: 590,226,556 nS
          2016-12-07 18:49:00,346 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 4 in 122 reads: 357,777,268 nS
          2016-12-07 18:49:00,635 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 5 in 121 reads: 288,089,615 nS
          2016-12-07 18:49:00,869 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 6 in 89 reads: 233,918,422 nS
          2016-12-07 18:49:01,094 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 7 in 69 reads: 224,188,050 nS
          2016-12-07 18:49:01,270 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 8 in 69 reads: 175,089,680 nS
          2016-12-07 18:49:01,508 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 9 in 69 reads: 237,915,953 nS
          2016-12-07 18:49:03,203 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 10 in 69 reads: 1,694,581,186 nS
          2016-12-07 18:49:03,562 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 11 in 69 reads: 358,629,013 nS
          2016-12-07 18:49:03,814 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 12 in 69 reads: 251,512,010 nS
          2016-12-07 18:49:04,080 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 13 in 69 reads: 265,251,582 nS
          2016-12-07 18:49:04,284 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 14 in 69 reads: 203,401,204 nS
          2016-12-07 18:49:04,488 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 15 in 69 reads: 203,355,837 nS
          2016-12-07 18:49:04,682 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 16 in 69 reads: 193,974,256 nS
          2016-12-07 18:49:04,902 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 17 in 69 reads: 219,901,121 nS
          2016-12-07 18:49:05,089 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 18 in 69 reads: 186,285,604 nS
          2016-12-07 18:49:05,323 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 19 in 69 reads: 233,165,569 nS
          2016-12-07 18:49:06,021 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 20 in 69 reads: 698,049,891 nS
          2016-12-07 18:49:06,350 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 21 in 69 reads: 328,816,127 nS
          2016-12-07 18:49:06,571 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 22 in 69 reads: 220,920,300 nS
          2016-12-07 18:49:06,785 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 23 in 69 reads: 213,269,165 nS
          2016-12-07 18:49:07,044 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 24 in 69 reads: 258,383,769 nS
          2016-12-07 18:49:07,260 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 25 in 69 reads: 215,997,541 nS
          2016-12-07 18:49:07,472 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 26 in 69 reads: 210,871,191 nS
          2016-12-07 18:49:07,675 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 27 in 69 reads: 203,160,739 nS
          2016-12-07 18:49:07,905 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 28 in 69 reads: 229,260,978 nS
          2016-12-07 18:49:08,107 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 29 in 69 reads: 202,102,165 nS
          2016-12-07 18:49:08,387 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 30 in 69 reads: 279,023,912 nS
          2016-12-07 18:49:08,751 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 31 in 69 reads: 363,367,889 nS
          2016-12-07 18:49:09,066 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 32 in 69 reads: 315,399,466 nS
          2016-12-07 18:49:09,312 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 33 in 69 reads: 245,392,893 nS
          2016-12-07 18:49:09,566 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 34 in 69 reads: 253,079,462 nS
          2016-12-07 18:49:09,813 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 35 in 69 reads: 246,456,730 nS
          2016-12-07 18:49:10,093 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 36 in 69 reads: 279,688,361 nS
          2016-12-07 18:49:10,093 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Time to read 38939342 bytes in 37 blocks: 403,130,941,090 nS
          2016-12-07 18:49:10,094 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:bandwidth(963)) - Bandwidth = 0.091782  MB/S
          2016-12-07 18:49:10,095 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  scale.ITestS3AInputStreamPerformance (ITestS3AInputStreamPerformance.java:logStreamStatistics(324)) - Stream Statistics
          StreamStatistics{OpenOperations=2, CloseOperations=1, Closed=0, Aborted=1, SeekOperations=0, ReadExceptions=1, ForwardSeekOperations=0, BackwardSeekOperations=0, BytesSkippedOnSeek=0, BytesBackwardsOnSeek=0, BytesRead=38797312, BytesRead excluding skipped=38797312, ReadOperations=2887, ReadFullyOperations=0, ReadsIncomplete=2850, BytesReadInClose=0, BytesDiscardedInAbort=36363389}
          2016-12-07 18:49:10,095 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.AbstractFSContractTestBase (AbstractFSContractTestBase.java:describe(255)) - cleanup
          2016-12-07 18:49:10,096 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG s3a.S3AFileSystem (S3AInputStream.java:closeStream(463)) - Stream s3a://landsat-pds/scene_list.gz aborted: close() operation; streamPos=38797312, nextReadPos=38797312, request range 2575953-38939342 length=38939342
          
          Show
          stevel@apache.org Steve Loughran added a comment - - edited good news, I've been able to replicate it on the new SDK too, meaning its a network/load balance problem, not SDK related 2016-12-07 18:42:22,553 [ Thread -1] DEBUG s3a.S3AFileSystem (S3ClientFactory.java:initUserAgent(187)) - Using User-Agent: Hadoop 2.8.0-SNAPSHOT 2016-12-07 18:42:22,913 [ Thread -1] DEBUG s3a.AWSCredentialProviderList (AWSCredentialProviderList.java:getCredentials(134)) - Using credentials from BasicAWSCredentialsProvider 2016-12-07 18:42:22,921 [ Thread -1] DEBUG amazonaws.request (AmazonHttpClient.java:executeOneRequest(1046)) - Sending Request: HEAD https: //hwdev-steve-ireland- new .s3-us-west-1.amazonaws.com / Headers: (User-Agent: Hadoop 2.8.0-SNAPSHOT, aws-sdk-java/1.11.45 Mac_OS_X/10.12.1 Java_HotSpot(TM)_64-Bit_Server_VM/25.102-b14/1.8.0_102, amz-sdk-invocation-id: 787e1df3-9887-fd11-ec8d-801a7836e20d, Content-Type: application/octet-stream, ) 2016-12-07 18:42:23,927 [ Thread -1] DEBUG amazonaws.request (AmazonHttpClient.java:handleErrorResponse(1497)) - Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Moved Permanently (Service: null ; Status Code: 301; Error Code: 301 Moved Permanently; Request ID: 0D2990D655FE6CDD), S3 Extended Request ID: zam+lWQRapkEc17mnHEoLgeWBuFXGGLncQaxBi/a12Y3uohpoEAzRS1frsGb81AnU1HDJn2b7IU= 2016-12-07 18:42:23,978 [ Thread -1] DEBUG amazonaws.request (AmazonHttpClient.java:executeOneRequest(1046)) - Sending Request: HEAD https: //hwdev-steve-ireland- new .s3-eu-west-1.amazonaws.com / Headers: (User-Agent: Hadoop 2.8.0-SNAPSHOT, aws-sdk-java/1.11.45 Mac_OS_X/10.12.1 Java_HotSpot(TM)_64-Bit_Server_VM/25.102-b14/1.8.0_102, amz-sdk-invocation-id: f9fabf3f-9809-ad08-7c2f-1683f6d593cf, Content-Type: application/octet-stream, ) 2016-12-07 18:42:24,163 [ Thread -1] DEBUG amazonaws.request (AwsResponseHandlerAdapter.java:handle(87)) - Received successful response: 200, AWS Request ID: null 2016-12-07 18:42:24,164 [ Thread -1] DEBUG amazonaws.request (AwsResponseHandlerAdapter.java:logHeaderRequestId(138)) - x-amzn-RequestId: not available 2016-12-07 18:42:24,164 [ Thread -1] DEBUG amazonaws.request (AwsResponseHandlerAdapter.java:logResponseRequestId(158)) - AWS Request ID: not available 2016-12-07 18:42:24,165 [ Thread -1] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:initialize(225)) - Using encryption null 2016-12-07 18:42:24,166 [ Thread -1] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:initialize(238)) - Using S3ABlockOutputStream with buffer = disk; block=8000000; queue limit=4 2016-12-07 18:42:24,169 [ Thread -1] INFO contract.AbstractFSContractTestBase (AbstractFSContractTestBase.java:setup(184)) - Test filesystem = s3a: //hwdev-steve-ireland- new implemented by S3AFileSystem{uri=s3a://hwdev-steve-ireland- new , workingDir=s3a://hwdev-steve-ireland- new /user/stevel, inputPolicy=normal, partSize=8000000, enableMultiObjectsDelete= true , maxKeys=5000, readAhead=65536, blockSize=33554432, multiPartThreshold=2147483647, blockFactory=org.apache.hadoop.fs.s3a.S3ADataBlocks$DiskBlockFactory@51dffc97, executor=BlockingThreadPoolExecutorService{SemaphoredDelegatingExecutor{permitCount=30, available=30, waiting=0}, activeCount=0}, statistics {0 bytes read, 0 bytes written, 0 read ops, 0 large read ops, 0 write ops}, metrics {{Context=S3AFileSystem} {FileSystemId=461d5b68-e18f-4e9b-b88f-ee6cee6196ac-hwdev-steve-ireland- new } {fsURI=s3a://hwdev-steve-ireland- new /} {files_created=0} {files_copied=0} {files_copied_bytes=0} {files_deleted=0} {fake_directories_deleted=0} {directories_created=0} {directories_deleted=0} {ignored_errors=0} {op_copy_from_local_file=0} {op_exists=0} {op_get_file_status=0} {op_glob_status=0} {op_is_directory=0} {op_is_file=0} {op_list_files=0} {op_list_located_status=0} {op_list_status=0} {op_mkdirs=0} {op_rename=0} {object_copy_requests=0} {object_delete_requests=0} {object_list_requests=0} {object_continue_list_requests=0} {object_metadata_requests=0} {object_multipart_aborted=0} {object_put_bytes=0} {object_put_requests=0} {object_put_requests_completed=0} {stream_write_failures=0} {stream_write_block_uploads=0} {stream_write_block_uploads_committed=0} {stream_write_block_uploads_aborted=0} {stream_write_total_time=0} {stream_write_total_data=0} {object_put_requests_active=0} {object_put_bytes_pending=0} {stream_write_block_uploads_active=0} {stream_write_block_uploads_pending=0} {stream_write_block_uploads_data_pending=0} {stream_read_fully_operations=0} {stream_opened=0} {stream_bytes_skipped_on_seek=0} {stream_closed=0} {stream_bytes_backwards_on_seek=0} {stream_bytes_read=0} {stream_read_operations_incomplete=0} {stream_bytes_discarded_in_abort=0} {stream_close_operations=0} {stream_read_operations=0} {stream_aborted=0} {stream_forward_seek_operations=0} {stream_backward_seek_operations=0} {stream_seek_operations=0} {stream_bytes_read_in_close=0} {stream_read_exceptions=0} }} 2016-12-07 18:42:24,171 [ Thread -1] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:innerMkdirs(1478)) - Making directory: /test 2016-12-07 18:42:24,171 [ Thread -1] DEBUG s3a.S3AFileSystem (S3AStorageStatistics.java:incrementCounter(60)) - op_mkdirs += 1 -> 1 2016-12-07 18:42:24,171 [ Thread -1] DEBUG s3a.S3AFileSystem (S3AStorageStatistics.java:incrementCounter(60)) - op_get_file_status += 1 -> 1 2016-12-07 18:42:24,172 [ Thread -1] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:getFileStatus(1525)) - Getting path status for s3a: //hwdev-steve-ireland- new /test (test) 2016-12-07 18:42:24,172 [ Thread -1] DEBUG s3a.S3AFileSystem (S3AStorageStatistics.java:incrementCounter(60)) - object_metadata_requests += 1 -> 1 2016-12-07 18:42:24,174 [ Thread -1] DEBUG amazonaws.request (AmazonHttpClient.java:executeOneRequest(1046)) - Sending Request: HEAD https: //hwdev-steve-ireland- new .s3-eu-west-1.amazonaws.com /test Headers: (User-Agent: Hadoop 2.8.0-SNAPSHOT, aws-sdk-java/1.11.45 Mac_OS_X/10.12.1 Java_HotSpot(TM)_64-Bit_Server_VM/25.102-b14/1.8.0_102, amz-sdk-invocation-id: 57239941-1812-a15f-736a-f3da7727b192, Content-Type: application/octet-stream, ) 2016-12-07 18:42:24,225 [ Thread -1] DEBUG amazonaws.request (AmazonHttpClient.java:handleErrorResponse(1497)) - Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: null ; Status Code: 404; Error Code: 404 Not Found; Request ID: 2B1E6AAB5BB13A50), S3 Extended Request ID: 37X2RRn7nQAt5RDTbO8jazu8ltPGdiUZ+OmvP0cPPtYaaoEGeI/qZ5sv4fEH3xFZOb+d5+wu/w4= 2016-12-07 18:42:24,225 [ Thread -1] DEBUG s3a.S3AFileSystem (S3AStorageStatistics.java:incrementCounter(60)) - object_metadata_requests += 1 -> 2 2016-12-07 18:42:24,226 [ Thread -1] DEBUG amazonaws.request (AmazonHttpClient.java:executeOneRequest(1046)) - Sending Request: HEAD https: //hwdev-steve-ireland- new .s3-eu-west-1.amazonaws.com /test/ Headers: (User-Agent: Hadoop 2.8.0-SNAPSHOT, aws-sdk-java/1.11.45 Mac_OS_X/10.12.1 Java_HotSpot(TM)_64-Bit_Server_VM/25.102-b14/1.8.0_102, amz-sdk-invocation-id: 52c578b8-6d85-49e9-d148-2db3079cc239, Content-Type: application/octet-stream, ) 2016-12-07 18:42:24,274 [ Thread -1] DEBUG amazonaws.request (AmazonHttpClient.java:handleErrorResponse(1497)) - Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: null ; Status Code: 404; Error Code: 404 Not Found; Request ID: 4B7CD9CADDF086A7), S3 Extended Request ID: 9AR/mRdKgwAnb8SnWzLSqrM8K/DWFK1utzFqKXZihDtTN9UXPsxGFGN2GoBHgHNTqY41YBBptn8= 2016-12-07 18:42:24,274 [ Thread -1] DEBUG s3a.S3AFileSystem (S3AStorageStatistics.java:incrementCounter(60)) - object_list_requests += 1 -> 1 2016-12-07 18:42:24,289 [ Thread -1] DEBUG amazonaws.request (AmazonHttpClient.java:executeOneRequest(1046)) - Sending Request: GET https: //hwdev-steve-ireland- new .s3-eu-west-1.amazonaws.com / Parameters: ({ "prefix" :[ "test/" ], "delimiter" :[ "/" ], "max-keys" :[ "1" ], "encoding-type" :[ "url" ]}Headers: (User-Agent: Hadoop 2.8.0-SNAPSHOT, aws-sdk-java/1.11.45 Mac_OS_X/10.12.1 Java_HotSpot(TM)_64-Bit_Server_VM/25.102-b14/1.8.0_102, amz-sdk-invocation-id: 5cb39b7e-4fb2-5401-25d5-0fdc15db05ad, Content-Type: application/octet-stream, ) 2016-12-07 18:42:24,364 [ Thread -1] DEBUG amazonaws.request (AwsResponseHandlerAdapter.java:handle(87)) - Received successful response: 200, AWS Request ID: A39D3AE5D27ABAC9 2016-12-07 18:42:24,365 [ Thread -1] DEBUG amazonaws.request (AwsResponseHandlerAdapter.java:logHeaderRequestId(138)) - x-amzn-RequestId: not available 2016-12-07 18:42:24,365 [ Thread -1] DEBUG amazonaws.request (AwsResponseHandlerAdapter.java:logResponseRequestId(158)) - AWS Request ID: A39D3AE5D27ABAC9 2016-12-07 18:42:24,365 [ Thread -1] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:getFileStatus(1616)) - Not Found: s3a: //hwdev-steve-ireland- new /test 2016-12-07 18:42:24,365 [ Thread -1] DEBUG s3a.S3AFileSystem (S3AStorageStatistics.java:incrementCounter(60)) - op_get_file_status += 1 -> 2 2016-12-07 18:42:24,365 [ Thread -1] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:getFileStatus(1525)) - Getting path status for s3a: //hwdev-steve-ireland- new / () 2016-12-07 18:42:24,365 [ Thread -1] DEBUG s3a.S3AFileSystem (S3AStorageStatistics.java:incrementCounter(60)) - object_list_requests += 1 -> 2 2016-12-07 18:42:24,366 [ Thread -1] DEBUG amazonaws.request (AmazonHttpClient.java:executeOneRequest(1046)) - Sending Request: GET https: //hwdev-steve-ireland- new .s3-eu-west-1.amazonaws.com / Parameters: ({ "prefix" :[ ""]," delimiter ":[" / "]," max-keys ":[" 1 "]," encoding-type ":[" url"]}Headers: (User-Agent: Hadoop 2.8.0-SNAPSHOT, aws-sdk-java/1.11.45 Mac_OS_X/10.12.1 Java_HotSpot(TM)_64-Bit_Server_VM/25.102-b14/1.8.0_102, amz-sdk-invocation-id: d68ca495-6dee-8f17-0ff8-008f25a75702, Content-Type: application/octet-stream, ) 2016-12-07 18:42:24,426 [ Thread -1] DEBUG amazonaws.request (AwsResponseHandlerAdapter.java:handle(87)) - Received successful response: 200, AWS Request ID: 7169BE12EAD059C9 2016-12-07 18:42:24,426 [ Thread -1] DEBUG amazonaws.request (AwsResponseHandlerAdapter.java:logHeaderRequestId(138)) - x-amzn-RequestId: not available 2016-12-07 18:42:24,426 [ Thread -1] DEBUG amazonaws.request (AwsResponseHandlerAdapter.java:logResponseRequestId(158)) - AWS Request ID: 7169BE12EAD059C9 2016-12-07 18:42:24,427 [ Thread -1] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:getFileStatus(1591)) - Found path as directory (with /): 1/0 2016-12-07 18:42:24,427 [ Thread -1] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:getFileStatus(1599)) - Prefix: Users/ 2016-12-07 18:42:24,429 [ Thread -1] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:incrementPutStartStatistics(1116)) - PUT start 0 bytes 2016-12-07 18:42:24,429 [ Thread -1] DEBUG s3a.S3AFileSystem (S3AStorageStatistics.java:incrementCounter(60)) - object_put_requests += 1 -> 1 2016-12-07 18:42:24,437 [ Thread -1] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:incrementPutCompletedStatistics(1133)) - PUT completed success= true ; 0 bytes 2016-12-07 18:42:24,437 [ Thread -1] DEBUG s3a.S3AFileSystem (S3AStorageStatistics.java:incrementCounter(60)) - object_put_requests_completed += 1 -> 1 2016-12-07 18:42:24,440 [s3a-transfer-shared-pool1-t1] DEBUG amazonaws.request (AmazonHttpClient.java:executeOneRequest(1046)) - Sending Request: PUT https: //hwdev-steve-ireland- new .s3-eu-west-1.amazonaws.com /test/ Headers: (User-Agent: Hadoop 2.8.0-SNAPSHOT, aws-sdk-java/1.11.45 Mac_OS_X/10.12.1 Java_HotSpot(TM)_64-Bit_Server_VM/25.102-b14/1.8.0_102 com.amazonaws.services.s3.transfer.TransferManager/1.11.45, amz-sdk-invocation-id: fad23d78-a104-156a-81ed-f3e29d668cd8, Content-Length: 0, Content-Type: application/octet-stream, ) 2016-12-07 18:42:24,509 [s3a-transfer-shared-pool1-t1] DEBUG amazonaws.request (AwsResponseHandlerAdapter.java:handle(87)) - Received successful response: 200, AWS Request ID: 1A25A7F6C78ED591 2016-12-07 18:42:24,509 [s3a-transfer-shared-pool1-t1] DEBUG amazonaws.request (AwsResponseHandlerAdapter.java:logHeaderRequestId(138)) - x-amzn-RequestId: not available 2016-12-07 18:42:24,509 [s3a-transfer-shared-pool1-t1] DEBUG amazonaws.request (AwsResponseHandlerAdapter.java:logResponseRequestId(158)) - AWS Request ID: 1A25A7F6C78ED591 2016-12-07 18:42:24,514 [ Thread -1] DEBUG S3AFileSystem.Progress (S3AFileSystem.java:incrementPutProgressStatistics(1151)) - PUT test/: 0 bytes 2016-12-07 18:42:24,514 [ Thread -1] DEBUG scale.S3AScaleTestBase (S3AScaleTestBase.java:setup(94)) - Scale test operation count = 1 2016-12-07 18:42:24,516 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG s3a.S3AFileSystem (S3ClientFactory.java:initUserAgent(187)) - Using User-Agent: Hadoop 2.8.0-SNAPSHOT 2016-12-07 18:42:24,519 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG s3a.AWSCredentialProviderList (AWSCredentialProviderList.java:getCredentials(134)) - Using credentials from BasicAWSCredentialsProvider 2016-12-07 18:42:24,520 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG amazonaws.request (AmazonHttpClient.java:executeOneRequest(1046)) - Sending Request: HEAD https: //landsat-pds.s3-us-west-1.amazonaws.com / Headers: (User-Agent: Hadoop 2.8.0-SNAPSHOT, aws-sdk-java/1.11.45 Mac_OS_X/10.12.1 Java_HotSpot(TM)_64-Bit_Server_VM/25.102-b14/1.8.0_102, amz-sdk-invocation-id: 40808c03-5cf7-c960-4c97-353313df657b, Content-Type: application/octet-stream, ) 2016-12-07 18:42:25,338 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG amazonaws.request (AmazonHttpClient.java:handleErrorResponse(1497)) - Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Moved Permanently (Service: null ; Status Code: 301; Error Code: 301 Moved Permanently; Request ID: 58DECB0934A6428C), S3 Extended Request ID: QQxGx+1eyQEbD/skrLYpaXg+XuW8EvgwtCkpqqTJttl/ONjzD1xC1jVX6IqFCQZdfrSsup85pTM= 2016-12-07 18:42:25,339 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG amazonaws.request (AmazonHttpClient.java:executeOneRequest(1046)) - Sending Request: HEAD https: //landsat-pds.s3-us-west-2.amazonaws.com / Headers: (User-Agent: Hadoop 2.8.0-SNAPSHOT, aws-sdk-java/1.11.45 Mac_OS_X/10.12.1 Java_HotSpot(TM)_64-Bit_Server_VM/25.102-b14/1.8.0_102, amz-sdk-invocation-id: 5eb0baa1-d0f0-1825-27c1-4c122afb0d72, Content-Type: application/octet-stream, ) 2016-12-07 18:42:26,225 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG amazonaws.request (AwsResponseHandlerAdapter.java:handle(87)) - Received successful response: 200, AWS Request ID: null 2016-12-07 18:42:26,225 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG amazonaws.request (AwsResponseHandlerAdapter.java:logHeaderRequestId(138)) - x-amzn-RequestId: not available 2016-12-07 18:42:26,225 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG amazonaws.request (AwsResponseHandlerAdapter.java:logResponseRequestId(158)) - AWS Request ID: not available 2016-12-07 18:42:26,226 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:initialize(225)) - Using encryption null 2016-12-07 18:42:26,226 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:initialize(238)) - Using S3ABlockOutputStream with buffer = disk; block=8000000; queue limit=4 2016-12-07 18:42:26,226 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG s3a.S3AFileSystem (S3AStorageStatistics.java:incrementCounter(60)) - op_get_file_status += 1 -> 3 2016-12-07 18:42:26,226 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:getFileStatus(1525)) - Getting path status for s3a: //landsat-pds/scene_list.gz (scene_list.gz) 2016-12-07 18:42:26,226 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG s3a.S3AFileSystem (S3AStorageStatistics.java:incrementCounter(60)) - object_metadata_requests += 1 -> 3 2016-12-07 18:42:26,227 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG amazonaws.request (AmazonHttpClient.java:executeOneRequest(1046)) - Sending Request: HEAD https: //landsat-pds.s3-us-west-2.amazonaws.com /scene_list.gz Headers: (User-Agent: Hadoop 2.8.0-SNAPSHOT, aws-sdk-java/1.11.45 Mac_OS_X/10.12.1 Java_HotSpot(TM)_64-Bit_Server_VM/25.102-b14/1.8.0_102, amz-sdk-invocation-id: 33d34c3d-ead9-809a-af05-257b905fede7, Content-Type: application/octet-stream, ) 2016-12-07 18:42:26,788 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG amazonaws.request (AwsResponseHandlerAdapter.java:handle(87)) - Received successful response: 200, AWS Request ID: C756074918F8BAB1 2016-12-07 18:42:26,788 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG amazonaws.request (AwsResponseHandlerAdapter.java:logHeaderRequestId(138)) - x-amzn-RequestId: not available 2016-12-07 18:42:26,788 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG amazonaws.request (AwsResponseHandlerAdapter.java:logResponseRequestId(158)) - AWS Request ID: C756074918F8BAB1 2016-12-07 18:42:26,788 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:getFileStatus(1534)) - Found exact file: normal file 2016-12-07 18:42:26,789 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO s3a.AbstractS3ATestBase (AbstractS3ATestBase.java:describe(87)) - testTimeToOpenAndReadWholeFileBlocks: Open the test file s3a: //landsat-pds/scene_list.gz and read it in blocks of size 1048576 2016-12-07 18:42:26,789 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:setInputPolicy(413)) - Setting input strategy: normal 2016-12-07 18:42:26,789 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:open(503)) - Opening 's3a: //landsat-pds/scene_list.gz' for reading. 2016-12-07 18:42:26,789 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG s3a.S3AFileSystem (S3AStorageStatistics.java:incrementCounter(60)) - op_get_file_status += 1 -> 4 2016-12-07 18:42:26,789 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:getFileStatus(1525)) - Getting path status for s3a: //landsat-pds/scene_list.gz (scene_list.gz) 2016-12-07 18:42:26,789 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG s3a.S3AFileSystem (S3AStorageStatistics.java:incrementCounter(60)) - object_metadata_requests += 1 -> 4 2016-12-07 18:42:26,790 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG amazonaws.request (AmazonHttpClient.java:executeOneRequest(1046)) - Sending Request: HEAD https: //landsat-pds.s3-us-west-2.amazonaws.com /scene_list.gz Headers: (User-Agent: Hadoop 2.8.0-SNAPSHOT, aws-sdk-java/1.11.45 Mac_OS_X/10.12.1 Java_HotSpot(TM)_64-Bit_Server_VM/25.102-b14/1.8.0_102, amz-sdk-invocation-id: 5d20098b-a1cc-a15d-7cf6-959355ceda8c, Content-Type: application/octet-stream, ) 2016-12-07 18:42:26,965 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG amazonaws.request (AwsResponseHandlerAdapter.java:handle(87)) - Received successful response: 200, AWS Request ID: 48BD932265D2E504 2016-12-07 18:42:26,965 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG amazonaws.request (AwsResponseHandlerAdapter.java:logHeaderRequestId(138)) - x-amzn-RequestId: not available 2016-12-07 18:42:26,965 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG amazonaws.request (AwsResponseHandlerAdapter.java:logResponseRequestId(158)) - AWS Request ID: 48BD932265D2E504 2016-12-07 18:42:26,965 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:getFileStatus(1534)) - Found exact file: normal file 2016-12-07 18:42:26,967 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:setInputPolicy(413)) - Setting input strategy: normal 2016-12-07 18:42:26,967 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG s3a.S3AFileSystem (S3AInputStream.java:reopen(140)) - reopen(s3a: //landsat-pds/scene_list.gz) for read from new offset range[0-38939342], length=1048576, streamPosition=0, nextReadPosition=0 2016-12-07 18:42:26,969 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG amazonaws.request (AmazonHttpClient.java:executeOneRequest(1046)) - Sending Request: GET https: //landsat-pds.s3-us-west-2.amazonaws.com /scene_list.gz Headers: (User-Agent: Hadoop 2.8.0-SNAPSHOT, aws-sdk-java/1.11.45 Mac_OS_X/10.12.1 Java_HotSpot(TM)_64-Bit_Server_VM/25.102-b14/1.8.0_102, amz-sdk-invocation-id: 6489ce83-1f56-b833-15dc-f78d8eed429f, Range: bytes=0-38939342, Content-Type: application/octet-stream, ) 2016-12-07 18:42:27,168 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG amazonaws.request (AwsResponseHandlerAdapter.java:handle(87)) - Received successful response: 206, AWS Request ID: 7BB5413C293FE587 2016-12-07 18:42:27,168 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG amazonaws.request (AwsResponseHandlerAdapter.java:logHeaderRequestId(138)) - x-amzn-RequestId: not available 2016-12-07 18:42:27,168 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG amazonaws.request (AwsResponseHandlerAdapter.java:logResponseRequestId(158)) - AWS Request ID: 7BB5413C293FE587 2016-12-07 18:43:32,677 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 0 in 121 reads: 65,708,704,300 nS 2016-12-07 18:44:50,868 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 1 in 121 reads: 78,191,068,156 nS 2016-12-07 18:48:57,544 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO s3a.S3AFileSystem (S3AInputStream.java:onReadFailure(334)) - Got exception while trying to read from stream s3a: //landsat-pds/scene_list.gz trying to recover: java.net.SocketTimeoutException: Read timed out 2016-12-07 18:48:57,546 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG s3a.S3AFileSystem (S3AInputStream.java:onReadFailure(336)) - While trying to read from stream s3a: //landsat-pds/scene_list.gz java.net.SocketTimeoutException: Read timed out at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) at java.net.SocketInputStream.read(SocketInputStream.java:170) at java.net.SocketInputStream.read(SocketInputStream.java:141) at sun.security.ssl.InputRecord.readFully(InputRecord.java:465) at sun.security.ssl.InputRecord.readV3Record(InputRecord.java:593) at sun.security.ssl.InputRecord.read(InputRecord.java:532) at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973) at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:930) at sun.security.ssl.AppInputStream.read(AppInputStream.java:105) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139) at org.apache.http.impl.io.SessionInputBufferImpl.read(SessionInputBufferImpl.java:200) at org.apache.http.impl.io.ContentLengthInputStream.read(ContentLengthInputStream.java:178) at org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:137) at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:72) at com.amazonaws.event.ProgressInputStream.read(ProgressInputStream.java:181) at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:72) at com.amazonaws.services.s3.model.S3ObjectInputStream.read(S3ObjectInputStream.java:155) at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:72) at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:72) at com.amazonaws.event.ProgressInputStream.read(ProgressInputStream.java:181) at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:72) at com.amazonaws.util.LengthCheckInputStream.read(LengthCheckInputStream.java:108) at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:72) at com.amazonaws.services.s3.model.S3ObjectInputStream.read(S3ObjectInputStream.java:155) at org.apache.hadoop.fs.s3a.S3AInputStream.read(S3AInputStream.java:373) at java.io.DataInputStream.read(DataInputStream.java:149) at org.apache.hadoop.fs.s3a.scale.ITestS3AInputStreamPerformance.testTimeToOpenAndReadWholeFileBlocks(ITestS3AInputStreamPerformance.java:225) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) 2016-12-07 18:48:57,550 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG s3a.S3AFileSystem (S3AInputStream.java:closeStream(463)) - Stream s3a: //landsat-pds/scene_list.gz aborted: reopen(failure recovery); streamPos=2575953, nextReadPos=2575953, request range 0-38939342 length=38939342 2016-12-07 18:48:57,550 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG s3a.S3AFileSystem (S3AInputStream.java:reopen(140)) - reopen(s3a: //landsat-pds/scene_list.gz) for failure recovery range[2575953-38939342], length=569775, streamPosition=2575953, nextReadPosition=2575953 2016-12-07 18:48:57,554 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG amazonaws.request (AmazonHttpClient.java:executeOneRequest(1046)) - Sending Request: GET https: //landsat-pds.s3-us-west-2.amazonaws.com /scene_list.gz Headers: (User-Agent: Hadoop 2.8.0-SNAPSHOT, aws-sdk-java/1.11.45 Mac_OS_X/10.12.1 Java_HotSpot(TM)_64-Bit_Server_VM/25.102-b14/1.8.0_102, amz-sdk-invocation-id: 4ae4b351-3374-49e5-fa73-81cff322f56c, Range: bytes=2575953-38939342, Content-Type: application/octet-stream, ) 2016-12-07 18:48:58,358 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG amazonaws.request (AwsResponseHandlerAdapter.java:handle(87)) - Received successful response: 206, AWS Request ID: 773084E647B9C370 2016-12-07 18:48:58,359 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG amazonaws.request (AwsResponseHandlerAdapter.java:logHeaderRequestId(138)) - x-amzn-RequestId: not available 2016-12-07 18:48:58,359 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG amazonaws.request (AwsResponseHandlerAdapter.java:logResponseRequestId(158)) - AWS Request ID: 773084E647B9C370 2016-12-07 18:48:59,397 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 2 in 121 reads: 248,531,807,139 nS 2016-12-07 18:48:59,988 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 3 in 122 reads: 590,226,556 nS 2016-12-07 18:49:00,346 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 4 in 122 reads: 357,777,268 nS 2016-12-07 18:49:00,635 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 5 in 121 reads: 288,089,615 nS 2016-12-07 18:49:00,869 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 6 in 89 reads: 233,918,422 nS 2016-12-07 18:49:01,094 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 7 in 69 reads: 224,188,050 nS 2016-12-07 18:49:01,270 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 8 in 69 reads: 175,089,680 nS 2016-12-07 18:49:01,508 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 9 in 69 reads: 237,915,953 nS 2016-12-07 18:49:03,203 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 10 in 69 reads: 1,694,581,186 nS 2016-12-07 18:49:03,562 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 11 in 69 reads: 358,629,013 nS 2016-12-07 18:49:03,814 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 12 in 69 reads: 251,512,010 nS 2016-12-07 18:49:04,080 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 13 in 69 reads: 265,251,582 nS 2016-12-07 18:49:04,284 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 14 in 69 reads: 203,401,204 nS 2016-12-07 18:49:04,488 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 15 in 69 reads: 203,355,837 nS 2016-12-07 18:49:04,682 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 16 in 69 reads: 193,974,256 nS 2016-12-07 18:49:04,902 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 17 in 69 reads: 219,901,121 nS 2016-12-07 18:49:05,089 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 18 in 69 reads: 186,285,604 nS 2016-12-07 18:49:05,323 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 19 in 69 reads: 233,165,569 nS 2016-12-07 18:49:06,021 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 20 in 69 reads: 698,049,891 nS 2016-12-07 18:49:06,350 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 21 in 69 reads: 328,816,127 nS 2016-12-07 18:49:06,571 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 22 in 69 reads: 220,920,300 nS 2016-12-07 18:49:06,785 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 23 in 69 reads: 213,269,165 nS 2016-12-07 18:49:07,044 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 24 in 69 reads: 258,383,769 nS 2016-12-07 18:49:07,260 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 25 in 69 reads: 215,997,541 nS 2016-12-07 18:49:07,472 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 26 in 69 reads: 210,871,191 nS 2016-12-07 18:49:07,675 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 27 in 69 reads: 203,160,739 nS 2016-12-07 18:49:07,905 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 28 in 69 reads: 229,260,978 nS 2016-12-07 18:49:08,107 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 29 in 69 reads: 202,102,165 nS 2016-12-07 18:49:08,387 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 30 in 69 reads: 279,023,912 nS 2016-12-07 18:49:08,751 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 31 in 69 reads: 363,367,889 nS 2016-12-07 18:49:09,066 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 32 in 69 reads: 315,399,466 nS 2016-12-07 18:49:09,312 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 33 in 69 reads: 245,392,893 nS 2016-12-07 18:49:09,566 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 34 in 69 reads: 253,079,462 nS 2016-12-07 18:49:09,813 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 35 in 69 reads: 246,456,730 nS 2016-12-07 18:49:10,093 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 36 in 69 reads: 279,688,361 nS 2016-12-07 18:49:10,093 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Time to read 38939342 bytes in 37 blocks: 403,130,941,090 nS 2016-12-07 18:49:10,094 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:bandwidth(963)) - Bandwidth = 0.091782 MB/S 2016-12-07 18:49:10,095 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO scale.ITestS3AInputStreamPerformance (ITestS3AInputStreamPerformance.java:logStreamStatistics(324)) - Stream Statistics StreamStatistics{OpenOperations=2, CloseOperations=1, Closed=0, Aborted=1, SeekOperations=0, ReadExceptions=1, ForwardSeekOperations=0, BackwardSeekOperations=0, BytesSkippedOnSeek=0, BytesBackwardsOnSeek=0, BytesRead=38797312, BytesRead excluding skipped=38797312, ReadOperations=2887, ReadFullyOperations=0, ReadsIncomplete=2850, BytesReadInClose=0, BytesDiscardedInAbort=36363389} 2016-12-07 18:49:10,095 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.AbstractFSContractTestBase (AbstractFSContractTestBase.java:describe(255)) - cleanup 2016-12-07 18:49:10,096 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG s3a.S3AFileSystem (S3AInputStream.java:closeStream(463)) - Stream s3a: //landsat-pds/scene_list.gz aborted: close() operation; streamPos=38797312, nextReadPos=38797312, request range 2575953-38939342 length=38939342
          Hide
          stevel@apache.org Steve Loughran added a comment - - edited

          Logging org.apache.http

          2016-12-08 13:26:20,458 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG s3a.S3AFileSystem (S3AInputStream.java:reopen(140)) - reopen(s3a://landsat-pds/scene_list.gz) for read from new offset range[0-39234217], length=1048576, streamPosition=0, nextReadPosition=0
          2016-12-08 13:26:20,460 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG amazonaws.request (AmazonHttpClient.java:executeOneRequest(671)) - Sending Request: GET https://landsat-pds.s3.amazonaws.com /scene_list.gz Headers: (User-Agent: Hadoop 2.8.0-SNAPSHOT, aws-sdk-java/1.10.6 Mac_OS_X/10.12.1 Java_HotSpot(TM)_64-Bit_Server_VM/25.102-b14/1.8.0_102, Range: bytes=0-39234217, Content-Type: application/x-www-form-urlencoded; charset=utf-8, ) 
          2016-12-08 13:26:20,460 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG conn.PoolingClientConnectionManager (PoolingClientConnectionManager.java:requestConnection(184)) - Connection request: [route: {s}->https://landsat-pds.s3.amazonaws.com:443][total kept alive: 1; route allocated: 1 of 25; total allocated: 1 of 25]
          2016-12-08 13:26:20,461 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG conn.PoolingClientConnectionManager (PoolingClientConnectionManager.java:leaseConnection(218)) - Connection leased: [id: 1][route: {s}->https://landsat-pds.s3.amazonaws.com:443][total kept alive: 0; route allocated: 1 of 25; total allocated: 1 of 25]
          2016-12-08 13:26:20,462 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG protocol.RequestAddCookies (RequestAddCookies.java:process(122)) - CookieSpec selected: default
          2016-12-08 13:26:20,463 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG protocol.RequestAuthCache (RequestAuthCache.java:process(76)) - Auth cache not set in the context
          2016-12-08 13:26:20,463 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG protocol.RequestProxyAuthentication (RequestProxyAuthentication.java:process(88)) - Proxy auth state: UNCHALLENGED
          2016-12-08 13:26:20,463 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG conn.DefaultClientConnection (DefaultClientConnection.java:sendRequestHeader(276)) - Sending request: GET /scene_list.gz HTTP/1.1
          2016-12-08 13:26:20,463 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG http.headers (DefaultClientConnection.java:sendRequestHeader(280)) - >> GET /scene_list.gz HTTP/1.1
          2016-12-08 13:26:20,463 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG http.headers (DefaultClientConnection.java:sendRequestHeader(283)) - >> Host: landsat-pds.s3.amazonaws.com
          2016-12-08 13:26:20,463 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG http.headers (DefaultClientConnection.java:sendRequestHeader(283)) - >> Authorization: *REMOVED*
          2016-12-08 13:26:20,463 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG http.headers (DefaultClientConnection.java:sendRequestHeader(283)) - >> User-Agent: Hadoop 2.8.0-SNAPSHOT, aws-sdk-java/1.10.6 Mac_OS_X/10.12.1 Java_HotSpot(TM)_64-Bit_Server_VM/25.102-b14/1.8.0_102
          2016-12-08 13:26:20,464 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG http.headers (DefaultClientConnection.java:sendRequestHeader(283)) - >> Range: bytes=0-39234217
          2016-12-08 13:26:20,464 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG http.headers (DefaultClientConnection.java:sendRequestHeader(283)) - >> Date: Thu, 08 Dec 2016 13:26:20 GMT
          2016-12-08 13:26:20,464 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG http.headers (DefaultClientConnection.java:sendRequestHeader(283)) - >> Content-Type: application/x-www-form-urlencoded; charset=utf-8
          2016-12-08 13:26:20,464 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG http.headers (DefaultClientConnection.java:sendRequestHeader(283)) - >> Connection: Keep-Alive
          2016-12-08 13:26:20,643 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG conn.DefaultClientConnection (DefaultClientConnection.java:receiveResponseHeader(261)) - Receiving response: HTTP/1.1 206 Partial Content
          2016-12-08 13:26:20,643 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG http.headers (DefaultClientConnection.java:receiveResponseHeader(264)) - << HTTP/1.1 206 Partial Content
          2016-12-08 13:26:20,643 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG http.headers (DefaultClientConnection.java:receiveResponseHeader(267)) - << x-amz-id-2: cZjR9+rI+ZlDKmRWEkFmnCQmj0p7jeF9c5/kXVKeM5oKLTQRf0rQOfR1ipw5r0lnmPbfknnj+o8=
          2016-12-08 13:26:20,643 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG http.headers (DefaultClientConnection.java:receiveResponseHeader(267)) - << x-amz-request-id: 71429013F16577FD
          2016-12-08 13:26:20,643 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG http.headers (DefaultClientConnection.java:receiveResponseHeader(267)) - << Date: Thu, 08 Dec 2016 13:26:21 GMT
          2016-12-08 13:26:20,644 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG http.headers (DefaultClientConnection.java:receiveResponseHeader(267)) - << Last-Modified: Thu, 08 Dec 2016 11:32:05 GMT
          2016-12-08 13:26:20,644 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG http.headers (DefaultClientConnection.java:receiveResponseHeader(267)) - << ETag: "00585bfc6fa4c4295c5a0073f7fa6922"
          2016-12-08 13:26:20,644 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG http.headers (DefaultClientConnection.java:receiveResponseHeader(267)) - << Accept-Ranges: bytes
          2016-12-08 13:26:20,644 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG http.headers (DefaultClientConnection.java:receiveResponseHeader(267)) - << Content-Range: bytes 0-39234216/39234217
          2016-12-08 13:26:20,644 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG http.headers (DefaultClientConnection.java:receiveResponseHeader(267)) - << Content-Type: application/octet-stream
          2016-12-08 13:26:20,645 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG http.headers (DefaultClientConnection.java:receiveResponseHeader(267)) - << Content-Length: 39234217
          2016-12-08 13:26:20,645 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG http.headers (DefaultClientConnection.java:receiveResponseHeader(267)) - << Server: AmazonS3
          2016-12-08 13:26:20,645 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG amazonaws.request (AmazonHttpClient.java:logHeaderRequestId(836)) - x-amzn-RequestId: not available
          2016-12-08 13:26:20,646 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG amazonaws.request (AmazonHttpClient.java:handleResponse(1091)) - Received successful response: 206, AWS Request ID: 71429013F16577FD
          2016-12-08 13:26:20,647 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG amazonaws.request (AmazonHttpClient.java:logResponseRequestId(856)) - AWS Request ID: 71429013F16577FD
          2016-12-08 13:27:16,428 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 0 in 121 reads: 55,968,522,151 nS
          2016-12-08 13:27:18,158 [java-sdk-http-connection-reaper] DEBUG conn.PoolingClientConnectionManager (PoolingClientConnectionManager.java:closeIdleConnections(293)) - Closing connections idle longer than 60 SECONDS
          2016-12-08 13:27:18,159 [java-sdk-http-connection-reaper] DEBUG conn.PoolingClientConnectionManager (PoolingClientConnectionManager.java:closeIdleConnections(293)) - Closing connections idle longer than 60 SECONDS
          2016-12-08 13:28:18,165 [java-sdk-http-connection-reaper] DEBUG conn.PoolingClientConnectionManager (PoolingClientConnectionManager.java:closeIdleConnections(293)) - Closing connections idle longer than 60 SECONDS
          2016-12-08 13:28:18,169 [java-sdk-http-connection-reaper] DEBUG conn.DefaultClientConnection (DefaultClientConnection.java:close(182)) - Connection 0.0.0.0:50399<->52.218.16.241:443 closed
          2016-12-08 13:28:18,170 [java-sdk-http-connection-reaper] DEBUG conn.PoolingClientConnectionManager (PoolingClientConnectionManager.java:closeIdleConnections(293)) - Closing connections idle longer than 60 SECONDS
          2016-12-08 13:28:19,931 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 1 in 121 reads: 63,503,155,385 nS
          
          Show
          stevel@apache.org Steve Loughran added a comment - - edited Logging org.apache.http 2016-12-08 13:26:20,458 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG s3a.S3AFileSystem (S3AInputStream.java:reopen(140)) - reopen(s3a: //landsat-pds/scene_list.gz) for read from new offset range[0-39234217], length=1048576, streamPosition=0, nextReadPosition=0 2016-12-08 13:26:20,460 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG amazonaws.request (AmazonHttpClient.java:executeOneRequest(671)) - Sending Request: GET https: //landsat-pds.s3.amazonaws.com /scene_list.gz Headers: (User-Agent: Hadoop 2.8.0-SNAPSHOT, aws-sdk-java/1.10.6 Mac_OS_X/10.12.1 Java_HotSpot(TM)_64-Bit_Server_VM/25.102-b14/1.8.0_102, Range: bytes=0-39234217, Content-Type: application/x-www-form-urlencoded; charset=utf-8, ) 2016-12-08 13:26:20,460 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG conn.PoolingClientConnectionManager (PoolingClientConnectionManager.java:requestConnection(184)) - Connection request: [route: {s}->https: //landsat-pds.s3.amazonaws.com:443][total kept alive: 1; route allocated: 1 of 25; total allocated: 1 of 25] 2016-12-08 13:26:20,461 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG conn.PoolingClientConnectionManager (PoolingClientConnectionManager.java:leaseConnection(218)) - Connection leased: [id: 1][route: {s}->https: //landsat-pds.s3.amazonaws.com:443][total kept alive: 0; route allocated: 1 of 25; total allocated: 1 of 25] 2016-12-08 13:26:20,462 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG protocol.RequestAddCookies (RequestAddCookies.java:process(122)) - CookieSpec selected: default 2016-12-08 13:26:20,463 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG protocol.RequestAuthCache (RequestAuthCache.java:process(76)) - Auth cache not set in the context 2016-12-08 13:26:20,463 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG protocol.RequestProxyAuthentication (RequestProxyAuthentication.java:process(88)) - Proxy auth state: UNCHALLENGED 2016-12-08 13:26:20,463 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG conn.DefaultClientConnection (DefaultClientConnection.java:sendRequestHeader(276)) - Sending request: GET /scene_list.gz HTTP/1.1 2016-12-08 13:26:20,463 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG http.headers (DefaultClientConnection.java:sendRequestHeader(280)) - >> GET /scene_list.gz HTTP/1.1 2016-12-08 13:26:20,463 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG http.headers (DefaultClientConnection.java:sendRequestHeader(283)) - >> Host: landsat-pds.s3.amazonaws.com 2016-12-08 13:26:20,463 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG http.headers (DefaultClientConnection.java:sendRequestHeader(283)) - >> Authorization: *REMOVED* 2016-12-08 13:26:20,463 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG http.headers (DefaultClientConnection.java:sendRequestHeader(283)) - >> User-Agent: Hadoop 2.8.0-SNAPSHOT, aws-sdk-java/1.10.6 Mac_OS_X/10.12.1 Java_HotSpot(TM)_64-Bit_Server_VM/25.102-b14/1.8.0_102 2016-12-08 13:26:20,464 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG http.headers (DefaultClientConnection.java:sendRequestHeader(283)) - >> Range: bytes=0-39234217 2016-12-08 13:26:20,464 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG http.headers (DefaultClientConnection.java:sendRequestHeader(283)) - >> Date: Thu, 08 Dec 2016 13:26:20 GMT 2016-12-08 13:26:20,464 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG http.headers (DefaultClientConnection.java:sendRequestHeader(283)) - >> Content-Type: application/x-www-form-urlencoded; charset=utf-8 2016-12-08 13:26:20,464 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG http.headers (DefaultClientConnection.java:sendRequestHeader(283)) - >> Connection: Keep-Alive 2016-12-08 13:26:20,643 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG conn.DefaultClientConnection (DefaultClientConnection.java:receiveResponseHeader(261)) - Receiving response: HTTP/1.1 206 Partial Content 2016-12-08 13:26:20,643 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG http.headers (DefaultClientConnection.java:receiveResponseHeader(264)) - << HTTP/1.1 206 Partial Content 2016-12-08 13:26:20,643 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG http.headers (DefaultClientConnection.java:receiveResponseHeader(267)) - << x-amz-id-2: cZjR9+rI+ZlDKmRWEkFmnCQmj0p7jeF9c5/kXVKeM5oKLTQRf0rQOfR1ipw5r0lnmPbfknnj+o8= 2016-12-08 13:26:20,643 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG http.headers (DefaultClientConnection.java:receiveResponseHeader(267)) - << x-amz-request-id: 71429013F16577FD 2016-12-08 13:26:20,643 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG http.headers (DefaultClientConnection.java:receiveResponseHeader(267)) - << Date: Thu, 08 Dec 2016 13:26:21 GMT 2016-12-08 13:26:20,644 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG http.headers (DefaultClientConnection.java:receiveResponseHeader(267)) - << Last-Modified: Thu, 08 Dec 2016 11:32:05 GMT 2016-12-08 13:26:20,644 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG http.headers (DefaultClientConnection.java:receiveResponseHeader(267)) - << ETag: "00585bfc6fa4c4295c5a0073f7fa6922" 2016-12-08 13:26:20,644 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG http.headers (DefaultClientConnection.java:receiveResponseHeader(267)) - << Accept-Ranges: bytes 2016-12-08 13:26:20,644 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG http.headers (DefaultClientConnection.java:receiveResponseHeader(267)) - << Content-Range: bytes 0-39234216/39234217 2016-12-08 13:26:20,644 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG http.headers (DefaultClientConnection.java:receiveResponseHeader(267)) - << Content-Type: application/octet-stream 2016-12-08 13:26:20,645 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG http.headers (DefaultClientConnection.java:receiveResponseHeader(267)) - << Content-Length: 39234217 2016-12-08 13:26:20,645 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG http.headers (DefaultClientConnection.java:receiveResponseHeader(267)) - << Server: AmazonS3 2016-12-08 13:26:20,645 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG amazonaws.request (AmazonHttpClient.java:logHeaderRequestId(836)) - x-amzn-RequestId: not available 2016-12-08 13:26:20,646 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG amazonaws.request (AmazonHttpClient.java:handleResponse(1091)) - Received successful response: 206, AWS Request ID: 71429013F16577FD 2016-12-08 13:26:20,647 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG amazonaws.request (AmazonHttpClient.java:logResponseRequestId(856)) - AWS Request ID: 71429013F16577FD 2016-12-08 13:27:16,428 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 0 in 121 reads: 55,968,522,151 nS 2016-12-08 13:27:18,158 [java-sdk-http-connection-reaper] DEBUG conn.PoolingClientConnectionManager (PoolingClientConnectionManager.java:closeIdleConnections(293)) - Closing connections idle longer than 60 SECONDS 2016-12-08 13:27:18,159 [java-sdk-http-connection-reaper] DEBUG conn.PoolingClientConnectionManager (PoolingClientConnectionManager.java:closeIdleConnections(293)) - Closing connections idle longer than 60 SECONDS 2016-12-08 13:28:18,165 [java-sdk-http-connection-reaper] DEBUG conn.PoolingClientConnectionManager (PoolingClientConnectionManager.java:closeIdleConnections(293)) - Closing connections idle longer than 60 SECONDS 2016-12-08 13:28:18,169 [java-sdk-http-connection-reaper] DEBUG conn.DefaultClientConnection (DefaultClientConnection.java:close(182)) - Connection 0.0.0.0:50399<->52.218.16.241:443 closed 2016-12-08 13:28:18,170 [java-sdk-http-connection-reaper] DEBUG conn.PoolingClientConnectionManager (PoolingClientConnectionManager.java:closeIdleConnections(293)) - Closing connections idle longer than 60 SECONDS 2016-12-08 13:28:19,931 [JUnit-testTimeToOpenAndReadWholeFileBlocks] INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 1 in 121 reads: 63,503,155,385 nS
          Hide
          stevel@apache.org Steve Loughran added a comment -

          it looks really suspiciously like that ~60s delay per read is somehow being triggered by blocking somewhere in the stack, with the connection releases triggering the completion. Or: it's unrelated events, and that all that's happened is that the original connections set up are now being released.

          Show
          stevel@apache.org Steve Loughran added a comment - it looks really suspiciously like that ~60s delay per read is somehow being triggered by blocking somewhere in the stack, with the connection releases triggering the completion. Or: it's unrelated events, and that all that's happened is that the original connections set up are now being released.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Attaching log with detail on every single read which takes place. For the initial operation, its taking hundreds of nS/byte, until eventually there's a socket timeout and a reconnect, after which I get expected performance.

          Tested over wifi as well as ether

          Show
          stevel@apache.org Steve Loughran added a comment - Attaching log with detail on every single read which takes place. For the initial operation, its taking hundreds of nS/byte, until eventually there's a socket timeout and a reconnect, after which I get expected performance. Tested over wifi as well as ether
          Hide
          stevel@apache.org Steve Loughran added a comment -

          And it is replicable with the latest SDK. This at least implies its not an SDK problem; some local environment or networking issue

          2016-12-08 14:23:21,981 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG http.headers (LoggingManagedHttpClientConnection.java:onRequestSubmitted(135)) - http-outgoing-3 >> GET /scene_list.gz HTTP/1.1
          2016-12-08 14:23:21,981 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG http.headers (LoggingManagedHttpClientConnection.java:onRequestSubmitted(138)) - http-outgoing-3 >> Host: landsat-pds.s3-us-west-2.amazonaws.com
          2016-12-08 14:23:21,981 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG http.headers (LoggingManagedHttpClientConnection.java:onRequestSubmitted(138)) - http-outgoing-3 >> x-amz-content-sha256: UNSIGNED-PAYLOAD
          2016-12-08 14:23:21,981 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG http.headers (LoggingManagedHttpClientConnection.java:onRequestSubmitted(138)) - http-outgoing-3 >> Authorization: AWS4-HMAC-SHA256 Credential=AKIAIYZ5JQOW3N5H6NPA/20161208/us-west-2/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-retry;content-type;host;range;user-agent;x-amz-content-sha256;x-amz-date, Signature=26e0566a3ef87493309d56eac330d20d2a071010ded7cbff9886e7c7cef1bd86
          2016-12-08 14:23:21,981 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG http.headers (LoggingManagedHttpClientConnection.java:onRequestSubmitted(138)) - http-outgoing-3 >> X-Amz-Date: 20161208T142321Z
          2016-12-08 14:23:21,981 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG http.headers (LoggingManagedHttpClientConnection.java:onRequestSubmitted(138)) - http-outgoing-3 >> User-Agent: Hadoop 2.8.0-SNAPSHOT, aws-sdk-java/1.11.45 Mac_OS_X/10.12.1 Java_HotSpot(TM)_64-Bit_Server_VM/25.102-b14/1.8.0_102
          2016-12-08 14:23:21,981 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG http.headers (LoggingManagedHttpClientConnection.java:onRequestSubmitted(138)) - http-outgoing-3 >> amz-sdk-invocation-id: ddf8d5b5-96a3-57a4-96d6-fbb66f63f6fa
          2016-12-08 14:23:21,981 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG http.headers (LoggingManagedHttpClientConnection.java:onRequestSubmitted(138)) - http-outgoing-3 >> amz-sdk-retry: 0/0/500
          2016-12-08 14:23:21,982 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG http.headers (LoggingManagedHttpClientConnection.java:onRequestSubmitted(138)) - http-outgoing-3 >> Range: bytes=0-39234217
          2016-12-08 14:23:21,982 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG http.headers (LoggingManagedHttpClientConnection.java:onRequestSubmitted(138)) - http-outgoing-3 >> Content-Type: application/octet-stream
          2016-12-08 14:23:21,982 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG http.headers (LoggingManagedHttpClientConnection.java:onRequestSubmitted(138)) - http-outgoing-3 >> Connection: Keep-Alive
          2016-12-08 14:23:22,301 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG http.headers (LoggingManagedHttpClientConnection.java:onResponseReceived(124)) - http-outgoing-3 << HTTP/1.1 206 Partial Content
          2016-12-08 14:23:22,301 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG http.headers (LoggingManagedHttpClientConnection.java:onResponseReceived(127)) - http-outgoing-3 << x-amz-id-2: SykItR7JBcTf1TsNkVK58VYN2164t1IDsMa+ZpLUBGuJW+Bxb9ONEAHFvi85JhhGvLZJxXlwc3k=
          2016-12-08 14:23:22,301 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG http.headers (LoggingManagedHttpClientConnection.java:onResponseReceived(127)) - http-outgoing-3 << x-amz-request-id: 0AE8E8AD401F45D6
          2016-12-08 14:23:22,301 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG http.headers (LoggingManagedHttpClientConnection.java:onResponseReceived(127)) - http-outgoing-3 << Date: Thu, 08 Dec 2016 14:23:23 GMT
          2016-12-08 14:23:22,301 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG http.headers (LoggingManagedHttpClientConnection.java:onResponseReceived(127)) - http-outgoing-3 << Last-Modified: Thu, 08 Dec 2016 11:32:05 GMT
          2016-12-08 14:23:22,302 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG http.headers (LoggingManagedHttpClientConnection.java:onResponseReceived(127)) - http-outgoing-3 << ETag: "00585bfc6fa4c4295c5a0073f7fa6922"
          2016-12-08 14:23:22,302 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG http.headers (LoggingManagedHttpClientConnection.java:onResponseReceived(127)) - http-outgoing-3 << Accept-Ranges: bytes
          2016-12-08 14:23:22,302 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG http.headers (LoggingManagedHttpClientConnection.java:onResponseReceived(127)) - http-outgoing-3 << Content-Range: bytes 0-39234216/39234217
          2016-12-08 14:23:22,302 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG http.headers (LoggingManagedHttpClientConnection.java:onResponseReceived(127)) - http-outgoing-3 << Content-Type: application/octet-stream
          2016-12-08 14:23:22,302 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG http.headers (LoggingManagedHttpClientConnection.java:onResponseReceived(127)) - http-outgoing-3 << Content-Length: 39234217
          2016-12-08 14:23:22,302 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG http.headers (LoggingManagedHttpClientConnection.java:onResponseReceived(127)) - http-outgoing-3 << Server: AmazonS3
          2016-12-08 14:23:22,303 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG execchain.MainClientExec (MainClientExec.java:execute(284)) - Connection can be kept alive for 60000 MILLISECONDS
          2016-12-08 14:23:22,304 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG amazonaws.request (AwsResponseHandlerAdapter.java:handle(87)) - Received successful response: 206, AWS Request ID: 0AE8E8AD401F45D6
          2016-12-08 14:23:22,304 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG amazonaws.request (AwsResponseHandlerAdapter.java:logHeaderRequestId(138)) - x-amzn-RequestId: not available
          2016-12-08 14:23:22,304 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG amazonaws.request (AwsResponseHandlerAdapter.java:logResponseRequestId(158)) - AWS Request ID: 0AE8E8AD401F45D6
          2016-12-08 14:23:22,486  - Bytes in read #1: 16347 , block bytes: 16347, remaining in block: 1032229 duration=501762710 nS; ns/byte: 30694, bandwidth=0.031070 MB/s
          2016-12-08 14:23:22,487  - Bytes in read #2: 630 , block bytes: 16977, remaining in block: 1031599 duration=160825 nS; ns/byte: 255, bandwidth=3.735830 MB/s
          2016-12-08 14:23:22,827  - Bytes in read #3: 16347 , block bytes: 33324, remaining in block: 1015252 duration=339676241 nS; ns/byte: 20779, bandwidth=0.045896 MB/s
          2016-12-08 14:23:22,827  - Bytes in read #4: 1061 , block bytes: 34385, remaining in block: 1014191 duration=204387 nS; ns/byte: 192, bandwidth=4.950650 MB/s
          2016-12-08 14:23:23,016  - Bytes in read #5: 16347 , block bytes: 50732, remaining in block: 997844 duration=188918899 nS; ns/byte: 11556, bandwidth=0.082521 MB/s
          2016-12-08 14:23:23,017  - Bytes in read #6: 1061 , block bytes: 51793, remaining in block: 996783 duration=149986 nS; ns/byte: 141, bandwidth=6.746286 MB/s
          2016-12-08 14:23:23,018  - Bytes in read #7: 16347 , block bytes: 68140, remaining in block: 980436 duration=668380 nS; ns/byte: 40, bandwidth=23.324627 MB/s
          2016-12-08 14:23:23,018  - Bytes in read #8: 1061 , block bytes: 69201, remaining in block: 979375 duration=109412 nS; ns/byte: 103, bandwidth=9.248057 MB/s
          2016-12-08 14:23:23,545  - Bytes in read #9: 16347 , block bytes: 85548, remaining in block: 963028 duration=526539853 nS; ns/byte: 32210, bandwidth=0.029608 MB/s
          2016-12-08 14:23:23,545  - Bytes in read #10: 1061 , block bytes: 86609, remaining in block: 961967 duration=160982 nS; ns/byte: 151, bandwidth=6.285476 MB/s
          2016-12-08 14:23:24,075  - Bytes in read #11: 16347 , block bytes: 102956, remaining in block: 945620 duration=529957790 nS; ns/byte: 32419, bandwidth=0.029417 MB/s
          2016-12-08 14:23:24,076  - Bytes in read #12: 1061 , block bytes: 104017, remaining in block: 944559 duration=176802 nS; ns/byte: 166, bandwidth=5.723060 MB/s
          2016-12-08 14:23:24,256  - Bytes in read #13: 16347 , block bytes: 120364, remaining in block: 928212 duration=180087310 nS; ns/byte: 11016, bandwidth=0.086568 MB/s
          2016-12-08 14:23:24,257  - Bytes in read #14: 1061 , block bytes: 121425, remaining in block: 927151 duration=162863 nS; ns/byte: 153, bandwidth=6.212881 MB/s
          2016-12-08 14:23:25,154  - Bytes in read #15: 16347 , block bytes: 137772, remaining in block: 910804 duration=896281315 nS; ns/byte: 54828, bandwidth=0.017394 MB/s
          2016-12-08 14:23:25,154  - Bytes in read #16: 1061 , block bytes: 138833, remaining in block: 909743 duration=144944 nS; ns/byte: 136, bandwidth=6.980961 MB/s
          2016-12-08 14:23:25,979  - Bytes in read #17: 16347 , block bytes: 155180, remaining in block: 893396 duration=824595866 nS; ns/byte: 50443, bandwidth=0.018906 MB/s
          2016-12-08 14:23:25,979  - Bytes in read #18: 1061 , block bytes: 156241, remaining in block: 892335 duration=177202 nS; ns/byte: 167, bandwidth=5.710141 MB/s
          2016-12-08 14:23:27,293  - Bytes in read #19: 16347 , block bytes: 172588, remaining in block: 875988 duration=1313351115 nS; ns/byte: 80342, bandwidth=0.011870 MB/s
          2016-12-08 14:23:27,294  - Bytes in read #20: 1061 , block bytes: 173649, remaining in block: 874927 duration=171179 nS; ns/byte: 161, bandwidth=5.911055 MB/s
          2016-12-08 14:23:28,102  - Bytes in read #21: 16347 , block bytes: 189996, remaining in block: 858580 duration=808444310 nS; ns/byte: 49455, bandwidth=0.019284 MB/s
          2016-12-08 14:23:28,103  - Bytes in read #22: 1061 , block bytes: 191057, remaining in block: 857519 duration=94308 nS; ns/byte: 88, bandwidth=10.729190 MB/s
          2016-12-08 14:23:28,657  - Bytes in read #23: 16347 , block bytes: 207404, remaining in block: 841172 duration=554500694 nS; ns/byte: 33920, bandwidth=0.028115 MB/s
          2016-12-08 14:23:28,983  - Bytes in read #24: 1061 , block bytes: 208465, remaining in block: 840111 duration=325204791 nS; ns/byte: 306507, bandwidth=0.003111 MB/s
          2016-12-08 14:23:31,339  - Bytes in read #25: 16347 , block bytes: 224812, remaining in block: 823764 duration=2355362418 nS; ns/byte: 144085, bandwidth=0.006619 MB/s
          2016-12-08 14:23:31,340  - Bytes in read #26: 1061 , block bytes: 225873, remaining in block: 822703 duration=1329505 nS; ns/byte: 1253, bandwidth=0.761072 MB/s
          2016-12-08 14:23:32,054  - Bytes in read #27: 16347 , block bytes: 242220, remaining in block: 806356 duration=713105706 nS; ns/byte: 43623, bandwidth=0.021862 MB/s
          2016-12-08 14:23:32,054  - Bytes in read #28: 1061 , block bytes: 243281, remaining in block: 805295 duration=140448 nS; ns/byte: 132, bandwidth=7.204435 MB/s
          2016-12-08 14:23:33,300  - Bytes in read #29: 16347 , block bytes: 259628, remaining in block: 788948 duration=1245922562 nS; ns/byte: 76217, bandwidth=0.012513 MB/s
          2016-12-08 14:23:33,767  - Bytes in read #30: 1061 , block bytes: 260689, remaining in block: 787887 duration=466030523 nS; ns/byte: 439237, bandwidth=0.002171 MB/s
          2016-12-08 14:23:34,300  - Bytes in read #31: 16347 , block bytes: 277036, remaining in block: 771540 duration=533252203 nS; ns/byte: 32620, bandwidth=0.029235 MB/s
          2016-12-08 14:23:34,382  - Bytes in read #32: 1061 , block bytes: 278097, remaining in block: 770479 duration=80960596 nS; ns/byte: 76305, bandwidth=0.012498 MB/s
          2016-12-08 14:23:35,027  - Bytes in read #33: 16347 , block bytes: 294444, remaining in block: 754132 duration=644705881 nS; ns/byte: 39438, bandwidth=0.024181 MB/s
          2016-12-08 14:23:35,027  - Bytes in read #34: 1061 , block bytes: 295505, remaining in block: 753071 duration=79536 nS; ns/byte: 74, bandwidth=12.721893 MB/s
          2016-12-08 14:23:35,579  - Bytes in read #35: 16347 , block bytes: 311852, remaining in block: 736724 duration=551360213 nS; ns/byte: 33728, bandwidth=0.028275 MB/s
          2016-12-08 14:23:35,904  - Bytes in read #36: 1061 , block bytes: 312913, remaining in block: 735663 duration=325436462 nS; ns/byte: 306726, bandwidth=0.003109 MB/s
          2016-12-08 14:23:36,776  - Bytes in read #37: 16347 , block bytes: 329260, remaining in block: 719316 duration=871155772 nS; ns/byte: 53291, bandwidth=0.017895 MB/s
          2016-12-08 14:23:36,777  - Bytes in read #38: 1061 , block bytes: 330321, remaining in block: 718255 duration=241257 nS; ns/byte: 227, bandwidth=4.194069 MB/s
          2016-12-08 14:23:37,838  - Bytes in read #39: 16347 , block bytes: 346668, remaining in block: 701908 duration=1061024046 nS; ns/byte: 64906, bandwidth=0.014693 MB/s
          2016-12-08 14:23:38,014  - Bytes in read #40: 1061 , block bytes: 347729, remaining in block: 700847 duration=175298804 nS; ns/byte: 165220, bandwidth=0.005772 MB/s
          2016-12-08 14:23:38,553  - Bytes in read #41: 16347 , block bytes: 364076, remaining in block: 684500 duration=538434297 nS; ns/byte: 32937, bandwidth=0.028954 MB/s
          2016-12-08 14:23:38,554  - Bytes in read #42: 1061 , block bytes: 365137, remaining in block: 683439 duration=1233409 nS; ns/byte: 1162, bandwidth=0.820367 MB/s
          2016-12-08 14:23:40,229  - Bytes in read #43: 16347 , block bytes: 381484, remaining in block: 667092 duration=1674577082 nS; ns/byte: 102439, bandwidth=0.009310 MB/s
          2016-12-08 14:23:40,586  - Bytes in read #44: 1061 , block bytes: 382545, remaining in block: 666031 duration=357021347 nS; ns/byte: 336495, bandwidth=0.002834 MB/s
          2016-12-08 14:23:41,275  - Bytes in read #45: 16347 , block bytes: 398892, remaining in block: 649684 duration=688294002 nS; ns/byte: 42105, bandwidth=0.022650 MB/s
          2016-12-08 14:23:41,276  - Bytes in read #46: 1061 , block bytes: 399953, remaining in block: 648623 duration=140726 nS; ns/byte: 132, bandwidth=7.190203 MB/s
          2016-12-08 14:23:42,389  - Bytes in read #47: 16347 , block bytes: 416300, remaining in block: 632276 duration=1113488278 nS; ns/byte: 68115, bandwidth=0.014001 MB/s
          2016-12-08 14:23:42,390  - Bytes in read #48: 1061 , block bytes: 417361, remaining in block: 631215 duration=163118 nS; ns/byte: 153, bandwidth=6.203169 MB/s
          2016-12-08 14:23:43,397  - Bytes in read #49: 16347 , block bytes: 433708, remaining in block: 614868 duration=1006992034 nS; ns/byte: 61601, bandwidth=0.015481 MB/s
          2016-12-08 14:23:43,573  - Bytes in read #50: 1061 , block bytes: 434769, remaining in block: 613807 duration=175484587 nS; ns/byte: 165395, bandwidth=0.005766 MB/s
          2016-12-08 14:23:43,927  - Bytes in read #51: 16347 , block bytes: 451116, remaining in block: 597460 duration=353339533 nS; ns/byte: 21614, bandwidth=0.044121 MB/s
          2016-12-08 14:23:44,102  - Bytes in read #52: 1061 , block bytes: 452177, remaining in block: 596399 duration=174634494 nS; ns/byte: 164594, bandwidth=0.005794 MB/s
          2016-12-08 14:23:44,985  - Bytes in read #53: 16347 , block bytes: 468524, remaining in block: 580052 duration=882841721 nS; ns/byte: 54006, bandwidth=0.017659 MB/s
          2016-12-08 14:23:44,986  - Bytes in read #54: 1061 , block bytes: 469585, remaining in block: 578991 duration=168354 nS; ns/byte: 158, bandwidth=6.010243 MB/s
          2016-12-08 14:23:46,337  - Bytes in read #55: 16347 , block bytes: 485932, remaining in block: 562644 duration=1350476638 nS; ns/byte: 82613, bandwidth=0.011544 MB/s
          2016-12-08 14:23:46,337  - Bytes in read #56: 1061 , block bytes: 486993, remaining in block: 561583 duration=168682 nS; ns/byte: 158, bandwidth=5.998556 MB/s
          2016-12-08 14:23:47,058  - Bytes in read #57: 16347 , block bytes: 503340, remaining in block: 545236 duration=720245455 nS; ns/byte: 44059, bandwidth=0.021645 MB/s
          2016-12-08 14:23:47,058  - Bytes in read #58: 1061 , block bytes: 504401, remaining in block: 544175 duration=109153 nS; ns/byte: 102, bandwidth=9.270001 MB/s
          2016-12-08 14:23:48,051  - Bytes in read #59: 16347 , block bytes: 520748, remaining in block: 527828 duration=992763828 nS; ns/byte: 60730, bandwidth=0.015703 MB/s
          2016-12-08 14:23:48,517  - Bytes in read #60: 1061 , block bytes: 521809, remaining in block: 526767 duration=465289883 nS; ns/byte: 438539, bandwidth=0.002175 MB/s
          2016-12-08 14:23:48,851  - Bytes in read #61: 16347 , block bytes: 538156, remaining in block: 510420 duration=333839573 nS; ns/byte: 20422, bandwidth=0.046698 MB/s
          2016-12-08 14:23:49,211  - Bytes in read #62: 1061 , block bytes: 539217, remaining in block: 509359 duration=359559220 nS; ns/byte: 338887, bandwidth=0.002814 MB/s
          2016-12-08 14:23:50,006  - Bytes in read #63: 16347 , block bytes: 555564, remaining in block: 493012 duration=794069765 nS; ns/byte: 48575, bandwidth=0.019633 MB/s
          2016-12-08 14:23:50,006  - Bytes in read #64: 1061 , block bytes: 556625, remaining in block: 491951 duration=116647 nS; ns/byte: 109, bandwidth=8.674449 MB/s
          2016-12-08 14:23:50,896  - Bytes in read #65: 16347 , block bytes: 572972, remaining in block: 475604 duration=889335208 nS; ns/byte: 54403, bandwidth=0.017530 MB/s
          2016-12-08 14:23:51,280  - Bytes in read #66: 1061 , block bytes: 574033, remaining in block: 474543 duration=384066718 nS; ns/byte: 361985, bandwidth=0.002635 MB/s
          2016-12-08 14:23:51,791  - Bytes in read #67: 16347 , block bytes: 590380, remaining in block: 458196 duration=509948788 nS; ns/byte: 31195, bandwidth=0.030571 MB/s
          2016-12-08 14:23:51,791  - Bytes in read #68: 1061 , block bytes: 591441, remaining in block: 457135 duration=168685 nS; ns/byte: 158, bandwidth=5.998449 MB/s
          2016-12-08 14:23:52,326  - Bytes in read #69: 16347 , block bytes: 607788, remaining in block: 440788 duration=534885343 nS; ns/byte: 32720, bandwidth=0.029146 MB/s
          2016-12-08 14:23:52,327  - Bytes in read #70: 1061 , block bytes: 608849, remaining in block: 439727 duration=130685 nS; ns/byte: 123, bandwidth=7.742652 MB/s
          2016-12-08 14:23:53,890  - Bytes in read #71: 16347 , block bytes: 625196, remaining in block: 423380 duration=1563070584 nS; ns/byte: 95618, bandwidth=0.009974 MB/s
          
          Show
          stevel@apache.org Steve Loughran added a comment - And it is replicable with the latest SDK. This at least implies its not an SDK problem; some local environment or networking issue 2016-12-08 14:23:21,981 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG http.headers (LoggingManagedHttpClientConnection.java:onRequestSubmitted(135)) - http-outgoing-3 >> GET /scene_list.gz HTTP/1.1 2016-12-08 14:23:21,981 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG http.headers (LoggingManagedHttpClientConnection.java:onRequestSubmitted(138)) - http-outgoing-3 >> Host: landsat-pds.s3-us-west-2.amazonaws.com 2016-12-08 14:23:21,981 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG http.headers (LoggingManagedHttpClientConnection.java:onRequestSubmitted(138)) - http-outgoing-3 >> x-amz-content-sha256: UNSIGNED-PAYLOAD 2016-12-08 14:23:21,981 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG http.headers (LoggingManagedHttpClientConnection.java:onRequestSubmitted(138)) - http-outgoing-3 >> Authorization: AWS4-HMAC-SHA256 Credential=AKIAIYZ5JQOW3N5H6NPA/20161208/us-west-2/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-retry;content-type;host;range;user-agent;x-amz-content-sha256;x-amz-date, Signature=26e0566a3ef87493309d56eac330d20d2a071010ded7cbff9886e7c7cef1bd86 2016-12-08 14:23:21,981 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG http.headers (LoggingManagedHttpClientConnection.java:onRequestSubmitted(138)) - http-outgoing-3 >> X-Amz-Date: 20161208T142321Z 2016-12-08 14:23:21,981 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG http.headers (LoggingManagedHttpClientConnection.java:onRequestSubmitted(138)) - http-outgoing-3 >> User-Agent: Hadoop 2.8.0-SNAPSHOT, aws-sdk-java/1.11.45 Mac_OS_X/10.12.1 Java_HotSpot(TM)_64-Bit_Server_VM/25.102-b14/1.8.0_102 2016-12-08 14:23:21,981 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG http.headers (LoggingManagedHttpClientConnection.java:onRequestSubmitted(138)) - http-outgoing-3 >> amz-sdk-invocation-id: ddf8d5b5-96a3-57a4-96d6-fbb66f63f6fa 2016-12-08 14:23:21,981 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG http.headers (LoggingManagedHttpClientConnection.java:onRequestSubmitted(138)) - http-outgoing-3 >> amz-sdk-retry: 0/0/500 2016-12-08 14:23:21,982 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG http.headers (LoggingManagedHttpClientConnection.java:onRequestSubmitted(138)) - http-outgoing-3 >> Range: bytes=0-39234217 2016-12-08 14:23:21,982 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG http.headers (LoggingManagedHttpClientConnection.java:onRequestSubmitted(138)) - http-outgoing-3 >> Content-Type: application/octet-stream 2016-12-08 14:23:21,982 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG http.headers (LoggingManagedHttpClientConnection.java:onRequestSubmitted(138)) - http-outgoing-3 >> Connection: Keep-Alive 2016-12-08 14:23:22,301 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG http.headers (LoggingManagedHttpClientConnection.java:onResponseReceived(124)) - http-outgoing-3 << HTTP/1.1 206 Partial Content 2016-12-08 14:23:22,301 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG http.headers (LoggingManagedHttpClientConnection.java:onResponseReceived(127)) - http-outgoing-3 << x-amz-id-2: SykItR7JBcTf1TsNkVK58VYN2164t1IDsMa+ZpLUBGuJW+Bxb9ONEAHFvi85JhhGvLZJxXlwc3k= 2016-12-08 14:23:22,301 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG http.headers (LoggingManagedHttpClientConnection.java:onResponseReceived(127)) - http-outgoing-3 << x-amz-request-id: 0AE8E8AD401F45D6 2016-12-08 14:23:22,301 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG http.headers (LoggingManagedHttpClientConnection.java:onResponseReceived(127)) - http-outgoing-3 << Date: Thu, 08 Dec 2016 14:23:23 GMT 2016-12-08 14:23:22,301 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG http.headers (LoggingManagedHttpClientConnection.java:onResponseReceived(127)) - http-outgoing-3 << Last-Modified: Thu, 08 Dec 2016 11:32:05 GMT 2016-12-08 14:23:22,302 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG http.headers (LoggingManagedHttpClientConnection.java:onResponseReceived(127)) - http-outgoing-3 << ETag: "00585bfc6fa4c4295c5a0073f7fa6922" 2016-12-08 14:23:22,302 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG http.headers (LoggingManagedHttpClientConnection.java:onResponseReceived(127)) - http-outgoing-3 << Accept-Ranges: bytes 2016-12-08 14:23:22,302 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG http.headers (LoggingManagedHttpClientConnection.java:onResponseReceived(127)) - http-outgoing-3 << Content-Range: bytes 0-39234216/39234217 2016-12-08 14:23:22,302 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG http.headers (LoggingManagedHttpClientConnection.java:onResponseReceived(127)) - http-outgoing-3 << Content-Type: application/octet-stream 2016-12-08 14:23:22,302 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG http.headers (LoggingManagedHttpClientConnection.java:onResponseReceived(127)) - http-outgoing-3 << Content-Length: 39234217 2016-12-08 14:23:22,302 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG http.headers (LoggingManagedHttpClientConnection.java:onResponseReceived(127)) - http-outgoing-3 << Server: AmazonS3 2016-12-08 14:23:22,303 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG execchain.MainClientExec (MainClientExec.java:execute(284)) - Connection can be kept alive for 60000 MILLISECONDS 2016-12-08 14:23:22,304 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG amazonaws.request (AwsResponseHandlerAdapter.java:handle(87)) - Received successful response: 206, AWS Request ID: 0AE8E8AD401F45D6 2016-12-08 14:23:22,304 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG amazonaws.request (AwsResponseHandlerAdapter.java:logHeaderRequestId(138)) - x-amzn-RequestId: not available 2016-12-08 14:23:22,304 [JUnit-testTimeToOpenAndReadWholeFileBlocks] DEBUG amazonaws.request (AwsResponseHandlerAdapter.java:logResponseRequestId(158)) - AWS Request ID: 0AE8E8AD401F45D6 2016-12-08 14:23:22,486 - Bytes in read #1: 16347 , block bytes: 16347, remaining in block: 1032229 duration=501762710 nS; ns/ byte : 30694, bandwidth=0.031070 MB/s 2016-12-08 14:23:22,487 - Bytes in read #2: 630 , block bytes: 16977, remaining in block: 1031599 duration=160825 nS; ns/ byte : 255, bandwidth=3.735830 MB/s 2016-12-08 14:23:22,827 - Bytes in read #3: 16347 , block bytes: 33324, remaining in block: 1015252 duration=339676241 nS; ns/ byte : 20779, bandwidth=0.045896 MB/s 2016-12-08 14:23:22,827 - Bytes in read #4: 1061 , block bytes: 34385, remaining in block: 1014191 duration=204387 nS; ns/ byte : 192, bandwidth=4.950650 MB/s 2016-12-08 14:23:23,016 - Bytes in read #5: 16347 , block bytes: 50732, remaining in block: 997844 duration=188918899 nS; ns/ byte : 11556, bandwidth=0.082521 MB/s 2016-12-08 14:23:23,017 - Bytes in read #6: 1061 , block bytes: 51793, remaining in block: 996783 duration=149986 nS; ns/ byte : 141, bandwidth=6.746286 MB/s 2016-12-08 14:23:23,018 - Bytes in read #7: 16347 , block bytes: 68140, remaining in block: 980436 duration=668380 nS; ns/ byte : 40, bandwidth=23.324627 MB/s 2016-12-08 14:23:23,018 - Bytes in read #8: 1061 , block bytes: 69201, remaining in block: 979375 duration=109412 nS; ns/ byte : 103, bandwidth=9.248057 MB/s 2016-12-08 14:23:23,545 - Bytes in read #9: 16347 , block bytes: 85548, remaining in block: 963028 duration=526539853 nS; ns/ byte : 32210, bandwidth=0.029608 MB/s 2016-12-08 14:23:23,545 - Bytes in read #10: 1061 , block bytes: 86609, remaining in block: 961967 duration=160982 nS; ns/ byte : 151, bandwidth=6.285476 MB/s 2016-12-08 14:23:24,075 - Bytes in read #11: 16347 , block bytes: 102956, remaining in block: 945620 duration=529957790 nS; ns/ byte : 32419, bandwidth=0.029417 MB/s 2016-12-08 14:23:24,076 - Bytes in read #12: 1061 , block bytes: 104017, remaining in block: 944559 duration=176802 nS; ns/ byte : 166, bandwidth=5.723060 MB/s 2016-12-08 14:23:24,256 - Bytes in read #13: 16347 , block bytes: 120364, remaining in block: 928212 duration=180087310 nS; ns/ byte : 11016, bandwidth=0.086568 MB/s 2016-12-08 14:23:24,257 - Bytes in read #14: 1061 , block bytes: 121425, remaining in block: 927151 duration=162863 nS; ns/ byte : 153, bandwidth=6.212881 MB/s 2016-12-08 14:23:25,154 - Bytes in read #15: 16347 , block bytes: 137772, remaining in block: 910804 duration=896281315 nS; ns/ byte : 54828, bandwidth=0.017394 MB/s 2016-12-08 14:23:25,154 - Bytes in read #16: 1061 , block bytes: 138833, remaining in block: 909743 duration=144944 nS; ns/ byte : 136, bandwidth=6.980961 MB/s 2016-12-08 14:23:25,979 - Bytes in read #17: 16347 , block bytes: 155180, remaining in block: 893396 duration=824595866 nS; ns/ byte : 50443, bandwidth=0.018906 MB/s 2016-12-08 14:23:25,979 - Bytes in read #18: 1061 , block bytes: 156241, remaining in block: 892335 duration=177202 nS; ns/ byte : 167, bandwidth=5.710141 MB/s 2016-12-08 14:23:27,293 - Bytes in read #19: 16347 , block bytes: 172588, remaining in block: 875988 duration=1313351115 nS; ns/ byte : 80342, bandwidth=0.011870 MB/s 2016-12-08 14:23:27,294 - Bytes in read #20: 1061 , block bytes: 173649, remaining in block: 874927 duration=171179 nS; ns/ byte : 161, bandwidth=5.911055 MB/s 2016-12-08 14:23:28,102 - Bytes in read #21: 16347 , block bytes: 189996, remaining in block: 858580 duration=808444310 nS; ns/ byte : 49455, bandwidth=0.019284 MB/s 2016-12-08 14:23:28,103 - Bytes in read #22: 1061 , block bytes: 191057, remaining in block: 857519 duration=94308 nS; ns/ byte : 88, bandwidth=10.729190 MB/s 2016-12-08 14:23:28,657 - Bytes in read #23: 16347 , block bytes: 207404, remaining in block: 841172 duration=554500694 nS; ns/ byte : 33920, bandwidth=0.028115 MB/s 2016-12-08 14:23:28,983 - Bytes in read #24: 1061 , block bytes: 208465, remaining in block: 840111 duration=325204791 nS; ns/ byte : 306507, bandwidth=0.003111 MB/s 2016-12-08 14:23:31,339 - Bytes in read #25: 16347 , block bytes: 224812, remaining in block: 823764 duration=2355362418 nS; ns/ byte : 144085, bandwidth=0.006619 MB/s 2016-12-08 14:23:31,340 - Bytes in read #26: 1061 , block bytes: 225873, remaining in block: 822703 duration=1329505 nS; ns/ byte : 1253, bandwidth=0.761072 MB/s 2016-12-08 14:23:32,054 - Bytes in read #27: 16347 , block bytes: 242220, remaining in block: 806356 duration=713105706 nS; ns/ byte : 43623, bandwidth=0.021862 MB/s 2016-12-08 14:23:32,054 - Bytes in read #28: 1061 , block bytes: 243281, remaining in block: 805295 duration=140448 nS; ns/ byte : 132, bandwidth=7.204435 MB/s 2016-12-08 14:23:33,300 - Bytes in read #29: 16347 , block bytes: 259628, remaining in block: 788948 duration=1245922562 nS; ns/ byte : 76217, bandwidth=0.012513 MB/s 2016-12-08 14:23:33,767 - Bytes in read #30: 1061 , block bytes: 260689, remaining in block: 787887 duration=466030523 nS; ns/ byte : 439237, bandwidth=0.002171 MB/s 2016-12-08 14:23:34,300 - Bytes in read #31: 16347 , block bytes: 277036, remaining in block: 771540 duration=533252203 nS; ns/ byte : 32620, bandwidth=0.029235 MB/s 2016-12-08 14:23:34,382 - Bytes in read #32: 1061 , block bytes: 278097, remaining in block: 770479 duration=80960596 nS; ns/ byte : 76305, bandwidth=0.012498 MB/s 2016-12-08 14:23:35,027 - Bytes in read #33: 16347 , block bytes: 294444, remaining in block: 754132 duration=644705881 nS; ns/ byte : 39438, bandwidth=0.024181 MB/s 2016-12-08 14:23:35,027 - Bytes in read #34: 1061 , block bytes: 295505, remaining in block: 753071 duration=79536 nS; ns/ byte : 74, bandwidth=12.721893 MB/s 2016-12-08 14:23:35,579 - Bytes in read #35: 16347 , block bytes: 311852, remaining in block: 736724 duration=551360213 nS; ns/ byte : 33728, bandwidth=0.028275 MB/s 2016-12-08 14:23:35,904 - Bytes in read #36: 1061 , block bytes: 312913, remaining in block: 735663 duration=325436462 nS; ns/ byte : 306726, bandwidth=0.003109 MB/s 2016-12-08 14:23:36,776 - Bytes in read #37: 16347 , block bytes: 329260, remaining in block: 719316 duration=871155772 nS; ns/ byte : 53291, bandwidth=0.017895 MB/s 2016-12-08 14:23:36,777 - Bytes in read #38: 1061 , block bytes: 330321, remaining in block: 718255 duration=241257 nS; ns/ byte : 227, bandwidth=4.194069 MB/s 2016-12-08 14:23:37,838 - Bytes in read #39: 16347 , block bytes: 346668, remaining in block: 701908 duration=1061024046 nS; ns/ byte : 64906, bandwidth=0.014693 MB/s 2016-12-08 14:23:38,014 - Bytes in read #40: 1061 , block bytes: 347729, remaining in block: 700847 duration=175298804 nS; ns/ byte : 165220, bandwidth=0.005772 MB/s 2016-12-08 14:23:38,553 - Bytes in read #41: 16347 , block bytes: 364076, remaining in block: 684500 duration=538434297 nS; ns/ byte : 32937, bandwidth=0.028954 MB/s 2016-12-08 14:23:38,554 - Bytes in read #42: 1061 , block bytes: 365137, remaining in block: 683439 duration=1233409 nS; ns/ byte : 1162, bandwidth=0.820367 MB/s 2016-12-08 14:23:40,229 - Bytes in read #43: 16347 , block bytes: 381484, remaining in block: 667092 duration=1674577082 nS; ns/ byte : 102439, bandwidth=0.009310 MB/s 2016-12-08 14:23:40,586 - Bytes in read #44: 1061 , block bytes: 382545, remaining in block: 666031 duration=357021347 nS; ns/ byte : 336495, bandwidth=0.002834 MB/s 2016-12-08 14:23:41,275 - Bytes in read #45: 16347 , block bytes: 398892, remaining in block: 649684 duration=688294002 nS; ns/ byte : 42105, bandwidth=0.022650 MB/s 2016-12-08 14:23:41,276 - Bytes in read #46: 1061 , block bytes: 399953, remaining in block: 648623 duration=140726 nS; ns/ byte : 132, bandwidth=7.190203 MB/s 2016-12-08 14:23:42,389 - Bytes in read #47: 16347 , block bytes: 416300, remaining in block: 632276 duration=1113488278 nS; ns/ byte : 68115, bandwidth=0.014001 MB/s 2016-12-08 14:23:42,390 - Bytes in read #48: 1061 , block bytes: 417361, remaining in block: 631215 duration=163118 nS; ns/ byte : 153, bandwidth=6.203169 MB/s 2016-12-08 14:23:43,397 - Bytes in read #49: 16347 , block bytes: 433708, remaining in block: 614868 duration=1006992034 nS; ns/ byte : 61601, bandwidth=0.015481 MB/s 2016-12-08 14:23:43,573 - Bytes in read #50: 1061 , block bytes: 434769, remaining in block: 613807 duration=175484587 nS; ns/ byte : 165395, bandwidth=0.005766 MB/s 2016-12-08 14:23:43,927 - Bytes in read #51: 16347 , block bytes: 451116, remaining in block: 597460 duration=353339533 nS; ns/ byte : 21614, bandwidth=0.044121 MB/s 2016-12-08 14:23:44,102 - Bytes in read #52: 1061 , block bytes: 452177, remaining in block: 596399 duration=174634494 nS; ns/ byte : 164594, bandwidth=0.005794 MB/s 2016-12-08 14:23:44,985 - Bytes in read #53: 16347 , block bytes: 468524, remaining in block: 580052 duration=882841721 nS; ns/ byte : 54006, bandwidth=0.017659 MB/s 2016-12-08 14:23:44,986 - Bytes in read #54: 1061 , block bytes: 469585, remaining in block: 578991 duration=168354 nS; ns/ byte : 158, bandwidth=6.010243 MB/s 2016-12-08 14:23:46,337 - Bytes in read #55: 16347 , block bytes: 485932, remaining in block: 562644 duration=1350476638 nS; ns/ byte : 82613, bandwidth=0.011544 MB/s 2016-12-08 14:23:46,337 - Bytes in read #56: 1061 , block bytes: 486993, remaining in block: 561583 duration=168682 nS; ns/ byte : 158, bandwidth=5.998556 MB/s 2016-12-08 14:23:47,058 - Bytes in read #57: 16347 , block bytes: 503340, remaining in block: 545236 duration=720245455 nS; ns/ byte : 44059, bandwidth=0.021645 MB/s 2016-12-08 14:23:47,058 - Bytes in read #58: 1061 , block bytes: 504401, remaining in block: 544175 duration=109153 nS; ns/ byte : 102, bandwidth=9.270001 MB/s 2016-12-08 14:23:48,051 - Bytes in read #59: 16347 , block bytes: 520748, remaining in block: 527828 duration=992763828 nS; ns/ byte : 60730, bandwidth=0.015703 MB/s 2016-12-08 14:23:48,517 - Bytes in read #60: 1061 , block bytes: 521809, remaining in block: 526767 duration=465289883 nS; ns/ byte : 438539, bandwidth=0.002175 MB/s 2016-12-08 14:23:48,851 - Bytes in read #61: 16347 , block bytes: 538156, remaining in block: 510420 duration=333839573 nS; ns/ byte : 20422, bandwidth=0.046698 MB/s 2016-12-08 14:23:49,211 - Bytes in read #62: 1061 , block bytes: 539217, remaining in block: 509359 duration=359559220 nS; ns/ byte : 338887, bandwidth=0.002814 MB/s 2016-12-08 14:23:50,006 - Bytes in read #63: 16347 , block bytes: 555564, remaining in block: 493012 duration=794069765 nS; ns/ byte : 48575, bandwidth=0.019633 MB/s 2016-12-08 14:23:50,006 - Bytes in read #64: 1061 , block bytes: 556625, remaining in block: 491951 duration=116647 nS; ns/ byte : 109, bandwidth=8.674449 MB/s 2016-12-08 14:23:50,896 - Bytes in read #65: 16347 , block bytes: 572972, remaining in block: 475604 duration=889335208 nS; ns/ byte : 54403, bandwidth=0.017530 MB/s 2016-12-08 14:23:51,280 - Bytes in read #66: 1061 , block bytes: 574033, remaining in block: 474543 duration=384066718 nS; ns/ byte : 361985, bandwidth=0.002635 MB/s 2016-12-08 14:23:51,791 - Bytes in read #67: 16347 , block bytes: 590380, remaining in block: 458196 duration=509948788 nS; ns/ byte : 31195, bandwidth=0.030571 MB/s 2016-12-08 14:23:51,791 - Bytes in read #68: 1061 , block bytes: 591441, remaining in block: 457135 duration=168685 nS; ns/ byte : 158, bandwidth=5.998449 MB/s 2016-12-08 14:23:52,326 - Bytes in read #69: 16347 , block bytes: 607788, remaining in block: 440788 duration=534885343 nS; ns/ byte : 32720, bandwidth=0.029146 MB/s 2016-12-08 14:23:52,327 - Bytes in read #70: 1061 , block bytes: 608849, remaining in block: 439727 duration=130685 nS; ns/ byte : 123, bandwidth=7.742652 MB/s 2016-12-08 14:23:53,890 - Bytes in read #71: 16347 , block bytes: 625196, remaining in block: 423380 duration=1563070584 nS; ns/ byte : 95618, bandwidth=0.009974 MB/s
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Patch 001: log time, bytes/ns and bandwidth of every read. This is what's generating the detailed log info in the attachments; it shows how performance can time out and then, on a new HTTP connect, fly

          Show
          stevel@apache.org Steve Loughran added a comment - Patch 001: log time, bytes/ns and bandwidth of every read. This is what's generating the detailed log info in the attachments; it shows how performance can time out and then, on a new HTTP connect, fly
          Hide
          stevel@apache.org Steve Loughran added a comment -

          tested, s3 ireland. All works well except this scale test

          Show
          stevel@apache.org Steve Loughran added a comment - tested, s3 ireland. All works well except this scale test
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 12s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 8m 42s trunk passed
          +1 compile 0m 20s trunk passed
          +1 checkstyle 0m 13s trunk passed
          +1 mvnsite 0m 29s trunk passed
          +1 mvneclipse 0m 25s trunk passed
          +1 findbugs 0m 33s trunk passed
          +1 javadoc 0m 15s trunk passed
          +1 mvninstall 0m 20s the patch passed
          +1 compile 0m 16s the patch passed
          +1 javac 0m 16s the patch passed
          +1 checkstyle 0m 10s the patch passed
          +1 mvnsite 0m 23s the patch passed
          +1 mvneclipse 0m 13s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 0m 34s the patch passed
          +1 javadoc 0m 13s the patch passed
          +1 unit 0m 21s hadoop-aws in the patch passed.
          +1 asflicense 0m 17s The patch does not generate ASF License warnings.
          15m 15s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HADOOP-13871
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12842366/HADOOP-13871-001.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 98b111417282 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 0ef7961
          Default Java 1.8.0_111
          findbugs v3.0.0
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11217/testReport/
          modules C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11217/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 42s trunk passed +1 compile 0m 20s trunk passed +1 checkstyle 0m 13s trunk passed +1 mvnsite 0m 29s trunk passed +1 mvneclipse 0m 25s trunk passed +1 findbugs 0m 33s trunk passed +1 javadoc 0m 15s trunk passed +1 mvninstall 0m 20s the patch passed +1 compile 0m 16s the patch passed +1 javac 0m 16s the patch passed +1 checkstyle 0m 10s the patch passed +1 mvnsite 0m 23s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 34s the patch passed +1 javadoc 0m 13s the patch passed +1 unit 0m 21s hadoop-aws in the patch passed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 15m 15s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-13871 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12842366/HADOOP-13871-001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 98b111417282 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 0ef7961 Default Java 1.8.0_111 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11217/testReport/ modules C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11217/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment - - edited

          Also now seen on trunk. netstat shows the link is up,

          tcp4       0      0  192.168.1.12.55256     s3-us-west-2-r-w.https ESTABLISHED
          

          and nettop shows inaction, though the rx_ooo counter seemed be incrementing at2 2KB/s for a bit, before hanging completely

                                                                          state      packets_in  bytes_in  packets_out bytes_out  rx_dupe    rx_ooo re-tx   rtt_avg   rtt_var   rcvsize    tx_win P C R W
          java.16828                                                                  24502      13 MiB        8       3507 B      37 KiB  4654 KiB 0 B
             tcp4 192.168.1.12:55256<->s3-us-west-2-r-w.amazonaws.com:443 Established 24502      13 MiB        8       3507 B      37 KiB  4654 KiB 0 B   185.31 ms  15.03 ms   256 KiB    21 KiB - - - -
          

          That's 4MB of OOO packets for 13 MB read, symptomatic of routing fun.

          Then, suddenly, that TCP connection got closed (socket timeout) and a new one opened that went through the full dataset in a second or two

                                                                                                         state      packets_in        bytes_in     packets_out       bytes_out   rx_dupe    rx_ooo     re-tx   rtt_avg   rtt_var   rcvsize    tx_win P C R W
          java.16828                                                                                                     41636          37 MiB              25        9210 B      37 KiB  4654 KiB     0 B
             tcp4 192.168.1.12:55256<->s3-us-west-2-r-w.amazonaws.com:443                             FinWait2           24502          13 MiB               9        3560 B      37 KiB  4654 KiB     0 B   184.16 ms  12.44 ms   256 KiB    21 KiB - - - -
          
          

          The really good news: curl is now suffering too. Which means its not a Java problem. Either the latop (which has been rebooted with SMC reset), or the rest of the network.

          
           $ curl -O https://landsat-pds.s3.amazonaws.com/scene_list.gz
            % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                           Dload  Upload   Total   Spent    Left  Speed
            5 37.4M    5 2090k    0     0  11824      0  0:55:21  0:03:01  0:52:20  7039
          
          $ nettop -p 17105
                                                                                                         state      packets_in        bytes_in     packets_out       bytes_out   rx_dupe    rx_ooo     re-tx   rtt_avg   rtt_var   rcvsize    tx_win P C R W
          curl.17105                                                                                                      3178        2323 KiB               4         482 B   10232 B     918 KiB     0 B
             tcp4 192.168.1.12:55731<->s3-us-west-2-w.amazonaws.com:443                            Established            3178        2323 KiB               4         482 B   10232 B     918 KiB     0 B   173.56 ms  20.41 ms   256 KiB    16 KiB - - - -
          
          

          And on another attempt

           curl -O https://landsat-pds.s3.amazonaws.com/scene_list.gz
            % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                           Dload  Upload   Total   Spent    Left  Speed
          100 37.4M  100 37.4M    0     0  4410k      0  0:00:08  0:00:08 --:--:-- 6382k
          
          

          Conclusions

          1. sometimes over a network, we can get awful S3 read performance
          2. which goes away on a reconnect, including those detected by socket timeouts
          3. and which can be seen on other processes, so it's not a JVM/SDK problem
          4. which means that curl can be used as a probe independent of everything else; nettop giving more details

          I'm going to try to set some more aggressive socket timeouts than 200 seconds. If it does address this, maybe we should consider having a smaller default.

          Also: time for that advanced troubleshooting document

          Show
          stevel@apache.org Steve Loughran added a comment - - edited Also now seen on trunk. netstat shows the link is up, tcp4 0 0 192.168.1.12.55256 s3-us-west-2-r-w.https ESTABLISHED and nettop shows inaction, though the rx_ooo counter seemed be incrementing at2 2KB/s for a bit, before hanging completely state packets_in bytes_in packets_out bytes_out rx_dupe rx_ooo re-tx rtt_avg rtt_var rcvsize tx_win P C R W java.16828 24502 13 MiB 8 3507 B 37 KiB 4654 KiB 0 B tcp4 192.168.1.12:55256<->s3-us-west-2-r-w.amazonaws.com:443 Established 24502 13 MiB 8 3507 B 37 KiB 4654 KiB 0 B 185.31 ms 15.03 ms 256 KiB 21 KiB - - - - That's 4MB of OOO packets for 13 MB read, symptomatic of routing fun. Then, suddenly, that TCP connection got closed (socket timeout) and a new one opened that went through the full dataset in a second or two state packets_in bytes_in packets_out bytes_out rx_dupe rx_ooo re-tx rtt_avg rtt_var rcvsize tx_win P C R W java.16828 41636 37 MiB 25 9210 B 37 KiB 4654 KiB 0 B tcp4 192.168.1.12:55256<->s3-us-west-2-r-w.amazonaws.com:443 FinWait2 24502 13 MiB 9 3560 B 37 KiB 4654 KiB 0 B 184.16 ms 12.44 ms 256 KiB 21 KiB - - - - The really good news: curl is now suffering too. Which means its not a Java problem. Either the latop (which has been rebooted with SMC reset), or the rest of the network. $ curl -O https: //landsat-pds.s3.amazonaws.com/scene_list.gz % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 5 37.4M 5 2090k 0 0 11824 0 0:55:21 0:03:01 0:52:20 7039 $ nettop -p 17105 state packets_in bytes_in packets_out bytes_out rx_dupe rx_ooo re-tx rtt_avg rtt_var rcvsize tx_win P C R W curl.17105 3178 2323 KiB 4 482 B 10232 B 918 KiB 0 B tcp4 192.168.1.12:55731<->s3-us-west-2-w.amazonaws.com:443 Established 3178 2323 KiB 4 482 B 10232 B 918 KiB 0 B 173.56 ms 20.41 ms 256 KiB 16 KiB - - - - And on another attempt curl -O https: //landsat-pds.s3.amazonaws.com/scene_list.gz % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 37.4M 100 37.4M 0 0 4410k 0 0:00:08 0:00:08 --:--:-- 6382k Conclusions sometimes over a network, we can get awful S3 read performance which goes away on a reconnect, including those detected by socket timeouts and which can be seen on other processes, so it's not a JVM/SDK problem which means that curl can be used as a probe independent of everything else; nettop giving more details I'm going to try to set some more aggressive socket timeouts than 200 seconds. If it does address this, maybe we should consider having a smaller default. Also: time for that advanced troubleshooting document
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Video appears to show that everything is coming in out of order.
          https://youtu.be/4lJnknNtZNI

          Having a tight timeout and expanded rx buffer isn't enough, as there's enough capacity for OOO packets to be buffered, so less discarded, hence pauses waiting for other bits of data to get resent.

          This almost argues for a smaller Rx buffer so it blockes faster, triggering timeouts. But that's some bits of TCP there that I'm not knowledgeable of. After all: ooo packets do appear to be arriving, so the channel is live, just not delivering data to the caller.

          We could collect effective read stats (as the test does) within the input stream...just do nanotime counts before and after each read, and build up long-term stats of the current stream, which can then be queried. although they could be aggregated, I tried that with the output and it doesn't work: if a network is full due to to many parallel connections, the effective bandwidth of each one is low; aggregating via just total bytes/total elapsed time doesn't work, as it generates false statistics implying very low bandwidth, rather than a saturated, shared, network link

          Show
          stevel@apache.org Steve Loughran added a comment - Video appears to show that everything is coming in out of order. https://youtu.be/4lJnknNtZNI Having a tight timeout and expanded rx buffer isn't enough, as there's enough capacity for OOO packets to be buffered, so less discarded, hence pauses waiting for other bits of data to get resent. This almost argues for a smaller Rx buffer so it blockes faster, triggering timeouts. But that's some bits of TCP there that I'm not knowledgeable of. After all: ooo packets do appear to be arriving, so the channel is live, just not delivering data to the caller. We could collect effective read stats (as the test does) within the input stream...just do nanotime counts before and after each read, and build up long-term stats of the current stream, which can then be queried. although they could be aggregated, I tried that with the output and it doesn't work: if a network is full due to to many parallel connections, the effective bandwidth of each one is low; aggregating via just total bytes/total elapsed time doesn't work, as it generates false statistics implying very low bandwidth, rather than a saturated, shared, network link
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Patch for trunk adds a new resetConnection() operation which aborts any active underlying connection, with this test modified to explicitly detect reads too slow and perform resets, failing if >8 resets happen during the run. This detects and reacts to bandwidth problems, albeit with a hard coded threshold of 128KB/s. If you are getting less than that, you shouldn't be running that test. Except: after the abort the TCP connection will be slow-starting up to full B/W, so on any-long haul link, that next read may also underperform. It's probably safe to have a lower value.

          It would be possible to track bandwidth per

          Example

          2016-12-08 21:29:45,965 [Thread-1] DEBUG s3a.S3AFileSystem (S3AUtils.java:intOption(512)) - Value of fs.s3a.connection.maximum is 25
          2016-12-08 21:29:45,967 [Thread-1] DEBUG s3a.S3AFileSystem (S3AUtils.java:intOption(512)) - Value of fs.s3a.attempts.maximum is 20
          2016-12-08 21:29:45,967 [Thread-1] DEBUG s3a.S3AFileSystem (S3AUtils.java:intOption(512)) - Value of fs.s3a.connection.establish.timeout is 5000
          2016-12-08 21:29:45,968 [Thread-1] DEBUG s3a.S3AFileSystem (S3AUtils.java:intOption(512)) - Value of fs.s3a.connection.timeout is 5000
          2016-12-08 21:29:45,968 [Thread-1] DEBUG s3a.S3AFileSystem (S3AUtils.java:intOption(512)) - Value of fs.s3a.socket.send.buffer is 65536
          2016-12-08 21:29:45,968 [Thread-1] DEBUG s3a.S3AFileSystem (S3AUtils.java:intOption(512)) - Value of fs.s3a.socket.recv.buffer is 32678
          2016-12-08 21:29:45,970 [Thread-1] DEBUG s3a.S3AFileSystem (S3ClientFactory.java:initUserAgent(187)) - Using User-Agent: Hadoop 2.8.0-SNAPSHOT
          2016-12-08 21:29:46,186 [Thread-1] DEBUG s3a.S3AFileSystem (S3AUtils.java:intOption(512)) - Value of fs.s3a.paging.maximum is 5000
          2016-12-08 21:29:46,188 [Thread-1] DEBUG s3a.S3AFileSystem (S3AUtils.java:longBytesOption(555)) - Value of fs.s3a.block.size is 33554432
          2016-12-08 21:29:46,188 [Thread-1] DEBUG s3a.S3AFileSystem (S3AUtils.java:longBytesOption(555)) - Value of fs.s3a.readahead.range is 65536
          2016-12-08 21:29:46,190 [Thread-1] DEBUG s3a.S3AFileSystem (S3AUtils.java:intOption(512)) - Value of fs.s3a.max.total.tasks is 10
          2016-12-08 21:29:46,190 [Thread-1] DEBUG s3a.S3AFileSystem (S3AUtils.java:longOption(533)) - Value of fs.s3a.threads.keepalivetime is 60
          
          
          2016-12-08 21:29:49,070 DEBUG amazonaws.request (AmazonHttpClient.java:logResponseRequestId(856)) - AWS Request ID: 1FEDCD17B6FBB2C8
          2016-12-08 21:29:49,381 DEBUG - Bytes in read #1: 16347 , block bytes: 16347, remaining in block: 1032229 duration=478865598 nS; ns/byte: 29293, bandwidth=0.032556 MB/s
          2016-12-08 21:29:49,382 DEBUG - Bytes in read #2: 630 , block bytes: 16977, remaining in block: 1031599 duration=212331 nS; ns/byte: 337, bandwidth=2.829614 MB/s
          2016-12-08 21:29:49,531 DEBUG - Bytes in read #3: 16347 , block bytes: 33324, remaining in block: 1015252 duration=149393770 nS; ns/byte: 9138, bandwidth=0.104353 MB/s
          2016-12-08 21:29:49,532 DEBUG - Bytes in read #4: 1061 , block bytes: 34385, remaining in block: 1014191 duration=167998 nS; ns/byte: 158, bandwidth=6.022979 MB/s
          2016-12-08 21:29:49,688 DEBUG - Bytes in read #5: 16347 , block bytes: 50732, remaining in block: 997844 duration=155828299 nS; ns/byte: 9532, bandwidth=0.100044 MB/s
          2016-12-08 21:29:49,688 DEBUG - Bytes in read #6: 1061 , block bytes: 51793, remaining in block: 996783 duration=128858 nS; ns/byte: 121, bandwidth=7.852430 MB/s
          2016-12-08 21:29:49,841 DEBUG - Bytes in read #7: 16347 , block bytes: 68140, remaining in block: 980436 duration=152823750 nS; ns/byte: 9348, bandwidth=0.102011 MB/s
          2016-12-08 21:29:49,842 DEBUG - Bytes in read #8: 1061 , block bytes: 69201, remaining in block: 979375 duration=280146 nS; ns/byte: 264, bandwidth=3.611861 MB/s
          2016-12-08 21:29:50,149 DEBUG - Bytes in read #9: 16347 , block bytes: 85548, remaining in block: 963028 duration=306567695 nS; ns/byte: 18753, bandwidth=0.050852 MB/s
          2016-12-08 21:29:50,149 DEBUG - Bytes in read #10: 1061 , block bytes: 86609, remaining in block: 961967 duration=161057 nS; ns/byte: 151, bandwidth=6.282549 MB/s
          2016-12-08 21:29:51,073 DEBUG - Bytes in read #11: 16347 , block bytes: 102956, remaining in block: 945620 duration=923209136 nS; ns/byte: 56475, bandwidth=0.016886 MB/s
          2016-12-08 21:29:51,073 DEBUG - Bytes in read #12: 1061 , block bytes: 104017, remaining in block: 944559 duration=172071 nS; ns/byte: 162, bandwidth=5.880412 MB/s
          2016-12-08 21:29:51,690 DEBUG - Bytes in read #13: 16347 , block bytes: 120364, remaining in block: 928212 duration=615842374 nS; ns/byte: 37673, bandwidth=0.025314 MB/s
          2016-12-08 21:29:51,843 DEBUG - Bytes in read #14: 1061 , block bytes: 121425, remaining in block: 927151 duration=152502578 nS; ns/byte: 143734, bandwidth=0.006635 MB/s
          2016-12-08 21:29:52,816 DEBUG - Bytes in read #15: 16347 , block bytes: 137772, remaining in block: 910804 duration=972690659 nS; ns/byte: 59502, bandwidth=0.016027 MB/s
          2016-12-08 21:29:52,816 DEBUG - Bytes in read #16: 1061 , block bytes: 138833, remaining in block: 909743 duration=154987 nS; ns/byte: 146, bandwidth=6.528602 MB/s
          2016-12-08 21:29:53,889 DEBUG - Bytes in read #17: 16347 , block bytes: 155180, remaining in block: 893396 duration=1072600915 nS; ns/byte: 65614, bandwidth=0.014534 MB/s
          2016-12-08 21:29:53,991 DEBUG - Bytes in read #18: 1061 , block bytes: 156241, remaining in block: 892335 duration=101460691 nS; ns/byte: 95627, bandwidth=0.009973 MB/s
          2016-12-08 21:29:55,165 DEBUG - Bytes in read #19: 16347 , block bytes: 172588, remaining in block: 875988 duration=1173337630 nS; ns/byte: 71776, bandwidth=0.013287 MB/s
          2016-12-08 21:29:55,165 DEBUG - Bytes in read #20: 1061 , block bytes: 173649, remaining in block: 874927 duration=116827 nS; ns/byte: 110, bandwidth=8.661084 MB/s
          2016-12-08 21:29:55,628 DEBUG - Bytes in read #21: 16347 , block bytes: 189996, remaining in block: 858580 duration=462231152 nS; ns/byte: 28276, bandwidth=0.033727 MB/s
          2016-12-08 21:29:55,935 DEBUG - Bytes in read #22: 1061 , block bytes: 191057, remaining in block: 857519 duration=306790128 nS; ns/byte: 289151, bandwidth=0.003298 MB/s
          2016-12-08 21:29:56,785 DEBUG - Bytes in read #23: 16347 , block bytes: 207404, remaining in block: 841172 duration=849333859 nS; ns/byte: 51956, bandwidth=0.018355 MB/s
          2016-12-08 21:29:56,786 DEBUG - Bytes in read #24: 1061 , block bytes: 208465, remaining in block: 840111 duration=223996 nS; ns/byte: 211, bandwidth=4.517261 MB/s
          2016-12-08 21:29:58,197 DEBUG - Bytes in read #25: 16347 , block bytes: 224812, remaining in block: 823764 duration=1411150169 nS; ns/byte: 86324, bandwidth=0.011048 MB/s
          2016-12-08 21:29:58,350 DEBUG - Bytes in read #26: 1061 , block bytes: 225873, remaining in block: 822703 duration=151618968 nS; ns/byte: 142901, bandwidth=0.006674 MB/s
          2016-12-08 21:29:59,275 DEBUG - Bytes in read #27: 16347 , block bytes: 242220, remaining in block: 806356 duration=924832331 nS; ns/byte: 56575, bandwidth=0.016857 MB/s
          2016-12-08 21:29:59,275 DEBUG - Bytes in read #28: 1061 , block bytes: 243281, remaining in block: 805295 duration=157331 nS; ns/byte: 148, bandwidth=6.431336 MB/s
          2016-12-08 21:30:00,197 DEBUG - Bytes in read #29: 16347 , block bytes: 259628, remaining in block: 788948 duration=919177546 nS; ns/byte: 56229, bandwidth=0.016961 MB/s
          2016-12-08 21:30:00,197 DEBUG - Bytes in read #30: 1061 , block bytes: 260689, remaining in block: 787887 duration=159754 nS; ns/byte: 150, bandwidth=6.333791 MB/s
          2016-12-08 21:30:01,116 DEBUG - Bytes in read #31: 16347 , block bytes: 277036, remaining in block: 771540 duration=918850994 nS; ns/byte: 56209, bandwidth=0.016967 MB/s
          2016-12-08 21:30:01,117 DEBUG - Bytes in read #32: 1061 , block bytes: 278097, remaining in block: 770479 duration=153206 nS; ns/byte: 144, bandwidth=6.604496 MB/s
          2016-12-08 21:30:02,040 DEBUG - Bytes in read #33: 16347 , block bytes: 294444, remaining in block: 754132 duration=923043728 nS; ns/byte: 56465, bandwidth=0.016889 MB/s
          2016-12-08 21:30:02,041 DEBUG - Bytes in read #34: 1061 , block bytes: 295505, remaining in block: 753071 duration=152100 nS; ns/byte: 143, bandwidth=6.652521 MB/s
          2016-12-08 21:30:03,119 DEBUG - Bytes in read #35: 16347 , block bytes: 311852, remaining in block: 736724 duration=1077535517 nS; ns/byte: 65916, bandwidth=0.014468 MB/s
          2016-12-08 21:30:03,119 DEBUG - Bytes in read #36: 1061 , block bytes: 312913, remaining in block: 735663 duration=114098 nS; ns/byte: 107, bandwidth=8.868240 MB/s
          2016-12-08 21:30:04,140 DEBUG - Bytes in read #37: 16347 , block bytes: 329260, remaining in block: 719316 duration=1020943631 nS; ns/byte: 62454, bandwidth=0.015270 MB/s
          2016-12-08 21:30:04,141 DEBUG - Bytes in read #38: 1061 , block bytes: 330321, remaining in block: 718255 duration=154676 nS; ns/byte: 145, bandwidth=6.541729 MB/s
          2016-12-08 21:30:05,063 DEBUG - Bytes in read #39: 16347 , block bytes: 346668, remaining in block: 701908 duration=921944703 nS; ns/byte: 56398, bandwidth=0.016910 MB/s
          2016-12-08 21:30:06,027 DEBUG - Bytes in read #40: 1061 , block bytes: 347729, remaining in block: 700847 duration=963349047 nS; ns/byte: 907963, bandwidth=0.001050 MB/s
          2016-12-08 21:30:07,488 DEBUG - Bytes in read #41: 16347 , block bytes: 364076, remaining in block: 684500 duration=1460036513 nS; ns/byte: 89315, bandwidth=0.010678 MB/s
          2016-12-08 21:30:07,795 DEBUG - Bytes in read #42: 1061 , block bytes: 365137, remaining in block: 683439 duration=306386850 nS; ns/byte: 288771, bandwidth=0.003303 MB/s
          2016-12-08 21:30:08,130 DEBUG - Bytes in read #43: 16347 , block bytes: 381484, remaining in block: 667092 duration=335450550 nS; ns/byte: 20520, bandwidth=0.046474 MB/s
          2016-12-08 21:30:08,262 DEBUG - Bytes in read #44: 1061 , block bytes: 382545, remaining in block: 666031 duration=130994948 nS; ns/byte: 123463, bandwidth=0.007724 MB/s
          2016-12-08 21:30:08,595 DEBUG - Bytes in read #45: 16347 , block bytes: 398892, remaining in block: 649684 duration=333166901 nS; ns/byte: 20380, bandwidth=0.046793 MB/s
          2016-12-08 21:30:08,596 DEBUG - Bytes in read #46: 1061 , block bytes: 399953, remaining in block: 648623 duration=131882 nS; ns/byte: 124, bandwidth=7.672377 MB/s
          2016-12-08 21:30:09,211 DEBUG - Bytes in read #47: 16347 , block bytes: 416300, remaining in block: 632276 duration=614798623 nS; ns/byte: 37609, bandwidth=0.025357 MB/s
          2016-12-08 21:30:09,211 DEBUG - Bytes in read #48: 1061 , block bytes: 417361, remaining in block: 631215 duration=186043 nS; ns/byte: 175, bandwidth=5.438788 MB/s
          2016-12-08 21:30:09,674 DEBUG - Bytes in read #49: 16347 , block bytes: 433708, remaining in block: 614868 duration=462221834 nS; ns/byte: 28275, bandwidth=0.033728 MB/s
          2016-12-08 21:30:09,674 DEBUG - Bytes in read #50: 1061 , block bytes: 434769, remaining in block: 613807 duration=124757 nS; ns/byte: 117, bandwidth=8.110555 MB/s
          2016-12-08 21:30:09,984 DEBUG - Bytes in read #51: 16347 , block bytes: 451116, remaining in block: 597460 duration=308999007 nS; ns/byte: 18902, bandwidth=0.050452 MB/s
          2016-12-08 21:30:09,984 DEBUG - Bytes in read #52: 1061 , block bytes: 452177, remaining in block: 596399 duration=84490 nS; ns/byte: 79, bandwidth=11.975955 MB/s
          2016-12-08 21:30:10,597 DEBUG - Bytes in read #53: 16347 , block bytes: 468524, remaining in block: 580052 duration=612539269 nS; ns/byte: 37471, bandwidth=0.025451 MB/s
          2016-12-08 21:30:10,597 DEBUG - Bytes in read #54: 1061 , block bytes: 469585, remaining in block: 578991 duration=147609 nS; ns/byte: 139, bandwidth=6.854924 MB/s
          2016-12-08 21:30:11,060 DEBUG - Bytes in read #55: 16347 , block bytes: 485932, remaining in block: 562644 duration=462436225 nS; ns/byte: 28288, bandwidth=0.033712 MB/s
          2016-12-08 21:30:11,060 DEBUG - Bytes in read #56: 1061 , block bytes: 486993, remaining in block: 561583 duration=100576 nS; ns/byte: 94, bandwidth=10.060536 MB/s
          2016-12-08 21:30:11,931 DEBUG - Bytes in read #57: 16347 , block bytes: 503340, remaining in block: 545236 duration=870489198 nS; ns/byte: 53250, bandwidth=0.017909 MB/s
          2016-12-08 21:30:11,981 DEBUG - Bytes in read #58: 1061 , block bytes: 504401, remaining in block: 544175 duration=49541837 nS; ns/byte: 46693, bandwidth=0.020424 MB/s
          2016-12-08 21:30:13,060 DEBUG - Bytes in read #59: 16347 , block bytes: 520748, remaining in block: 527828 duration=1078170319 nS; ns/byte: 65955, bandwidth=0.014459 MB/s
          2016-12-08 21:30:13,060 DEBUG - Bytes in read #60: 1061 , block bytes: 521809, remaining in block: 526767 duration=88905 nS; ns/byte: 83, bandwidth=11.381232 MB/s
          2016-12-08 21:30:13,832 DEBUG - Bytes in read #61: 16347 , block bytes: 538156, remaining in block: 510420 duration=771710163 nS; ns/byte: 47208, bandwidth=0.020202 MB/s
          2016-12-08 21:30:13,833 DEBUG - Bytes in read #62: 1061 , block bytes: 539217, remaining in block: 509359 duration=195473 nS; ns/byte: 184, bandwidth=5.176410 MB/s
          2016-12-08 21:30:14,985 DEBUG - Bytes in read #63: 16347 , block bytes: 555564, remaining in block: 493012 duration=1152517210 nS; ns/byte: 70503, bandwidth=0.013527 MB/s
          2016-12-08 21:30:14,986 DEBUG - Bytes in read #64: 1061 , block bytes: 556625, remaining in block: 491951 duration=137869 nS; ns/byte: 129, bandwidth=7.339202 MB/s
          2016-12-08 21:30:15,756 DEBUG - Bytes in read #65: 16347 , block bytes: 572972, remaining in block: 475604 duration=770143725 nS; ns/byte: 47112, bandwidth=0.020243 MB/s
          2016-12-08 21:30:15,757 DEBUG - Bytes in read #66: 1061 , block bytes: 574033, remaining in block: 474543 duration=161316 nS; ns/byte: 152, bandwidth=6.272462 MB/s
          2016-12-08 21:30:16,907 DEBUG - Bytes in read #67: 16347 , block bytes: 590380, remaining in block: 458196 duration=1149954555 nS; ns/byte: 70346, bandwidth=0.013557 MB/s
          2016-12-08 21:30:16,908 DEBUG - Bytes in read #68: 1061 , block bytes: 591441, remaining in block: 457135 duration=166415 nS; ns/byte: 156, bandwidth=6.080272 MB/s
          2016-12-08 21:30:17,776 DEBUG - Bytes in read #69: 16347 , block bytes: 607788, remaining in block: 440788 duration=867495557 nS; ns/byte: 53067, bandwidth=0.017971 MB/s
          2016-12-08 21:30:17,777 DEBUG - Bytes in read #70: 1061 , block bytes: 608849, remaining in block: 439727 duration=938146 nS; ns/byte: 884, bandwidth=1.078562 MB/s
          2016-12-08 21:30:18,948 DEBUG - Bytes in read #71: 16347 , block bytes: 625196, remaining in block: 423380 duration=1170544837 nS; ns/byte: 71606, bandwidth=0.013318 MB/s
          2016-12-08 21:30:19,256 DEBUG - Bytes in read #72: 1061 , block bytes: 626257, remaining in block: 422319 duration=307536130 nS; ns/byte: 289854, bandwidth=0.003290 MB/s
          2016-12-08 21:30:19,721 DEBUG - Bytes in read #73: 16347 , block bytes: 642604, remaining in block: 405972 duration=464770187 nS; ns/byte: 28431, bandwidth=0.033543 MB/s
          2016-12-08 21:30:19,721 DEBUG - Bytes in read #74: 1061 , block bytes: 643665, remaining in block: 404911 duration=145775 nS; ns/byte: 137, bandwidth=6.941166 MB/s
          2016-12-08 21:30:20,030 DEBUG - Bytes in read #75: 16347 , block bytes: 660012, remaining in block: 388564 duration=307973423 nS; ns/byte: 18839, bandwidth=0.050620 MB/s
          2016-12-08 21:30:20,338 DEBUG - Bytes in read #76: 1061 , block bytes: 661073, remaining in block: 387503 duration=307555981 nS; ns/byte: 289873, bandwidth=0.003290 MB/s
          2016-12-08 21:30:20,798 DEBUG - Bytes in read #77: 16347 , block bytes: 677420, remaining in block: 371156 duration=459891175 nS; ns/byte: 28133, bandwidth=0.033899 MB/s
          2016-12-08 21:30:20,951 DEBUG - Bytes in read #78: 1061 , block bytes: 678481, remaining in block: 370095 duration=152066369 nS; ns/byte: 143323, bandwidth=0.006654 MB/s
          2016-12-08 21:30:21,414 DEBUG - Bytes in read #79: 16347 , block bytes: 694828, remaining in block: 353748 duration=463288362 nS; ns/byte: 28340, bandwidth=0.033650 MB/s
          2016-12-08 21:30:21,565 DEBUG - Bytes in read #80: 1061 , block bytes: 695889, remaining in block: 352687 duration=150431750 nS; ns/byte: 141782, bandwidth=0.006726 MB/s
          2016-12-08 21:30:22,183 DEBUG - Bytes in read #81: 16347 , block bytes: 712236, remaining in block: 336340 duration=617356332 nS; ns/byte: 37765, bandwidth=0.025252 MB/s
          2016-12-08 21:30:22,183 DEBUG - Bytes in read #82: 1061 , block bytes: 713297, remaining in block: 335279 duration=91692 nS; ns/byte: 86, bandwidth=11.035297 MB/s
          2016-12-08 21:30:23,223 DEBUG - Bytes in read #83: 16347 , block bytes: 729644, remaining in block: 318932 duration=1039467778 nS; ns/byte: 63587, bandwidth=0.014998 MB/s
          2016-12-08 21:30:23,223 DEBUG - Bytes in read #84: 1061 , block bytes: 730705, remaining in block: 317871 duration=119170 nS; ns/byte: 112, bandwidth=8.490798 MB/s
          2016-12-08 21:30:23,687 DEBUG - Bytes in read #85: 16347 , block bytes: 747052, remaining in block: 301524 duration=463642449 nS; ns/byte: 28362, bandwidth=0.033624 MB/s
          2016-12-08 21:30:23,996 DEBUG - Bytes in read #86: 1061 , block bytes: 748113, remaining in block: 300463 duration=308626480 nS; ns/byte: 290882, bandwidth=0.003279 MB/s
          2016-12-08 21:30:24,763 DEBUG - Bytes in read #87: 16347 , block bytes: 764460, remaining in block: 284116 duration=766976410 nS; ns/byte: 46918, bandwidth=0.020326 MB/s
          2016-12-08 21:30:24,764 DEBUG - Bytes in read #88: 1061 , block bytes: 765521, remaining in block: 283055 duration=94835 nS; ns/byte: 89, bandwidth=10.669568 MB/s
          2016-12-08 21:30:26,695 DEBUG - Bytes in read #89: 16347 , block bytes: 781868, remaining in block: 266708 duration=1930730977 nS; ns/byte: 118109, bandwidth=0.008075 MB/s
          2016-12-08 21:30:27,003 DEBUG - Bytes in read #90: 1061 , block bytes: 782929, remaining in block: 265647 duration=307614364 nS; ns/byte: 289928, bandwidth=0.003289 MB/s
          2016-12-08 21:30:27,465 DEBUG - Bytes in read #91: 16347 , block bytes: 799276, remaining in block: 249300 duration=461784239 nS; ns/byte: 28248, bandwidth=0.033760 MB/s
          2016-12-08 21:30:27,465 DEBUG - Bytes in read #92: 1061 , block bytes: 800337, remaining in block: 248239 duration=62419 nS; ns/byte: 58, bandwidth=16.210584 MB/s
          2016-12-08 21:30:28,333 DEBUG - Bytes in read #93: 16347 , block bytes: 816684, remaining in block: 231892 duration=868005446 nS; ns/byte: 53098, bandwidth=0.017960 MB/s
          2016-12-08 21:30:28,895 DEBUG - Bytes in read #94: 1061 , block bytes: 817745, remaining in block: 230831 duration=560647267 nS; ns/byte: 528414, bandwidth=0.001805 MB/s
          2016-12-08 21:30:29,762 DEBUG - Bytes in read #95: 16347 , block bytes: 834092, remaining in block: 214484 duration=866587960 nS; ns/byte: 53012, bandwidth=0.017990 MB/s
          2016-12-08 21:30:29,864 DEBUG - Bytes in read #96: 1061 , block bytes: 835153, remaining in block: 213423 duration=101564552 nS; ns/byte: 95725, bandwidth=0.009963 MB/s
          2016-12-08 21:30:30,379 DEBUG - Bytes in read #97: 16347 , block bytes: 851500, remaining in block: 197076 duration=514754938 nS; ns/byte: 31489, bandwidth=0.030286 MB/s
          2016-12-08 21:30:30,688 DEBUG - Bytes in read #98: 1061 , block bytes: 852561, remaining in block: 196015 duration=309420585 nS; ns/byte: 291631, bandwidth=0.003270 MB/s
          2016-12-08 21:30:31,457 DEBUG - Bytes in read #99: 16347 , block bytes: 868908, remaining in block: 179668 duration=768580519 nS; ns/byte: 47016, bandwidth=0.020284 MB/s
          2016-12-08 21:30:31,458 DEBUG - Bytes in read #100: 1061 , block bytes: 869969, remaining in block: 178607 duration=110392 nS; ns/byte: 104, bandwidth=9.165958 MB/s
          2016-12-08 21:30:32,225 DEBUG - Bytes in read #101: 16347 , block bytes: 886316, remaining in block: 162260 duration=767381785 nS; ns/byte: 46943, bandwidth=0.020315 MB/s
          2016-12-08 21:30:32,226 DEBUG - Bytes in read #102: 1061 , block bytes: 887377, remaining in block: 161199 duration=182729 nS; ns/byte: 172, bandwidth=5.537427 MB/s
          2016-12-08 21:30:33,095 DEBUG - Bytes in read #103: 16347 , block bytes: 903724, remaining in block: 144852 duration=869101542 nS; ns/byte: 53165, bandwidth=0.017938 MB/s
          2016-12-08 21:30:33,247 DEBUG - Bytes in read #104: 1061 , block bytes: 904785, remaining in block: 143791 duration=151712419 nS; ns/byte: 142990, bandwidth=0.006670 MB/s
          2016-12-08 21:30:33,711 DEBUG - Bytes in read #105: 16347 , block bytes: 921132, remaining in block: 127444 duration=463346639 nS; ns/byte: 28344, bandwidth=0.033646 MB/s
          2016-12-08 21:30:33,864 DEBUG - Bytes in read #106: 1061 , block bytes: 922193, remaining in block: 126383 duration=152830422 nS; ns/byte: 144043, bandwidth=0.006621 MB/s
          2016-12-08 21:30:34,641 DEBUG - Bytes in read #107: 16347 , block bytes: 938540, remaining in block: 110036 duration=777184061 nS; ns/byte: 47542, bandwidth=0.020059 MB/s
          2016-12-08 21:30:34,642 DEBUG - Bytes in read #108: 1061 , block bytes: 939601, remaining in block: 108975 duration=92668 nS; ns/byte: 87, bandwidth=10.919071 MB/s
          2016-12-08 21:30:35,655 DEBUG - Bytes in read #109: 16347 , block bytes: 955948, remaining in block: 92628 duration=1013110625 nS; ns/byte: 61975, bandwidth=0.015388 MB/s
          2016-12-08 21:30:35,810 DEBUG - Bytes in read #110: 1061 , block bytes: 957009, remaining in block: 91567 duration=154218951 nS; ns/byte: 145352, bandwidth=0.006561 MB/s
          2016-12-08 21:30:36,275 DEBUG - Bytes in read #111: 16347 , block bytes: 973356, remaining in block: 75220 duration=465093303 nS; ns/byte: 28451, bandwidth=0.033520 MB/s
          2016-12-08 21:30:36,276 DEBUG - Bytes in read #112: 1061 , block bytes: 974417, remaining in block: 74159 duration=171536 nS; ns/byte: 161, bandwidth=5.898753 MB/s
          2016-12-08 21:30:36,745 DEBUG - Bytes in read #113: 16347 , block bytes: 990764, remaining in block: 57812 duration=468606455 nS; ns/byte: 28666, bandwidth=0.033268 MB/s
          2016-12-08 21:30:36,745 DEBUG - Bytes in read #114: 1061 , block bytes: 991825, remaining in block: 56751 duration=91092 nS; ns/byte: 85, bandwidth=11.107984 MB/s
          2016-12-08 21:30:37,206 DEBUG - Bytes in read #115: 16347 , block bytes: 1008172, remaining in block: 40404 duration=460276607 nS; ns/byte: 28156, bandwidth=0.033870 MB/s
          2016-12-08 21:30:37,514 DEBUG - Bytes in read #116: 1061 , block bytes: 1009233, remaining in block: 39343 duration=307705385 nS; ns/byte: 290014, bandwidth=0.003288 MB/s
          2016-12-08 21:30:37,975 DEBUG - Bytes in read #117: 16347 , block bytes: 1025580, remaining in block: 22996 duration=461043201 nS; ns/byte: 28203, bandwidth=0.033814 MB/s
          2016-12-08 21:30:37,975 DEBUG - Bytes in read #118: 1061 , block bytes: 1026641, remaining in block: 21935 duration=74629 nS; ns/byte: 70, bandwidth=13.558381 MB/s
          2016-12-08 21:30:38,745 DEBUG - Bytes in read #119: 16347 , block bytes: 1042988, remaining in block: 5588 duration=768900663 nS; ns/byte: 47036, bandwidth=0.020275 MB/s
          2016-12-08 21:30:38,745 DEBUG - Bytes in read #120: 1061 , block bytes: 1044049, remaining in block: 4527 duration=161553 nS; ns/byte: 152, bandwidth=6.263260 MB/s
          2016-12-08 21:30:39,362 DEBUG - Bytes in read #121: 4527 , block bytes: 1048576, remaining in block: 0 duration=616755333 nS; ns/byte: 136239, bandwidth=0.007000 MB/s
          2016-12-08 21:30:39,363 INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 1 in 121 reads: 50,463,661,382 nS
          2016-12-08 21:30:39,363 INFO  (testTimeToOpenAndReadWholeFileBlocks(254)) - Bandwidth of block 1: 0.019816 MB/s: 
          2016-12-08 21:30:39,363 WARN  (testTimeToOpenAndReadWholeFileBlocks(256)) - Bandwidth 0.019816 too low on block 1: resetting connection
          2016-12-08 21:30:39,366 DEBUG conn.DefaultClientConnection (DefaultClientConnection.java:shutdown(166)) - Connection 0.0.0.0:57950<->54.231.169.2:443 shut down
          2016-12-08 21:30:39,366 DEBUG conn.DefaultClientConnection (DefaultClientConnection.java:close(182)) - Connection 0.0.0.0:57950<->54.231.169.2:443 closed
          2016-12-08 21:30:39,366 DEBUG conn.PoolingClientConnectionManager (PoolingClientConnectionManager.java:releaseConnection(274)) - Connection released: [id: 1][route: {s}->https://landsat-pds.s3.amazonaws.com:443][total kept alive: 0; route allocated: 0 of 25; total allocated: 0 of 25]
          2016-12-08 21:30:39,366 DEBUG s3a.S3AFileSystem (S3AInputStream.java:closeStream(463)) - Stream s3a://landsat-pds/scene_list.gz aborted: reset(); streamPos=1048576, nextReadPos=1048576, request range 0-39278765 length=39278765
          2016-12-08 21:30:39,367 DEBUG s3a.S3AFileSystem (S3AInputStream.java:reopen(140)) - reopen(s3a://landsat-pds/scene_list.gz) for read from new offset range[1048576-39278765], length=1048576, streamPosition=1048576, nextReadPosition=1048576
          2016-12-08 21:30:39,367 DEBUG amazonaws.request (AmazonHttpClient.java:executeOneRequest(671)) - Sending Request: GET https://landsat-pds.s3.amazonaws.com /scene_list.gz Headers: (User-Agent: Hadoop 2.8.0-SNAPSHOT, aws-sdk-java/1.10.6 Mac_OS_X/10.12.1 Java_HotSpot(TM)_64-Bit_Server_VM/25.102-b14/1.8.0_102, Range: bytes=1048576-39278765, Content-Type: application/x-www-form-urlencoded; charset=utf-8, ) 
          2016-12-08 21:30:39,367 DEBUG conn.PoolingClientConnectionManager (PoolingClientConnectionManager.java:requestConnection(184)) - Connection request: [route: {s}->https://landsat-pds.s3.amazonaws.com:443][total kept alive: 0; route allocated: 0 of 25; total allocated: 0 of 25]
          2016-12-08 21:30:39,368 DEBUG conn.PoolingClientConnectionManager (PoolingClientConnectionManager.java:leaseConnection(218)) - Connection leased: [id: 2][route: {s}->https://landsat-pds.s3.amazonaws.com:443][total kept alive: 0; route allocated: 1 of 25; total allocated: 1 of 25]
          2016-12-08 21:30:39,394 DEBUG conn.DefaultClientConnectionOperator (DefaultClientConnectionOperator.java:openConnection(176)) - Connecting to landsat-pds.s3.amazonaws.com:443
          2016-12-08 21:30:40,040 DEBUG protocol.RequestAddCookies (RequestAddCookies.java:process(122)) - CookieSpec selected: default
          2016-12-08 21:30:40,040 DEBUG protocol.RequestAuthCache (RequestAuthCache.java:process(76)) - Auth cache not set in the context
          2016-12-08 21:30:40,040 DEBUG protocol.RequestProxyAuthentication (RequestProxyAuthentication.java:process(88)) - Proxy auth state: UNCHALLENGED
          2016-12-08 21:30:40,040 DEBUG conn.DefaultClientConnection (DefaultClientConnection.java:sendRequestHeader(276)) - Sending request: GET /scene_list.gz HTTP/1.1
          2016-12-08 21:30:40,041 DEBUG http.headers (DefaultClientConnection.java:sendRequestHeader(280)) - >> GET /scene_list.gz HTTP/1.1
          2016-12-08 21:30:40,041 DEBUG http.headers (DefaultClientConnection.java:sendRequestHeader(283)) - >> Host: landsat-pds.s3.amazonaws.com
          2016-12-08 21:30:40,041 DEBUG http.headers (DefaultClientConnection.java:sendRequestHeader(283)) - >> Authorization: AWS AKIAIYZ5JQOW3N5H6NPA:3c0hXBn3xs2erSymklPv8hV1MKM=
          2016-12-08 21:30:40,041 DEBUG http.headers (DefaultClientConnection.java:sendRequestHeader(283)) - >> User-Agent: Hadoop 2.8.0-SNAPSHOT, aws-sdk-java/1.10.6 Mac_OS_X/10.12.1 Java_HotSpot(TM)_64-Bit_Server_VM/25.102-b14/1.8.0_102
          2016-12-08 21:30:40,041 DEBUG http.headers (DefaultClientConnection.java:sendRequestHeader(283)) - >> Range: bytes=1048576-39278765
          2016-12-08 21:30:40,041 DEBUG http.headers (DefaultClientConnection.java:sendRequestHeader(283)) - >> Date: Thu, 08 Dec 2016 21:30:39 GMT
          2016-12-08 21:30:40,041 DEBUG http.headers (DefaultClientConnection.java:sendRequestHeader(283)) - >> Content-Type: application/x-www-form-urlencoded; charset=utf-8
          2016-12-08 21:30:40,041 DEBUG http.headers (DefaultClientConnection.java:sendRequestHeader(283)) - >> Connection: Keep-Alive
          2016-12-08 21:30:40,210 DEBUG conn.DefaultClientConnection (DefaultClientConnection.java:receiveResponseHeader(261)) - Receiving response: HTTP/1.1 206 Partial Content
          2016-12-08 21:30:40,210 DEBUG http.headers (DefaultClientConnection.java:receiveResponseHeader(264)) - << HTTP/1.1 206 Partial Content
          2016-12-08 21:30:40,211 DEBUG http.headers (DefaultClientConnection.java:receiveResponseHeader(267)) - << x-amz-id-2: qKwcRkY1K8w22XWihP8NPtJWfcVbmX7jADNiNYk53DXSsbYOAxUobZwEpIPt0TN27ru9TBJNNA0=
          2016-12-08 21:30:40,211 DEBUG http.headers (DefaultClientConnection.java:receiveResponseHeader(267)) - << x-amz-request-id: CA85E25E70613150
          2016-12-08 21:30:40,211 DEBUG http.headers (DefaultClientConnection.java:receiveResponseHeader(267)) - << Date: Thu, 08 Dec 2016 21:30:41 GMT
          2016-12-08 21:30:40,211 DEBUG http.headers (DefaultClientConnection.java:receiveResponseHeader(267)) - << Last-Modified: Thu, 08 Dec 2016 17:21:24 GMT
          2016-12-08 21:30:40,211 DEBUG http.headers (DefaultClientConnection.java:receiveResponseHeader(267)) - << ETag: "6efe666c9bc2affbdba4f43dce9d0d55"
          2016-12-08 21:30:40,211 DEBUG http.headers (DefaultClientConnection.java:receiveResponseHeader(267)) - << Accept-Ranges: bytes
          2016-12-08 21:30:40,211 DEBUG http.headers (DefaultClientConnection.java:receiveResponseHeader(267)) - << Content-Range: bytes 1048576-39278764/39278765
          2016-12-08 21:30:40,211 DEBUG http.headers (DefaultClientConnection.java:receiveResponseHeader(267)) - << Content-Type: application/octet-stream
          2016-12-08 21:30:40,211 DEBUG http.headers (DefaultClientConnection.java:receiveResponseHeader(267)) - << Content-Length: 38230189
          2016-12-08 21:30:40,213 DEBUG http.headers (DefaultClientConnection.java:receiveResponseHeader(267)) - << Server: AmazonS3
          2016-12-08 21:30:40,213 DEBUG amazonaws.request (AmazonHttpClient.java:logHeaderRequestId(836)) - x-amzn-RequestId: not available
          2016-12-08 21:30:40,214 DEBUG amazonaws.request (AmazonHttpClient.java:handleResponse(1091)) - Received successful response: 206, AWS Request ID: CA85E25E70613150
          2016-12-08 21:30:40,214 DEBUG amazonaws.request (AmazonHttpClient.java:logResponseRequestId(856)) - AWS Request ID: CA85E25E70613150
          2016-12-08 21:30:40,364 DEBUG - Bytes in read #1: 16347 , block bytes: 16347, remaining in block: 1032229 duration=997179473 nS; ns/byte: 61000, bandwidth=0.015634 MB/s
          2016-12-08 21:30:40,365 DEBUG - Bytes in read #2: 624 , block bytes: 16971, remaining in block: 1031605 duration=101431 nS; ns/byte: 162, bandwidth=5.866971 MB/s
          2016-12-08 21:30:40,368 DEBUG - Bytes in read #3: 16347 , block bytes: 33318, remaining in block: 1015258 duration=3318034 nS; ns/byte: 202, bandwidth=4.698479 MB/s
          2016-12-08 21:30:40,368 DEBUG - Bytes in read #4: 1061 , block bytes: 34379, remaining in block: 1014197 duration=82636 nS; ns/byte: 77, bandwidth=12.244645 MB/s
          2016-12-08 21:30:40,519 DEBUG - Bytes in read #5: 16347 , block bytes: 50726, remaining in block: 997850 duration=150854582 nS; ns/byte: 9228, bandwidth=0.103343 MB/s
          2016-12-08 21:30:40,520 DEBUG - Bytes in read #6: 1061 , block bytes: 51787, remaining in block: 996789 duration=76297 nS; ns/byte: 71, bandwidth=13.261969 MB/s
          2016-12-08 21:30:40,522 DEBUG - Bytes in read #7: 16347 , block bytes: 68134, remaining in block: 980442 duration=2337916 nS; ns/byte: 143, bandwidth=6.668210 MB/s
          2016-12-08 21:30:40,523 DEBUG - Bytes in read #8: 1061 , block bytes: 69195, remaining in block: 979381 duration=687085 nS; ns/byte: 647, bandwidth=1.472669 MB/s
          2016-12-08 21:30:40,672 DEBUG - Bytes in read #9: 16347 , block bytes: 85542, remaining in block: 963034 duration=148946723 nS; ns/byte: 9111, bandwidth=0.104666 MB/s
          2016-12-08 21:30:40,673 DEBUG - Bytes in read #10: 1061 , block bytes: 86603, remaining in block: 961973 duration=88030 nS; ns/byte: 82, bandwidth=11.494359 MB/s
          2016-12-08 21:30:40,675 DEBUG - Bytes in read #11: 16347 , block bytes: 102950, remaining in block: 945626 duration=1850448 nS; ns/byte: 113, bandwidth=8.424832 MB/s
          2016-12-08 21:30:40,675 DEBUG - Bytes in read #12: 1061 , block bytes: 104011, remaining in block: 944565 duration=80706 nS; ns/byte: 76, bandwidth=12.537463 MB/s
          2016-12-08 21:30:40,677 DEBUG - Bytes in read #13: 16347 , block bytes: 120358, remaining in block: 928218 duration=1939042 nS; ns/byte: 118, bandwidth=8.039905 MB/s
          2016-12-08 21:30:40,677 DEBUG - Bytes in read #14: 1061 , block bytes: 121419, remaining in block: 927157 duration=70915 nS; ns/byte: 66, bandwidth=14.268469 MB/s
          2016-12-08 21:30:40,824 DEBUG - Bytes in read #15: 16347 , block bytes: 137766, remaining in block: 910810 duration=146688052 nS; ns/byte: 8973, bandwidth=0.106278 MB/s
          2016-12-08 21:30:40,825 DEBUG - Bytes in read #16: 1061 , block bytes: 138827, remaining in block: 909749 duration=135687 nS; ns/byte: 127, bandwidth=7.457225 MB/s
          2016-12-08 21:30:40,827 DEBUG - Bytes in read #17: 16347 , block bytes: 155174, remaining in block: 893402 duration=2159648 nS; ns/byte: 132, bandwidth=7.218637 MB/s
          2016-12-08 21:30:40,828 DEBUG - Bytes in read #18: 1061 , block bytes: 156235, remaining in block: 892341 duration=81945 nS; ns/byte: 77, bandwidth=12.347897 MB/s
          2016-12-08 21:30:40,830 DEBUG - Bytes in read #19: 16347 , block bytes: 172582, remaining in block: 875994 duration=1960358 nS; ns/byte: 119, bandwidth=7.952483 MB/s
          2016-12-08 21:30:40,830 DEBUG - Bytes in read #20: 1061 , block bytes: 173643, remaining in block: 874933 duration=80031 nS; ns/byte: 75, bandwidth=12.643206 MB/s
          2016-12-08 21:30:40,833 DEBUG - Bytes in read #21: 16347 , block bytes: 189990, remaining in block: 858586 duration=2730078 nS; ns/byte: 167, bandwidth=5.710355 MB/s
          2016-12-08 21:30:40,833 DEBUG - Bytes in read #22: 1061 , block bytes: 191051, remaining in block: 857525 duration=82210 nS; ns/byte: 77, bandwidth=12.308095 MB/s
          2016-12-08 21:30:40,836 DEBUG - Bytes in read #23: 16347 , block bytes: 207398, remaining in block: 841178 duration=2466188 nS; ns/byte: 150, bandwidth=6.321381 MB/s
          2016-12-08 21:30:40,836 DEBUG - Bytes in read #24: 1061 , block bytes: 208459, remaining in block: 840117 duration=75878 nS; ns/byte: 71, bandwidth=13.335202 MB/s
          
          ...
          2016-12-08 21:30:41,456 DEBUG - Bytes in read #118: 1061 , block bytes: 1026635, remaining in block: 21941 duration=69393 nS; ns/byte: 65, bandwidth=14.581420 MB/s
          2016-12-08 21:30:41,459 DEBUG - Bytes in read #119: 16347 , block bytes: 1042982, remaining in block: 5594 duration=2944939 nS; ns/byte: 180, bandwidth=5.293731 MB/s
          2016-12-08 21:30:41,459 DEBUG - Bytes in read #120: 1061 , block bytes: 1044043, remaining in block: 4533 duration=68294 nS; ns/byte: 64, bandwidth=14.816067 MB/s
          2016-12-08 21:30:41,462 DEBUG - Bytes in read #121: 4533 , block bytes: 1048576, remaining in block: 0 duration=3124460 nS; ns/byte: 689, bandwidth=1.383601 MB/s
          2016-12-08 21:30:41,463 INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 2 in 121 reads: 2,095,902,133 nS
          2016-12-08 21:30:41,911 INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 3 in 121 reads: 447,989,141 nS
          2016-12-08 21:30:41,911 INFO  (testTimeToOpenAndReadWholeFileBlocks(254)) - Bandwidth of block 3: 2.232197 MB/s: 
          2016-12-08 21:30:42,212 INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 4 in 121 reads: 300,567,979 nS
          2016-12-08 21:30:42,212 INFO  (testTimeToOpenAndReadWholeFileBlocks(254)) - Bandwidth of block 4: 3.327034 MB/s: 
          2016-12-08 21:30:42,445 INFO  (testTimeToOpenAndReadWholeFileBlocks(254)) - Bandwidth of block 5: 4.304842 MB/s: 
          2016-12-08 21:30:42,641 INFO  (testTimeToOpenAndReadWholeFileBlocks(254)) - Bandwidth of block 6: 5.102818 MB/s: 
          2016-12-08 21:30:43,709 INFO  (testTimeToOpenAndReadWholeFileBlocks(254)) - Bandwidth of block 8: 1.389905 MB/s: 
          2016-12-08 21:30:44,705 INFO  (testTimeToOpenAndReadWholeFileBlocks(254)) - Bandwidth of block 9: 1.003957 MB/s: 
          2016-12-08 21:30:45,020 INFO  (testTimeToOpenAndReadWholeFileBlocks(254)) - Bandwidth of block 10: 3.184452 MB/s: 
          2016-12-08 21:30:45,269 INFO  (testTimeToOpenAndReadWholeFileBlocks(254)) - Bandwidth of block 11: 4.012955 MB/s: 
          2016-12-08 21:30:45,608 INFO  (testTimeToOpenAndReadWholeFileBlocks(254)) - Bandwidth of block 12: 2.951960 MB/s: 
          2016-12-08 21:30:45,886 INFO  (testTimeToOpenAndReadWholeFileBlocks(254)) - Bandwidth of block 13: 3.598163 MB/s: 
          2016-12-08 21:30:46,212 INFO  (testTimeToOpenAndReadWholeFileBlocks(254)) - Bandwidth of block 14: 3.070112 MB/s: 
          2016-12-08 21:30:46,542 INFO  (testTimeToOpenAndReadWholeFileBlocks(254)) - Bandwidth of block 15: 3.029587 MB/s: 
          2016-12-08 21:30:46,858 INFO  (testTimeToOpenAndReadWholeFileBlocks(254)) - Bandwidth of block 16: 3.175305 MB/s: 
          2016-12-08 21:30:47,173 INFO  (testTimeToOpenAndReadWholeFileBlocks(254)) - Bandwidth of block 17: 3.175283 MB/s: 
          2016-12-08 21:30:47,494 INFO  (testTimeToOpenAndReadWholeFileBlocks(254)) - Bandwidth of block 18: 3.111757 MB/s: 
          2016-12-08 21:30:47,809 INFO  (testTimeToOpenAndReadWholeFileBlocks(254)) - Bandwidth of block 19: 3.182531 MB/s: 
          
          ....
          
          2016-12-08 21:30:53,746 INFO  (testTimeToOpenAndReadWholeFileBlocks(254)) - Bandwidth of block 37: 2.895820 MB/s: 
          2016-12-08 21:30:53,746 INFO  contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Time to read 39278765 bytes in 37 blocks: 64,847,176,023 nS
          
          
          2016-12-08 21:30:53,746 INFO  (logStreamStatistics(366)) - Stream Statistics
          StreamStatistics{OpenOperations=2, CloseOperations=1, Closed=0, Aborted=1, SeekOperations=0, ReadExceptions=0, ForwardSeekOperations=0, BackwardSeekOperations=0, BytesSkippedOnSeek=0, BytesBackwardsOnSeek=0, BytesRead=38797312, BytesRead excluding skipped=38797312, ReadOperations=2833, ReadFullyOperations=0, ReadsIncomplete=2796, BytesReadInClose=0, BytesDiscardedInAbort=38230189}
          2016-12-08 21:30:53,746 WARN  (testTimeToOpenAndReadWholeFileBlocks(269)) - Reset the connection 1 time(s)
          2016-12-08 21:30:53,747 INFO  contract.AbstractFSContractTestBase (AbstractFSContractTestBase.java:describe(255)) - cleanup
          2016-12-08 21:30:53,747 DEBUG conn.DefaultClientConnection (DefaultClientConnection.java:shutdown(166)) - Connection 0.0.0.0:57959<->54.231.169.22:443 shut down
          2016-12-08 21:30:53,747 DEBUG conn.DefaultClientConnection (DefaultClientConnection.java:close(182)) - Connection 0.0.0.0:57959<->54.231.169.22:443 closed
          2016-12-08 21:30:53,747 DEBUG conn.PoolingClientConnectionManager (PoolingClientConnectionManager.java:releaseConnection(274)) - Connection released: [id: 2][route: {s}->https://landsat-pds.s3.amazonaws.com:443][total kept alive: 0; route allocated: 0 of 25; total allocated: 0 of 25]
          2016-12-08 21:30:53,747 DEBUG s3a.S3AFileSystem (S3AInputStream.java:closeStream(463)) - Stream s3a://landsat-pds/scene_list.gz aborted: close() operation; streamPos=38797312, nextReadPos=38797312, request range 1048576-39278765 length=39278765
          
          
          Show
          stevel@apache.org Steve Loughran added a comment - Patch for trunk adds a new resetConnection() operation which aborts any active underlying connection, with this test modified to explicitly detect reads too slow and perform resets, failing if >8 resets happen during the run. This detects and reacts to bandwidth problems, albeit with a hard coded threshold of 128KB/s. If you are getting less than that, you shouldn't be running that test. Except: after the abort the TCP connection will be slow-starting up to full B/W, so on any-long haul link, that next read may also underperform. It's probably safe to have a lower value. It would be possible to track bandwidth per Example 2016-12-08 21:29:45,965 [ Thread -1] DEBUG s3a.S3AFileSystem (S3AUtils.java:intOption(512)) - Value of fs.s3a.connection.maximum is 25 2016-12-08 21:29:45,967 [ Thread -1] DEBUG s3a.S3AFileSystem (S3AUtils.java:intOption(512)) - Value of fs.s3a.attempts.maximum is 20 2016-12-08 21:29:45,967 [ Thread -1] DEBUG s3a.S3AFileSystem (S3AUtils.java:intOption(512)) - Value of fs.s3a.connection.establish.timeout is 5000 2016-12-08 21:29:45,968 [ Thread -1] DEBUG s3a.S3AFileSystem (S3AUtils.java:intOption(512)) - Value of fs.s3a.connection.timeout is 5000 2016-12-08 21:29:45,968 [ Thread -1] DEBUG s3a.S3AFileSystem (S3AUtils.java:intOption(512)) - Value of fs.s3a.socket.send.buffer is 65536 2016-12-08 21:29:45,968 [ Thread -1] DEBUG s3a.S3AFileSystem (S3AUtils.java:intOption(512)) - Value of fs.s3a.socket.recv.buffer is 32678 2016-12-08 21:29:45,970 [ Thread -1] DEBUG s3a.S3AFileSystem (S3ClientFactory.java:initUserAgent(187)) - Using User-Agent: Hadoop 2.8.0-SNAPSHOT 2016-12-08 21:29:46,186 [ Thread -1] DEBUG s3a.S3AFileSystem (S3AUtils.java:intOption(512)) - Value of fs.s3a.paging.maximum is 5000 2016-12-08 21:29:46,188 [ Thread -1] DEBUG s3a.S3AFileSystem (S3AUtils.java:longBytesOption(555)) - Value of fs.s3a.block.size is 33554432 2016-12-08 21:29:46,188 [ Thread -1] DEBUG s3a.S3AFileSystem (S3AUtils.java:longBytesOption(555)) - Value of fs.s3a.readahead.range is 65536 2016-12-08 21:29:46,190 [ Thread -1] DEBUG s3a.S3AFileSystem (S3AUtils.java:intOption(512)) - Value of fs.s3a.max.total.tasks is 10 2016-12-08 21:29:46,190 [ Thread -1] DEBUG s3a.S3AFileSystem (S3AUtils.java:longOption(533)) - Value of fs.s3a.threads.keepalivetime is 60 2016-12-08 21:29:49,070 DEBUG amazonaws.request (AmazonHttpClient.java:logResponseRequestId(856)) - AWS Request ID: 1FEDCD17B6FBB2C8 2016-12-08 21:29:49,381 DEBUG - Bytes in read #1: 16347 , block bytes: 16347, remaining in block: 1032229 duration=478865598 nS; ns/ byte : 29293, bandwidth=0.032556 MB/s 2016-12-08 21:29:49,382 DEBUG - Bytes in read #2: 630 , block bytes: 16977, remaining in block: 1031599 duration=212331 nS; ns/ byte : 337, bandwidth=2.829614 MB/s 2016-12-08 21:29:49,531 DEBUG - Bytes in read #3: 16347 , block bytes: 33324, remaining in block: 1015252 duration=149393770 nS; ns/ byte : 9138, bandwidth=0.104353 MB/s 2016-12-08 21:29:49,532 DEBUG - Bytes in read #4: 1061 , block bytes: 34385, remaining in block: 1014191 duration=167998 nS; ns/ byte : 158, bandwidth=6.022979 MB/s 2016-12-08 21:29:49,688 DEBUG - Bytes in read #5: 16347 , block bytes: 50732, remaining in block: 997844 duration=155828299 nS; ns/ byte : 9532, bandwidth=0.100044 MB/s 2016-12-08 21:29:49,688 DEBUG - Bytes in read #6: 1061 , block bytes: 51793, remaining in block: 996783 duration=128858 nS; ns/ byte : 121, bandwidth=7.852430 MB/s 2016-12-08 21:29:49,841 DEBUG - Bytes in read #7: 16347 , block bytes: 68140, remaining in block: 980436 duration=152823750 nS; ns/ byte : 9348, bandwidth=0.102011 MB/s 2016-12-08 21:29:49,842 DEBUG - Bytes in read #8: 1061 , block bytes: 69201, remaining in block: 979375 duration=280146 nS; ns/ byte : 264, bandwidth=3.611861 MB/s 2016-12-08 21:29:50,149 DEBUG - Bytes in read #9: 16347 , block bytes: 85548, remaining in block: 963028 duration=306567695 nS; ns/ byte : 18753, bandwidth=0.050852 MB/s 2016-12-08 21:29:50,149 DEBUG - Bytes in read #10: 1061 , block bytes: 86609, remaining in block: 961967 duration=161057 nS; ns/ byte : 151, bandwidth=6.282549 MB/s 2016-12-08 21:29:51,073 DEBUG - Bytes in read #11: 16347 , block bytes: 102956, remaining in block: 945620 duration=923209136 nS; ns/ byte : 56475, bandwidth=0.016886 MB/s 2016-12-08 21:29:51,073 DEBUG - Bytes in read #12: 1061 , block bytes: 104017, remaining in block: 944559 duration=172071 nS; ns/ byte : 162, bandwidth=5.880412 MB/s 2016-12-08 21:29:51,690 DEBUG - Bytes in read #13: 16347 , block bytes: 120364, remaining in block: 928212 duration=615842374 nS; ns/ byte : 37673, bandwidth=0.025314 MB/s 2016-12-08 21:29:51,843 DEBUG - Bytes in read #14: 1061 , block bytes: 121425, remaining in block: 927151 duration=152502578 nS; ns/ byte : 143734, bandwidth=0.006635 MB/s 2016-12-08 21:29:52,816 DEBUG - Bytes in read #15: 16347 , block bytes: 137772, remaining in block: 910804 duration=972690659 nS; ns/ byte : 59502, bandwidth=0.016027 MB/s 2016-12-08 21:29:52,816 DEBUG - Bytes in read #16: 1061 , block bytes: 138833, remaining in block: 909743 duration=154987 nS; ns/ byte : 146, bandwidth=6.528602 MB/s 2016-12-08 21:29:53,889 DEBUG - Bytes in read #17: 16347 , block bytes: 155180, remaining in block: 893396 duration=1072600915 nS; ns/ byte : 65614, bandwidth=0.014534 MB/s 2016-12-08 21:29:53,991 DEBUG - Bytes in read #18: 1061 , block bytes: 156241, remaining in block: 892335 duration=101460691 nS; ns/ byte : 95627, bandwidth=0.009973 MB/s 2016-12-08 21:29:55,165 DEBUG - Bytes in read #19: 16347 , block bytes: 172588, remaining in block: 875988 duration=1173337630 nS; ns/ byte : 71776, bandwidth=0.013287 MB/s 2016-12-08 21:29:55,165 DEBUG - Bytes in read #20: 1061 , block bytes: 173649, remaining in block: 874927 duration=116827 nS; ns/ byte : 110, bandwidth=8.661084 MB/s 2016-12-08 21:29:55,628 DEBUG - Bytes in read #21: 16347 , block bytes: 189996, remaining in block: 858580 duration=462231152 nS; ns/ byte : 28276, bandwidth=0.033727 MB/s 2016-12-08 21:29:55,935 DEBUG - Bytes in read #22: 1061 , block bytes: 191057, remaining in block: 857519 duration=306790128 nS; ns/ byte : 289151, bandwidth=0.003298 MB/s 2016-12-08 21:29:56,785 DEBUG - Bytes in read #23: 16347 , block bytes: 207404, remaining in block: 841172 duration=849333859 nS; ns/ byte : 51956, bandwidth=0.018355 MB/s 2016-12-08 21:29:56,786 DEBUG - Bytes in read #24: 1061 , block bytes: 208465, remaining in block: 840111 duration=223996 nS; ns/ byte : 211, bandwidth=4.517261 MB/s 2016-12-08 21:29:58,197 DEBUG - Bytes in read #25: 16347 , block bytes: 224812, remaining in block: 823764 duration=1411150169 nS; ns/ byte : 86324, bandwidth=0.011048 MB/s 2016-12-08 21:29:58,350 DEBUG - Bytes in read #26: 1061 , block bytes: 225873, remaining in block: 822703 duration=151618968 nS; ns/ byte : 142901, bandwidth=0.006674 MB/s 2016-12-08 21:29:59,275 DEBUG - Bytes in read #27: 16347 , block bytes: 242220, remaining in block: 806356 duration=924832331 nS; ns/ byte : 56575, bandwidth=0.016857 MB/s 2016-12-08 21:29:59,275 DEBUG - Bytes in read #28: 1061 , block bytes: 243281, remaining in block: 805295 duration=157331 nS; ns/ byte : 148, bandwidth=6.431336 MB/s 2016-12-08 21:30:00,197 DEBUG - Bytes in read #29: 16347 , block bytes: 259628, remaining in block: 788948 duration=919177546 nS; ns/ byte : 56229, bandwidth=0.016961 MB/s 2016-12-08 21:30:00,197 DEBUG - Bytes in read #30: 1061 , block bytes: 260689, remaining in block: 787887 duration=159754 nS; ns/ byte : 150, bandwidth=6.333791 MB/s 2016-12-08 21:30:01,116 DEBUG - Bytes in read #31: 16347 , block bytes: 277036, remaining in block: 771540 duration=918850994 nS; ns/ byte : 56209, bandwidth=0.016967 MB/s 2016-12-08 21:30:01,117 DEBUG - Bytes in read #32: 1061 , block bytes: 278097, remaining in block: 770479 duration=153206 nS; ns/ byte : 144, bandwidth=6.604496 MB/s 2016-12-08 21:30:02,040 DEBUG - Bytes in read #33: 16347 , block bytes: 294444, remaining in block: 754132 duration=923043728 nS; ns/ byte : 56465, bandwidth=0.016889 MB/s 2016-12-08 21:30:02,041 DEBUG - Bytes in read #34: 1061 , block bytes: 295505, remaining in block: 753071 duration=152100 nS; ns/ byte : 143, bandwidth=6.652521 MB/s 2016-12-08 21:30:03,119 DEBUG - Bytes in read #35: 16347 , block bytes: 311852, remaining in block: 736724 duration=1077535517 nS; ns/ byte : 65916, bandwidth=0.014468 MB/s 2016-12-08 21:30:03,119 DEBUG - Bytes in read #36: 1061 , block bytes: 312913, remaining in block: 735663 duration=114098 nS; ns/ byte : 107, bandwidth=8.868240 MB/s 2016-12-08 21:30:04,140 DEBUG - Bytes in read #37: 16347 , block bytes: 329260, remaining in block: 719316 duration=1020943631 nS; ns/ byte : 62454, bandwidth=0.015270 MB/s 2016-12-08 21:30:04,141 DEBUG - Bytes in read #38: 1061 , block bytes: 330321, remaining in block: 718255 duration=154676 nS; ns/ byte : 145, bandwidth=6.541729 MB/s 2016-12-08 21:30:05,063 DEBUG - Bytes in read #39: 16347 , block bytes: 346668, remaining in block: 701908 duration=921944703 nS; ns/ byte : 56398, bandwidth=0.016910 MB/s 2016-12-08 21:30:06,027 DEBUG - Bytes in read #40: 1061 , block bytes: 347729, remaining in block: 700847 duration=963349047 nS; ns/ byte : 907963, bandwidth=0.001050 MB/s 2016-12-08 21:30:07,488 DEBUG - Bytes in read #41: 16347 , block bytes: 364076, remaining in block: 684500 duration=1460036513 nS; ns/ byte : 89315, bandwidth=0.010678 MB/s 2016-12-08 21:30:07,795 DEBUG - Bytes in read #42: 1061 , block bytes: 365137, remaining in block: 683439 duration=306386850 nS; ns/ byte : 288771, bandwidth=0.003303 MB/s 2016-12-08 21:30:08,130 DEBUG - Bytes in read #43: 16347 , block bytes: 381484, remaining in block: 667092 duration=335450550 nS; ns/ byte : 20520, bandwidth=0.046474 MB/s 2016-12-08 21:30:08,262 DEBUG - Bytes in read #44: 1061 , block bytes: 382545, remaining in block: 666031 duration=130994948 nS; ns/ byte : 123463, bandwidth=0.007724 MB/s 2016-12-08 21:30:08,595 DEBUG - Bytes in read #45: 16347 , block bytes: 398892, remaining in block: 649684 duration=333166901 nS; ns/ byte : 20380, bandwidth=0.046793 MB/s 2016-12-08 21:30:08,596 DEBUG - Bytes in read #46: 1061 , block bytes: 399953, remaining in block: 648623 duration=131882 nS; ns/ byte : 124, bandwidth=7.672377 MB/s 2016-12-08 21:30:09,211 DEBUG - Bytes in read #47: 16347 , block bytes: 416300, remaining in block: 632276 duration=614798623 nS; ns/ byte : 37609, bandwidth=0.025357 MB/s 2016-12-08 21:30:09,211 DEBUG - Bytes in read #48: 1061 , block bytes: 417361, remaining in block: 631215 duration=186043 nS; ns/ byte : 175, bandwidth=5.438788 MB/s 2016-12-08 21:30:09,674 DEBUG - Bytes in read #49: 16347 , block bytes: 433708, remaining in block: 614868 duration=462221834 nS; ns/ byte : 28275, bandwidth=0.033728 MB/s 2016-12-08 21:30:09,674 DEBUG - Bytes in read #50: 1061 , block bytes: 434769, remaining in block: 613807 duration=124757 nS; ns/ byte : 117, bandwidth=8.110555 MB/s 2016-12-08 21:30:09,984 DEBUG - Bytes in read #51: 16347 , block bytes: 451116, remaining in block: 597460 duration=308999007 nS; ns/ byte : 18902, bandwidth=0.050452 MB/s 2016-12-08 21:30:09,984 DEBUG - Bytes in read #52: 1061 , block bytes: 452177, remaining in block: 596399 duration=84490 nS; ns/ byte : 79, bandwidth=11.975955 MB/s 2016-12-08 21:30:10,597 DEBUG - Bytes in read #53: 16347 , block bytes: 468524, remaining in block: 580052 duration=612539269 nS; ns/ byte : 37471, bandwidth=0.025451 MB/s 2016-12-08 21:30:10,597 DEBUG - Bytes in read #54: 1061 , block bytes: 469585, remaining in block: 578991 duration=147609 nS; ns/ byte : 139, bandwidth=6.854924 MB/s 2016-12-08 21:30:11,060 DEBUG - Bytes in read #55: 16347 , block bytes: 485932, remaining in block: 562644 duration=462436225 nS; ns/ byte : 28288, bandwidth=0.033712 MB/s 2016-12-08 21:30:11,060 DEBUG - Bytes in read #56: 1061 , block bytes: 486993, remaining in block: 561583 duration=100576 nS; ns/ byte : 94, bandwidth=10.060536 MB/s 2016-12-08 21:30:11,931 DEBUG - Bytes in read #57: 16347 , block bytes: 503340, remaining in block: 545236 duration=870489198 nS; ns/ byte : 53250, bandwidth=0.017909 MB/s 2016-12-08 21:30:11,981 DEBUG - Bytes in read #58: 1061 , block bytes: 504401, remaining in block: 544175 duration=49541837 nS; ns/ byte : 46693, bandwidth=0.020424 MB/s 2016-12-08 21:30:13,060 DEBUG - Bytes in read #59: 16347 , block bytes: 520748, remaining in block: 527828 duration=1078170319 nS; ns/ byte : 65955, bandwidth=0.014459 MB/s 2016-12-08 21:30:13,060 DEBUG - Bytes in read #60: 1061 , block bytes: 521809, remaining in block: 526767 duration=88905 nS; ns/ byte : 83, bandwidth=11.381232 MB/s 2016-12-08 21:30:13,832 DEBUG - Bytes in read #61: 16347 , block bytes: 538156, remaining in block: 510420 duration=771710163 nS; ns/ byte : 47208, bandwidth=0.020202 MB/s 2016-12-08 21:30:13,833 DEBUG - Bytes in read #62: 1061 , block bytes: 539217, remaining in block: 509359 duration=195473 nS; ns/ byte : 184, bandwidth=5.176410 MB/s 2016-12-08 21:30:14,985 DEBUG - Bytes in read #63: 16347 , block bytes: 555564, remaining in block: 493012 duration=1152517210 nS; ns/ byte : 70503, bandwidth=0.013527 MB/s 2016-12-08 21:30:14,986 DEBUG - Bytes in read #64: 1061 , block bytes: 556625, remaining in block: 491951 duration=137869 nS; ns/ byte : 129, bandwidth=7.339202 MB/s 2016-12-08 21:30:15,756 DEBUG - Bytes in read #65: 16347 , block bytes: 572972, remaining in block: 475604 duration=770143725 nS; ns/ byte : 47112, bandwidth=0.020243 MB/s 2016-12-08 21:30:15,757 DEBUG - Bytes in read #66: 1061 , block bytes: 574033, remaining in block: 474543 duration=161316 nS; ns/ byte : 152, bandwidth=6.272462 MB/s 2016-12-08 21:30:16,907 DEBUG - Bytes in read #67: 16347 , block bytes: 590380, remaining in block: 458196 duration=1149954555 nS; ns/ byte : 70346, bandwidth=0.013557 MB/s 2016-12-08 21:30:16,908 DEBUG - Bytes in read #68: 1061 , block bytes: 591441, remaining in block: 457135 duration=166415 nS; ns/ byte : 156, bandwidth=6.080272 MB/s 2016-12-08 21:30:17,776 DEBUG - Bytes in read #69: 16347 , block bytes: 607788, remaining in block: 440788 duration=867495557 nS; ns/ byte : 53067, bandwidth=0.017971 MB/s 2016-12-08 21:30:17,777 DEBUG - Bytes in read #70: 1061 , block bytes: 608849, remaining in block: 439727 duration=938146 nS; ns/ byte : 884, bandwidth=1.078562 MB/s 2016-12-08 21:30:18,948 DEBUG - Bytes in read #71: 16347 , block bytes: 625196, remaining in block: 423380 duration=1170544837 nS; ns/ byte : 71606, bandwidth=0.013318 MB/s 2016-12-08 21:30:19,256 DEBUG - Bytes in read #72: 1061 , block bytes: 626257, remaining in block: 422319 duration=307536130 nS; ns/ byte : 289854, bandwidth=0.003290 MB/s 2016-12-08 21:30:19,721 DEBUG - Bytes in read #73: 16347 , block bytes: 642604, remaining in block: 405972 duration=464770187 nS; ns/ byte : 28431, bandwidth=0.033543 MB/s 2016-12-08 21:30:19,721 DEBUG - Bytes in read #74: 1061 , block bytes: 643665, remaining in block: 404911 duration=145775 nS; ns/ byte : 137, bandwidth=6.941166 MB/s 2016-12-08 21:30:20,030 DEBUG - Bytes in read #75: 16347 , block bytes: 660012, remaining in block: 388564 duration=307973423 nS; ns/ byte : 18839, bandwidth=0.050620 MB/s 2016-12-08 21:30:20,338 DEBUG - Bytes in read #76: 1061 , block bytes: 661073, remaining in block: 387503 duration=307555981 nS; ns/ byte : 289873, bandwidth=0.003290 MB/s 2016-12-08 21:30:20,798 DEBUG - Bytes in read #77: 16347 , block bytes: 677420, remaining in block: 371156 duration=459891175 nS; ns/ byte : 28133, bandwidth=0.033899 MB/s 2016-12-08 21:30:20,951 DEBUG - Bytes in read #78: 1061 , block bytes: 678481, remaining in block: 370095 duration=152066369 nS; ns/ byte : 143323, bandwidth=0.006654 MB/s 2016-12-08 21:30:21,414 DEBUG - Bytes in read #79: 16347 , block bytes: 694828, remaining in block: 353748 duration=463288362 nS; ns/ byte : 28340, bandwidth=0.033650 MB/s 2016-12-08 21:30:21,565 DEBUG - Bytes in read #80: 1061 , block bytes: 695889, remaining in block: 352687 duration=150431750 nS; ns/ byte : 141782, bandwidth=0.006726 MB/s 2016-12-08 21:30:22,183 DEBUG - Bytes in read #81: 16347 , block bytes: 712236, remaining in block: 336340 duration=617356332 nS; ns/ byte : 37765, bandwidth=0.025252 MB/s 2016-12-08 21:30:22,183 DEBUG - Bytes in read #82: 1061 , block bytes: 713297, remaining in block: 335279 duration=91692 nS; ns/ byte : 86, bandwidth=11.035297 MB/s 2016-12-08 21:30:23,223 DEBUG - Bytes in read #83: 16347 , block bytes: 729644, remaining in block: 318932 duration=1039467778 nS; ns/ byte : 63587, bandwidth=0.014998 MB/s 2016-12-08 21:30:23,223 DEBUG - Bytes in read #84: 1061 , block bytes: 730705, remaining in block: 317871 duration=119170 nS; ns/ byte : 112, bandwidth=8.490798 MB/s 2016-12-08 21:30:23,687 DEBUG - Bytes in read #85: 16347 , block bytes: 747052, remaining in block: 301524 duration=463642449 nS; ns/ byte : 28362, bandwidth=0.033624 MB/s 2016-12-08 21:30:23,996 DEBUG - Bytes in read #86: 1061 , block bytes: 748113, remaining in block: 300463 duration=308626480 nS; ns/ byte : 290882, bandwidth=0.003279 MB/s 2016-12-08 21:30:24,763 DEBUG - Bytes in read #87: 16347 , block bytes: 764460, remaining in block: 284116 duration=766976410 nS; ns/ byte : 46918, bandwidth=0.020326 MB/s 2016-12-08 21:30:24,764 DEBUG - Bytes in read #88: 1061 , block bytes: 765521, remaining in block: 283055 duration=94835 nS; ns/ byte : 89, bandwidth=10.669568 MB/s 2016-12-08 21:30:26,695 DEBUG - Bytes in read #89: 16347 , block bytes: 781868, remaining in block: 266708 duration=1930730977 nS; ns/ byte : 118109, bandwidth=0.008075 MB/s 2016-12-08 21:30:27,003 DEBUG - Bytes in read #90: 1061 , block bytes: 782929, remaining in block: 265647 duration=307614364 nS; ns/ byte : 289928, bandwidth=0.003289 MB/s 2016-12-08 21:30:27,465 DEBUG - Bytes in read #91: 16347 , block bytes: 799276, remaining in block: 249300 duration=461784239 nS; ns/ byte : 28248, bandwidth=0.033760 MB/s 2016-12-08 21:30:27,465 DEBUG - Bytes in read #92: 1061 , block bytes: 800337, remaining in block: 248239 duration=62419 nS; ns/ byte : 58, bandwidth=16.210584 MB/s 2016-12-08 21:30:28,333 DEBUG - Bytes in read #93: 16347 , block bytes: 816684, remaining in block: 231892 duration=868005446 nS; ns/ byte : 53098, bandwidth=0.017960 MB/s 2016-12-08 21:30:28,895 DEBUG - Bytes in read #94: 1061 , block bytes: 817745, remaining in block: 230831 duration=560647267 nS; ns/ byte : 528414, bandwidth=0.001805 MB/s 2016-12-08 21:30:29,762 DEBUG - Bytes in read #95: 16347 , block bytes: 834092, remaining in block: 214484 duration=866587960 nS; ns/ byte : 53012, bandwidth=0.017990 MB/s 2016-12-08 21:30:29,864 DEBUG - Bytes in read #96: 1061 , block bytes: 835153, remaining in block: 213423 duration=101564552 nS; ns/ byte : 95725, bandwidth=0.009963 MB/s 2016-12-08 21:30:30,379 DEBUG - Bytes in read #97: 16347 , block bytes: 851500, remaining in block: 197076 duration=514754938 nS; ns/ byte : 31489, bandwidth=0.030286 MB/s 2016-12-08 21:30:30,688 DEBUG - Bytes in read #98: 1061 , block bytes: 852561, remaining in block: 196015 duration=309420585 nS; ns/ byte : 291631, bandwidth=0.003270 MB/s 2016-12-08 21:30:31,457 DEBUG - Bytes in read #99: 16347 , block bytes: 868908, remaining in block: 179668 duration=768580519 nS; ns/ byte : 47016, bandwidth=0.020284 MB/s 2016-12-08 21:30:31,458 DEBUG - Bytes in read #100: 1061 , block bytes: 869969, remaining in block: 178607 duration=110392 nS; ns/ byte : 104, bandwidth=9.165958 MB/s 2016-12-08 21:30:32,225 DEBUG - Bytes in read #101: 16347 , block bytes: 886316, remaining in block: 162260 duration=767381785 nS; ns/ byte : 46943, bandwidth=0.020315 MB/s 2016-12-08 21:30:32,226 DEBUG - Bytes in read #102: 1061 , block bytes: 887377, remaining in block: 161199 duration=182729 nS; ns/ byte : 172, bandwidth=5.537427 MB/s 2016-12-08 21:30:33,095 DEBUG - Bytes in read #103: 16347 , block bytes: 903724, remaining in block: 144852 duration=869101542 nS; ns/ byte : 53165, bandwidth=0.017938 MB/s 2016-12-08 21:30:33,247 DEBUG - Bytes in read #104: 1061 , block bytes: 904785, remaining in block: 143791 duration=151712419 nS; ns/ byte : 142990, bandwidth=0.006670 MB/s 2016-12-08 21:30:33,711 DEBUG - Bytes in read #105: 16347 , block bytes: 921132, remaining in block: 127444 duration=463346639 nS; ns/ byte : 28344, bandwidth=0.033646 MB/s 2016-12-08 21:30:33,864 DEBUG - Bytes in read #106: 1061 , block bytes: 922193, remaining in block: 126383 duration=152830422 nS; ns/ byte : 144043, bandwidth=0.006621 MB/s 2016-12-08 21:30:34,641 DEBUG - Bytes in read #107: 16347 , block bytes: 938540, remaining in block: 110036 duration=777184061 nS; ns/ byte : 47542, bandwidth=0.020059 MB/s 2016-12-08 21:30:34,642 DEBUG - Bytes in read #108: 1061 , block bytes: 939601, remaining in block: 108975 duration=92668 nS; ns/ byte : 87, bandwidth=10.919071 MB/s 2016-12-08 21:30:35,655 DEBUG - Bytes in read #109: 16347 , block bytes: 955948, remaining in block: 92628 duration=1013110625 nS; ns/ byte : 61975, bandwidth=0.015388 MB/s 2016-12-08 21:30:35,810 DEBUG - Bytes in read #110: 1061 , block bytes: 957009, remaining in block: 91567 duration=154218951 nS; ns/ byte : 145352, bandwidth=0.006561 MB/s 2016-12-08 21:30:36,275 DEBUG - Bytes in read #111: 16347 , block bytes: 973356, remaining in block: 75220 duration=465093303 nS; ns/ byte : 28451, bandwidth=0.033520 MB/s 2016-12-08 21:30:36,276 DEBUG - Bytes in read #112: 1061 , block bytes: 974417, remaining in block: 74159 duration=171536 nS; ns/ byte : 161, bandwidth=5.898753 MB/s 2016-12-08 21:30:36,745 DEBUG - Bytes in read #113: 16347 , block bytes: 990764, remaining in block: 57812 duration=468606455 nS; ns/ byte : 28666, bandwidth=0.033268 MB/s 2016-12-08 21:30:36,745 DEBUG - Bytes in read #114: 1061 , block bytes: 991825, remaining in block: 56751 duration=91092 nS; ns/ byte : 85, bandwidth=11.107984 MB/s 2016-12-08 21:30:37,206 DEBUG - Bytes in read #115: 16347 , block bytes: 1008172, remaining in block: 40404 duration=460276607 nS; ns/ byte : 28156, bandwidth=0.033870 MB/s 2016-12-08 21:30:37,514 DEBUG - Bytes in read #116: 1061 , block bytes: 1009233, remaining in block: 39343 duration=307705385 nS; ns/ byte : 290014, bandwidth=0.003288 MB/s 2016-12-08 21:30:37,975 DEBUG - Bytes in read #117: 16347 , block bytes: 1025580, remaining in block: 22996 duration=461043201 nS; ns/ byte : 28203, bandwidth=0.033814 MB/s 2016-12-08 21:30:37,975 DEBUG - Bytes in read #118: 1061 , block bytes: 1026641, remaining in block: 21935 duration=74629 nS; ns/ byte : 70, bandwidth=13.558381 MB/s 2016-12-08 21:30:38,745 DEBUG - Bytes in read #119: 16347 , block bytes: 1042988, remaining in block: 5588 duration=768900663 nS; ns/ byte : 47036, bandwidth=0.020275 MB/s 2016-12-08 21:30:38,745 DEBUG - Bytes in read #120: 1061 , block bytes: 1044049, remaining in block: 4527 duration=161553 nS; ns/ byte : 152, bandwidth=6.263260 MB/s 2016-12-08 21:30:39,362 DEBUG - Bytes in read #121: 4527 , block bytes: 1048576, remaining in block: 0 duration=616755333 nS; ns/ byte : 136239, bandwidth=0.007000 MB/s 2016-12-08 21:30:39,363 INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 1 in 121 reads: 50,463,661,382 nS 2016-12-08 21:30:39,363 INFO (testTimeToOpenAndReadWholeFileBlocks(254)) - Bandwidth of block 1: 0.019816 MB/s: 2016-12-08 21:30:39,363 WARN (testTimeToOpenAndReadWholeFileBlocks(256)) - Bandwidth 0.019816 too low on block 1: resetting connection 2016-12-08 21:30:39,366 DEBUG conn.DefaultClientConnection (DefaultClientConnection.java:shutdown(166)) - Connection 0.0.0.0:57950<->54.231.169.2:443 shut down 2016-12-08 21:30:39,366 DEBUG conn.DefaultClientConnection (DefaultClientConnection.java:close(182)) - Connection 0.0.0.0:57950<->54.231.169.2:443 closed 2016-12-08 21:30:39,366 DEBUG conn.PoolingClientConnectionManager (PoolingClientConnectionManager.java:releaseConnection(274)) - Connection released: [id: 1][route: {s}->https: //landsat-pds.s3.amazonaws.com:443][total kept alive: 0; route allocated: 0 of 25; total allocated: 0 of 25] 2016-12-08 21:30:39,366 DEBUG s3a.S3AFileSystem (S3AInputStream.java:closeStream(463)) - Stream s3a: //landsat-pds/scene_list.gz aborted: reset(); streamPos=1048576, nextReadPos=1048576, request range 0-39278765 length=39278765 2016-12-08 21:30:39,367 DEBUG s3a.S3AFileSystem (S3AInputStream.java:reopen(140)) - reopen(s3a: //landsat-pds/scene_list.gz) for read from new offset range[1048576-39278765], length=1048576, streamPosition=1048576, nextReadPosition=1048576 2016-12-08 21:30:39,367 DEBUG amazonaws.request (AmazonHttpClient.java:executeOneRequest(671)) - Sending Request: GET https: //landsat-pds.s3.amazonaws.com /scene_list.gz Headers: (User-Agent: Hadoop 2.8.0-SNAPSHOT, aws-sdk-java/1.10.6 Mac_OS_X/10.12.1 Java_HotSpot(TM)_64-Bit_Server_VM/25.102-b14/1.8.0_102, Range: bytes=1048576-39278765, Content-Type: application/x-www-form-urlencoded; charset=utf-8, ) 2016-12-08 21:30:39,367 DEBUG conn.PoolingClientConnectionManager (PoolingClientConnectionManager.java:requestConnection(184)) - Connection request: [route: {s}->https: //landsat-pds.s3.amazonaws.com:443][total kept alive: 0; route allocated: 0 of 25; total allocated: 0 of 25] 2016-12-08 21:30:39,368 DEBUG conn.PoolingClientConnectionManager (PoolingClientConnectionManager.java:leaseConnection(218)) - Connection leased: [id: 2][route: {s}->https: //landsat-pds.s3.amazonaws.com:443][total kept alive: 0; route allocated: 1 of 25; total allocated: 1 of 25] 2016-12-08 21:30:39,394 DEBUG conn.DefaultClientConnectionOperator (DefaultClientConnectionOperator.java:openConnection(176)) - Connecting to landsat-pds.s3.amazonaws.com:443 2016-12-08 21:30:40,040 DEBUG protocol.RequestAddCookies (RequestAddCookies.java:process(122)) - CookieSpec selected: default 2016-12-08 21:30:40,040 DEBUG protocol.RequestAuthCache (RequestAuthCache.java:process(76)) - Auth cache not set in the context 2016-12-08 21:30:40,040 DEBUG protocol.RequestProxyAuthentication (RequestProxyAuthentication.java:process(88)) - Proxy auth state: UNCHALLENGED 2016-12-08 21:30:40,040 DEBUG conn.DefaultClientConnection (DefaultClientConnection.java:sendRequestHeader(276)) - Sending request: GET /scene_list.gz HTTP/1.1 2016-12-08 21:30:40,041 DEBUG http.headers (DefaultClientConnection.java:sendRequestHeader(280)) - >> GET /scene_list.gz HTTP/1.1 2016-12-08 21:30:40,041 DEBUG http.headers (DefaultClientConnection.java:sendRequestHeader(283)) - >> Host: landsat-pds.s3.amazonaws.com 2016-12-08 21:30:40,041 DEBUG http.headers (DefaultClientConnection.java:sendRequestHeader(283)) - >> Authorization: AWS AKIAIYZ5JQOW3N5H6NPA:3c0hXBn3xs2erSymklPv8hV1MKM= 2016-12-08 21:30:40,041 DEBUG http.headers (DefaultClientConnection.java:sendRequestHeader(283)) - >> User-Agent: Hadoop 2.8.0-SNAPSHOT, aws-sdk-java/1.10.6 Mac_OS_X/10.12.1 Java_HotSpot(TM)_64-Bit_Server_VM/25.102-b14/1.8.0_102 2016-12-08 21:30:40,041 DEBUG http.headers (DefaultClientConnection.java:sendRequestHeader(283)) - >> Range: bytes=1048576-39278765 2016-12-08 21:30:40,041 DEBUG http.headers (DefaultClientConnection.java:sendRequestHeader(283)) - >> Date: Thu, 08 Dec 2016 21:30:39 GMT 2016-12-08 21:30:40,041 DEBUG http.headers (DefaultClientConnection.java:sendRequestHeader(283)) - >> Content-Type: application/x-www-form-urlencoded; charset=utf-8 2016-12-08 21:30:40,041 DEBUG http.headers (DefaultClientConnection.java:sendRequestHeader(283)) - >> Connection: Keep-Alive 2016-12-08 21:30:40,210 DEBUG conn.DefaultClientConnection (DefaultClientConnection.java:receiveResponseHeader(261)) - Receiving response: HTTP/1.1 206 Partial Content 2016-12-08 21:30:40,210 DEBUG http.headers (DefaultClientConnection.java:receiveResponseHeader(264)) - << HTTP/1.1 206 Partial Content 2016-12-08 21:30:40,211 DEBUG http.headers (DefaultClientConnection.java:receiveResponseHeader(267)) - << x-amz-id-2: qKwcRkY1K8w22XWihP8NPtJWfcVbmX7jADNiNYk53DXSsbYOAxUobZwEpIPt0TN27ru9TBJNNA0= 2016-12-08 21:30:40,211 DEBUG http.headers (DefaultClientConnection.java:receiveResponseHeader(267)) - << x-amz-request-id: CA85E25E70613150 2016-12-08 21:30:40,211 DEBUG http.headers (DefaultClientConnection.java:receiveResponseHeader(267)) - << Date: Thu, 08 Dec 2016 21:30:41 GMT 2016-12-08 21:30:40,211 DEBUG http.headers (DefaultClientConnection.java:receiveResponseHeader(267)) - << Last-Modified: Thu, 08 Dec 2016 17:21:24 GMT 2016-12-08 21:30:40,211 DEBUG http.headers (DefaultClientConnection.java:receiveResponseHeader(267)) - << ETag: "6efe666c9bc2affbdba4f43dce9d0d55" 2016-12-08 21:30:40,211 DEBUG http.headers (DefaultClientConnection.java:receiveResponseHeader(267)) - << Accept-Ranges: bytes 2016-12-08 21:30:40,211 DEBUG http.headers (DefaultClientConnection.java:receiveResponseHeader(267)) - << Content-Range: bytes 1048576-39278764/39278765 2016-12-08 21:30:40,211 DEBUG http.headers (DefaultClientConnection.java:receiveResponseHeader(267)) - << Content-Type: application/octet-stream 2016-12-08 21:30:40,211 DEBUG http.headers (DefaultClientConnection.java:receiveResponseHeader(267)) - << Content-Length: 38230189 2016-12-08 21:30:40,213 DEBUG http.headers (DefaultClientConnection.java:receiveResponseHeader(267)) - << Server: AmazonS3 2016-12-08 21:30:40,213 DEBUG amazonaws.request (AmazonHttpClient.java:logHeaderRequestId(836)) - x-amzn-RequestId: not available 2016-12-08 21:30:40,214 DEBUG amazonaws.request (AmazonHttpClient.java:handleResponse(1091)) - Received successful response: 206, AWS Request ID: CA85E25E70613150 2016-12-08 21:30:40,214 DEBUG amazonaws.request (AmazonHttpClient.java:logResponseRequestId(856)) - AWS Request ID: CA85E25E70613150 2016-12-08 21:30:40,364 DEBUG - Bytes in read #1: 16347 , block bytes: 16347, remaining in block: 1032229 duration=997179473 nS; ns/ byte : 61000, bandwidth=0.015634 MB/s 2016-12-08 21:30:40,365 DEBUG - Bytes in read #2: 624 , block bytes: 16971, remaining in block: 1031605 duration=101431 nS; ns/ byte : 162, bandwidth=5.866971 MB/s 2016-12-08 21:30:40,368 DEBUG - Bytes in read #3: 16347 , block bytes: 33318, remaining in block: 1015258 duration=3318034 nS; ns/ byte : 202, bandwidth=4.698479 MB/s 2016-12-08 21:30:40,368 DEBUG - Bytes in read #4: 1061 , block bytes: 34379, remaining in block: 1014197 duration=82636 nS; ns/ byte : 77, bandwidth=12.244645 MB/s 2016-12-08 21:30:40,519 DEBUG - Bytes in read #5: 16347 , block bytes: 50726, remaining in block: 997850 duration=150854582 nS; ns/ byte : 9228, bandwidth=0.103343 MB/s 2016-12-08 21:30:40,520 DEBUG - Bytes in read #6: 1061 , block bytes: 51787, remaining in block: 996789 duration=76297 nS; ns/ byte : 71, bandwidth=13.261969 MB/s 2016-12-08 21:30:40,522 DEBUG - Bytes in read #7: 16347 , block bytes: 68134, remaining in block: 980442 duration=2337916 nS; ns/ byte : 143, bandwidth=6.668210 MB/s 2016-12-08 21:30:40,523 DEBUG - Bytes in read #8: 1061 , block bytes: 69195, remaining in block: 979381 duration=687085 nS; ns/ byte : 647, bandwidth=1.472669 MB/s 2016-12-08 21:30:40,672 DEBUG - Bytes in read #9: 16347 , block bytes: 85542, remaining in block: 963034 duration=148946723 nS; ns/ byte : 9111, bandwidth=0.104666 MB/s 2016-12-08 21:30:40,673 DEBUG - Bytes in read #10: 1061 , block bytes: 86603, remaining in block: 961973 duration=88030 nS; ns/ byte : 82, bandwidth=11.494359 MB/s 2016-12-08 21:30:40,675 DEBUG - Bytes in read #11: 16347 , block bytes: 102950, remaining in block: 945626 duration=1850448 nS; ns/ byte : 113, bandwidth=8.424832 MB/s 2016-12-08 21:30:40,675 DEBUG - Bytes in read #12: 1061 , block bytes: 104011, remaining in block: 944565 duration=80706 nS; ns/ byte : 76, bandwidth=12.537463 MB/s 2016-12-08 21:30:40,677 DEBUG - Bytes in read #13: 16347 , block bytes: 120358, remaining in block: 928218 duration=1939042 nS; ns/ byte : 118, bandwidth=8.039905 MB/s 2016-12-08 21:30:40,677 DEBUG - Bytes in read #14: 1061 , block bytes: 121419, remaining in block: 927157 duration=70915 nS; ns/ byte : 66, bandwidth=14.268469 MB/s 2016-12-08 21:30:40,824 DEBUG - Bytes in read #15: 16347 , block bytes: 137766, remaining in block: 910810 duration=146688052 nS; ns/ byte : 8973, bandwidth=0.106278 MB/s 2016-12-08 21:30:40,825 DEBUG - Bytes in read #16: 1061 , block bytes: 138827, remaining in block: 909749 duration=135687 nS; ns/ byte : 127, bandwidth=7.457225 MB/s 2016-12-08 21:30:40,827 DEBUG - Bytes in read #17: 16347 , block bytes: 155174, remaining in block: 893402 duration=2159648 nS; ns/ byte : 132, bandwidth=7.218637 MB/s 2016-12-08 21:30:40,828 DEBUG - Bytes in read #18: 1061 , block bytes: 156235, remaining in block: 892341 duration=81945 nS; ns/ byte : 77, bandwidth=12.347897 MB/s 2016-12-08 21:30:40,830 DEBUG - Bytes in read #19: 16347 , block bytes: 172582, remaining in block: 875994 duration=1960358 nS; ns/ byte : 119, bandwidth=7.952483 MB/s 2016-12-08 21:30:40,830 DEBUG - Bytes in read #20: 1061 , block bytes: 173643, remaining in block: 874933 duration=80031 nS; ns/ byte : 75, bandwidth=12.643206 MB/s 2016-12-08 21:30:40,833 DEBUG - Bytes in read #21: 16347 , block bytes: 189990, remaining in block: 858586 duration=2730078 nS; ns/ byte : 167, bandwidth=5.710355 MB/s 2016-12-08 21:30:40,833 DEBUG - Bytes in read #22: 1061 , block bytes: 191051, remaining in block: 857525 duration=82210 nS; ns/ byte : 77, bandwidth=12.308095 MB/s 2016-12-08 21:30:40,836 DEBUG - Bytes in read #23: 16347 , block bytes: 207398, remaining in block: 841178 duration=2466188 nS; ns/ byte : 150, bandwidth=6.321381 MB/s 2016-12-08 21:30:40,836 DEBUG - Bytes in read #24: 1061 , block bytes: 208459, remaining in block: 840117 duration=75878 nS; ns/ byte : 71, bandwidth=13.335202 MB/s ... 2016-12-08 21:30:41,456 DEBUG - Bytes in read #118: 1061 , block bytes: 1026635, remaining in block: 21941 duration=69393 nS; ns/ byte : 65, bandwidth=14.581420 MB/s 2016-12-08 21:30:41,459 DEBUG - Bytes in read #119: 16347 , block bytes: 1042982, remaining in block: 5594 duration=2944939 nS; ns/ byte : 180, bandwidth=5.293731 MB/s 2016-12-08 21:30:41,459 DEBUG - Bytes in read #120: 1061 , block bytes: 1044043, remaining in block: 4533 duration=68294 nS; ns/ byte : 64, bandwidth=14.816067 MB/s 2016-12-08 21:30:41,462 DEBUG - Bytes in read #121: 4533 , block bytes: 1048576, remaining in block: 0 duration=3124460 nS; ns/ byte : 689, bandwidth=1.383601 MB/s 2016-12-08 21:30:41,463 INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 2 in 121 reads: 2,095,902,133 nS 2016-12-08 21:30:41,911 INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 3 in 121 reads: 447,989,141 nS 2016-12-08 21:30:41,911 INFO (testTimeToOpenAndReadWholeFileBlocks(254)) - Bandwidth of block 3: 2.232197 MB/s: 2016-12-08 21:30:42,212 INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Reading block 4 in 121 reads: 300,567,979 nS 2016-12-08 21:30:42,212 INFO (testTimeToOpenAndReadWholeFileBlocks(254)) - Bandwidth of block 4: 3.327034 MB/s: 2016-12-08 21:30:42,445 INFO (testTimeToOpenAndReadWholeFileBlocks(254)) - Bandwidth of block 5: 4.304842 MB/s: 2016-12-08 21:30:42,641 INFO (testTimeToOpenAndReadWholeFileBlocks(254)) - Bandwidth of block 6: 5.102818 MB/s: 2016-12-08 21:30:43,709 INFO (testTimeToOpenAndReadWholeFileBlocks(254)) - Bandwidth of block 8: 1.389905 MB/s: 2016-12-08 21:30:44,705 INFO (testTimeToOpenAndReadWholeFileBlocks(254)) - Bandwidth of block 9: 1.003957 MB/s: 2016-12-08 21:30:45,020 INFO (testTimeToOpenAndReadWholeFileBlocks(254)) - Bandwidth of block 10: 3.184452 MB/s: 2016-12-08 21:30:45,269 INFO (testTimeToOpenAndReadWholeFileBlocks(254)) - Bandwidth of block 11: 4.012955 MB/s: 2016-12-08 21:30:45,608 INFO (testTimeToOpenAndReadWholeFileBlocks(254)) - Bandwidth of block 12: 2.951960 MB/s: 2016-12-08 21:30:45,886 INFO (testTimeToOpenAndReadWholeFileBlocks(254)) - Bandwidth of block 13: 3.598163 MB/s: 2016-12-08 21:30:46,212 INFO (testTimeToOpenAndReadWholeFileBlocks(254)) - Bandwidth of block 14: 3.070112 MB/s: 2016-12-08 21:30:46,542 INFO (testTimeToOpenAndReadWholeFileBlocks(254)) - Bandwidth of block 15: 3.029587 MB/s: 2016-12-08 21:30:46,858 INFO (testTimeToOpenAndReadWholeFileBlocks(254)) - Bandwidth of block 16: 3.175305 MB/s: 2016-12-08 21:30:47,173 INFO (testTimeToOpenAndReadWholeFileBlocks(254)) - Bandwidth of block 17: 3.175283 MB/s: 2016-12-08 21:30:47,494 INFO (testTimeToOpenAndReadWholeFileBlocks(254)) - Bandwidth of block 18: 3.111757 MB/s: 2016-12-08 21:30:47,809 INFO (testTimeToOpenAndReadWholeFileBlocks(254)) - Bandwidth of block 19: 3.182531 MB/s: .... 2016-12-08 21:30:53,746 INFO (testTimeToOpenAndReadWholeFileBlocks(254)) - Bandwidth of block 37: 2.895820 MB/s: 2016-12-08 21:30:53,746 INFO contract.ContractTestUtils (ContractTestUtils.java:end(1441)) - Duration of Time to read 39278765 bytes in 37 blocks: 64,847,176,023 nS 2016-12-08 21:30:53,746 INFO (logStreamStatistics(366)) - Stream Statistics StreamStatistics{OpenOperations=2, CloseOperations=1, Closed=0, Aborted=1, SeekOperations=0, ReadExceptions=0, ForwardSeekOperations=0, BackwardSeekOperations=0, BytesSkippedOnSeek=0, BytesBackwardsOnSeek=0, BytesRead=38797312, BytesRead excluding skipped=38797312, ReadOperations=2833, ReadFullyOperations=0, ReadsIncomplete=2796, BytesReadInClose=0, BytesDiscardedInAbort=38230189} 2016-12-08 21:30:53,746 WARN (testTimeToOpenAndReadWholeFileBlocks(269)) - Reset the connection 1 time(s) 2016-12-08 21:30:53,747 INFO contract.AbstractFSContractTestBase (AbstractFSContractTestBase.java:describe(255)) - cleanup 2016-12-08 21:30:53,747 DEBUG conn.DefaultClientConnection (DefaultClientConnection.java:shutdown(166)) - Connection 0.0.0.0:57959<->54.231.169.22:443 shut down 2016-12-08 21:30:53,747 DEBUG conn.DefaultClientConnection (DefaultClientConnection.java:close(182)) - Connection 0.0.0.0:57959<->54.231.169.22:443 closed 2016-12-08 21:30:53,747 DEBUG conn.PoolingClientConnectionManager (PoolingClientConnectionManager.java:releaseConnection(274)) - Connection released: [id: 2][route: {s}->https: //landsat-pds.s3.amazonaws.com:443][total kept alive: 0; route allocated: 0 of 25; total allocated: 0 of 25] 2016-12-08 21:30:53,747 DEBUG s3a.S3AFileSystem (S3AInputStream.java:closeStream(463)) - Stream s3a: //landsat-pds/scene_list.gz aborted: close() operation; streamPos=38797312, nextReadPos=38797312, request range 1048576-39278765 length=39278765
          Hide
          stevel@apache.org Steve Loughran added a comment -

          This is clearly some network problem; I'm going reset everything locally and if that's not it, start escalating.

          what I can do, given there's an observable perf problem, is improve detection, diagnostics and reaction to it.

          This patch

          1. adds the ability to abort the current connection
          2. tracks bandwidth performance in the test, and aborts the connection if reading a 1MB block takes too long (the definition of long is hard coded)
          3. and fails the test if there are too many resets
          4. starts a troubleshooting doc, but doesn't yet link to it from the aws index page

          I'd thought about measuring duration in all reads and providing bandwidth since last reset as an input stream statistic, but it gets complex. For now, avoiding.

          Show
          stevel@apache.org Steve Loughran added a comment - This is clearly some network problem; I'm going reset everything locally and if that's not it, start escalating. what I can do, given there's an observable perf problem, is improve detection, diagnostics and reaction to it. This patch adds the ability to abort the current connection tracks bandwidth performance in the test, and aborts the connection if reading a 1MB block takes too long (the definition of long is hard coded) and fails the test if there are too many resets starts a troubleshooting doc, but doesn't yet link to it from the aws index page I'd thought about measuring duration in all reads and providing bandwidth since last reset as an input stream statistic, but it gets complex. For now, avoiding.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 9s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 6m 44s trunk passed
          +1 compile 0m 18s trunk passed
          +1 checkstyle 0m 13s trunk passed
          +1 mvnsite 0m 24s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 0m 26s trunk passed
          +1 javadoc 0m 14s trunk passed
          +1 mvninstall 0m 17s the patch passed
          +1 compile 0m 16s the patch passed
          +1 javac 0m 16s the patch passed
          -0 checkstyle 0m 10s hadoop-tools/hadoop-aws: The patch generated 1 new + 14 unchanged - 0 fixed = 15 total (was 14)
          +1 mvnsite 0m 21s the patch passed
          +1 mvneclipse 0m 12s the patch passed
          -1 whitespace 0m 0s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
          +1 findbugs 0m 29s the patch passed
          -1 javadoc 0m 11s hadoop-tools_hadoop-aws generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0)
          +1 unit 0m 19s hadoop-aws in the patch passed.
          +1 asflicense 0m 15s The patch does not generate ASF License warnings.
          12m 29s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HADOOP-13871
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12842424/HADOOP-13871-002.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 90df937809ab 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 13d8e55
          Default Java 1.8.0_111
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11223/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-aws.txt
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/11223/artifact/patchprocess/whitespace-eol.txt
          javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/11223/artifact/patchprocess/diff-javadoc-javadoc-hadoop-tools_hadoop-aws.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11223/testReport/
          modules C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11223/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 9s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 6m 44s trunk passed +1 compile 0m 18s trunk passed +1 checkstyle 0m 13s trunk passed +1 mvnsite 0m 24s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 0m 26s trunk passed +1 javadoc 0m 14s trunk passed +1 mvninstall 0m 17s the patch passed +1 compile 0m 16s the patch passed +1 javac 0m 16s the patch passed -0 checkstyle 0m 10s hadoop-tools/hadoop-aws: The patch generated 1 new + 14 unchanged - 0 fixed = 15 total (was 14) +1 mvnsite 0m 21s the patch passed +1 mvneclipse 0m 12s the patch passed -1 whitespace 0m 0s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply +1 findbugs 0m 29s the patch passed -1 javadoc 0m 11s hadoop-tools_hadoop-aws generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) +1 unit 0m 19s hadoop-aws in the patch passed. +1 asflicense 0m 15s The patch does not generate ASF License warnings. 12m 29s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-13871 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12842424/HADOOP-13871-002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 90df937809ab 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 13d8e55 Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11223/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-aws.txt whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/11223/artifact/patchprocess/whitespace-eol.txt javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/11223/artifact/patchprocess/diff-javadoc-javadoc-hadoop-tools_hadoop-aws.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11223/testReport/ modules C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11223/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          patch 003; addresses yetus complaints

          I am not seeing the problem today, I'm assuming it was a transient network outage. However, I would like the changes in, so if it surfaces again I get more information.

          Show
          stevel@apache.org Steve Loughran added a comment - patch 003; addresses yetus complaints I am not seeing the problem today, I'm assuming it was a transient network outage. However, I would like the changes in, so if it surfaces again I get more information.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 13s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 6m 43s trunk passed
          +1 compile 0m 19s trunk passed
          +1 checkstyle 0m 13s trunk passed
          +1 mvnsite 0m 25s trunk passed
          +1 mvneclipse 0m 15s trunk passed
          +1 findbugs 0m 27s trunk passed
          +1 javadoc 0m 14s trunk passed
          +1 mvninstall 0m 18s the patch passed
          +1 compile 0m 15s the patch passed
          +1 javac 0m 15s the patch passed
          +1 checkstyle 0m 11s the patch passed
          +1 mvnsite 0m 21s the patch passed
          +1 mvneclipse 0m 13s the patch passed
          -1 whitespace 0m 0s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
          +1 findbugs 0m 31s the patch passed
          +1 javadoc 0m 11s the patch passed
          +1 unit 0m 21s hadoop-aws in the patch passed.
          +1 asflicense 0m 15s The patch does not generate ASF License warnings.
          12m 37s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HADOOP-13871
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12842813/HADOOP-13871-003.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 6b8007591272 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / f66f618
          Default Java 1.8.0_111
          findbugs v3.0.0
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/11247/artifact/patchprocess/whitespace-eol.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11247/testReport/
          modules C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11247/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 6m 43s trunk passed +1 compile 0m 19s trunk passed +1 checkstyle 0m 13s trunk passed +1 mvnsite 0m 25s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 0m 27s trunk passed +1 javadoc 0m 14s trunk passed +1 mvninstall 0m 18s the patch passed +1 compile 0m 15s the patch passed +1 javac 0m 15s the patch passed +1 checkstyle 0m 11s the patch passed +1 mvnsite 0m 21s the patch passed +1 mvneclipse 0m 13s the patch passed -1 whitespace 0m 0s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply +1 findbugs 0m 31s the patch passed +1 javadoc 0m 11s the patch passed +1 unit 0m 21s hadoop-aws in the patch passed. +1 asflicense 0m 15s The patch does not generate ASF License warnings. 12m 37s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-13871 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12842813/HADOOP-13871-003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 6b8007591272 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / f66f618 Default Java 1.8.0_111 findbugs v3.0.0 whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/11247/artifact/patchprocess/whitespace-eol.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11247/testReport/ modules C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11247/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          mackrorysd Sean Mackrory added a comment -

          I think the JavaDoc comment for getS3AInputStream is out of date - still says it returns statistics, rather than being for the stream itself. +1 (non-binding) otherwise.

          Meant to post last week, but I wasn't seeing any S3 issues that day (and I was doing quite a lot on S3 - but not on branch-2.8).

          Show
          mackrorysd Sean Mackrory added a comment - I think the JavaDoc comment for getS3AInputStream is out of date - still says it returns statistics, rather than being for the stream itself. +1 (non-binding) otherwise. Meant to post last week, but I wasn't seeing any S3 issues that day (and I was doing quite a lot on S3 - but not on branch-2.8).
          Hide
          liuml07 Mingliang Liu added a comment -

          +1

          Show
          liuml07 Mingliang Liu added a comment - +1
          Hide
          liuml07 Mingliang Liu added a comment -

          Ran integration tests against US standard.

          Thanks Steve Loughran for great analysis and patch contribution. I have committed to trunk through branch-2.8 branches. For the commit, I fixed the whitespace warnings; and also addressed javadoc of getS3AInputStream() method, per Sean Mackrory's comment. Thanks for reviewing this.

          Show
          liuml07 Mingliang Liu added a comment - Ran integration tests against US standard. Thanks Steve Loughran for great analysis and patch contribution. I have committed to trunk through branch-2.8 branches. For the commit, I fixed the whitespace warnings; and also addressed javadoc of getS3AInputStream() method, per Sean Mackrory 's comment. Thanks for reviewing this.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10987 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10987/)
          HADOOP-13871. (liuml07: rev c6a39232456fa0c98b2b9b6dbeaec762294ca01e)

          • (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/scale/ITestS3AInputStreamPerformance.java
          • (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AInputStream.java
          • (add) hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/troubleshooting_s3a.md
          • (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AUtils.java
          • (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/scale/S3AScaleTestBase.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10987 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10987/ ) HADOOP-13871 . (liuml07: rev c6a39232456fa0c98b2b9b6dbeaec762294ca01e) (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/scale/ITestS3AInputStreamPerformance.java (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AInputStream.java (add) hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/troubleshooting_s3a.md (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AUtils.java (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/scale/S3AScaleTestBase.java

            People

            • Assignee:
              stevel@apache.org Steve Loughran
              Reporter:
              stevel@apache.org Steve Loughran
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development