[OAK-6611] [upgrade][oak-blob-cloud] Many S3DataStore errors during migration with oak-upgrade

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.7.7, 1.8.0
• Fix Version/s: 1.7.7, 1.6.6, 1.0.40, 1.4.19, 1.8.0, 1.2.28
• Component/s: blob-cloud, upgrade
• Assignee: Amit Jain
• Reporter: Arek Kita
• Created: 01/Sep/17 12:14
• Updated: 21/Nov/17 10:12
• Resolved: 06/Sep/17 09:06

Description

tomek.rekawek, amitjain Due to async nature of S3 datastore format/upload process the migration ends up way quicker than S3 datastore is being migrated. This leads to a huge number of exceptions shown due to non synchronised nature of oak-upgrade migration process vs async S3 datastore background processes.

I see a few possible solutions for that:

• disable migration/uploading of S3 cache for the time of migration (bad idea IMHO)
• wait for it (it might be desired or a bad idea as it might take longer than migration for a few cases)
• pause it when migration is completed in a clean way (so some binaries aren't uploaded and moved to a new datastore format) – not sure if such mixed state is ok at all

WDYT?

Please also note that this happens only when --src-s3config --src-s3datastore options are specified during migration which in many cases is true (this would be the same for the destination DataStore options).
Referencing a source datastore is needed (even if --copy-binaries is not included) in example to copy checkpoints properly.

The example exception is like the below:

01.09.2017 11:39:41.088 ERROR  o.a.j.o.p.b.UploadStagingCache: Error adding file to backend
java.lang.IllegalStateException: Connection pool shut down
	at org.apache.http.util.Asserts.check(Asserts.java:34)
	at org.apache.http.pool.AbstractConnPool.lease(AbstractConnPool.java:184)
	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.requestConnection(PoolingHttpClientConnectionManager.java:251)
	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76)
	at com.amazonaws.http.conn.$Proxy3.requestConnection(Unknown Source)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:175)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
	at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
	at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:880)
	at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:723)
	at com.amazonaws.http.AmazonHttpClient.doExecute(AmazonHttpClient.java:475)
	at com.amazonaws.http.AmazonHttpClient.executeWithTimer(AmazonHttpClient.java:437)
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:386)
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3996)
	at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1161)
	at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1136)
	at org.apache.jackrabbit.oak.blob.cloud.s3.S3Backend.write(S3Backend.java:201)
	at org.apache.jackrabbit.oak.plugins.blob.AbstractSharedCachingDataStore$2.write(AbstractSharedCachingDataStore.java:170)
	at org.apache.jackrabbit.oak.plugins.blob.UploadStagingCache$4.call(UploadStagingCache.java:341)
	at org.apache.jackrabbit.oak.plugins.blob.UploadStagingCache$4.call(UploadStagingCache.java:336)
	at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108)
	at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41)
	at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

Issue Links

• relates to: OAK-5135 The flush of written data via TarRevisions is asynchronous in relation to FileStore.close()
• relates to: OAK-5326 Not able to move segment store directory on filesystem after closing FileStore
• relates to: OAK-5436 o.a.j.o.segment.file.Manifest#load leaks a file descriptor
• relates to: OAK-5438 FileStore and ReadOnlyFileStore might leak journal file descriptors
• requires: OAK-6604 Oak Blob Cloud is not used by oak-upgrade