[OAK-5135] The flush of written data via TarRevisions is asynchronous in relation to FileStore.close()

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.5.13
• Fix Version/s: 1.5.15, 1.6.0
• Component/s: segment-tar
• Flags: Important
• Assignee: Francesco Mari
• Reporter: Arek Kita
• Created: 21/Nov/16 14:06
• Updated: 01/Sep/17 12:30
• Resolved: 23/Nov/16 14:08

Description

Expected behavior: FileStore.close() should wait for any written data to be flushed to disk
Actual behavior: FileStore.close() does not wait for flushing data

Migration log

21.11.2016 12:00:59.792 WARN   o.a.j.o.s.f.Scheduler: The scheduler FileStore background tasks takes too long to shutdown
21.11.2016 12:01:10.363 INFO   o.a.j.o.s.f.FileStore: TarMK closed: /data/cq/crx-quickstart/repository-segment-tar-20161121-120039/segmentstore
21.11.2016 12:01:10.625 INFO   o.a.j.o.p.s.f.FileStore: TarMK closed: /data/cq/crx-quickstart/repository/segmentstore
...
### The above directories are swapped/moved here etc ###
...
21.11.2016 12:01:10.646 WARN   o.a.j.o.s.f.FileStore: Failed to flush the TarMK at /data/cq/crx-quickstart/repository-segment-tar-20161121-120039/segmentstore
java.io.IOException: Stream Closed
    at java.io.RandomAccessFile.writeBytes(Native Method) ~[na:1.8.0_101]
    at java.io.RandomAccessFile.writeBytes(RandomAccessFile.java:1100) ~[na:1.8.0_101]
    at org.apache.jackrabbit.oak.segment.file.TarRevisions.flush(TarRevisions.java:209) 
    at org.apache.jackrabbit.oak.segment.file.FileStore.flush(FileStore.java:358) 
    at org.apache.jackrabbit.oak.segment.file.FileStore$2.run(FileStore.java:221) 
    at org.apache.jackrabbit.oak.segment.file.SafeRunnable.run(SafeRunnable.java:67) 
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_101]
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_101]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_101]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_101]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_101]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_101]
    at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]

Issue Links

• is related to: OAK-6611 [upgrade][oak-blob-cloud] Many S3DataStore errors during migration with oak-upgrade
• relates to: OAK-5326 Not able to move segment store directory on filesystem after closing FileStore