[OAK-6884] TarMK disk space check is not synchronized with FileStore opened state

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.7.9
• Fix Version/s: 1.7.14, 1.8.0
• Component/s: segment-tar, upgrade
• Labels: concurrency production
• Assignee: Michael Dürig
• Reporter: Arek Kita
• Created: 30/Oct/17 14:50
• Updated: 30/Dec/17 09:06
• Resolved: 07/Dec/17 11:28

Description

It seems that the disk space check is not properly synchronized with FileStore as I revealed a race condition while using oak-upgrade during migration to segment-tar.

The FileStore instance is closed while TarMK disk check tries to execute and it seems it is dependent on the state of segment (org.apache.jackrabbit.oak.segment.file.FileStore.checkDiskSpace(FileStore.java:541) that needs to be opened.

30.10.2017 11:26:05.834 WARN   o.a.j.o.s.f.Scheduler: The scheduler FileStore background tasks takes too long to shut down
30.10.2017 11:26:11.674 INFO   o.a.j.o.s.f.FileStore: TarMK closed: /data/cq/crx-quickstart/repository-segment-tar-20171030-112401/segmentstore
30.10.2017 11:26:11.676 ERROR  o.a.j.o.s.f.SafeRunnable: Uncaught exception in TarMK disk space check [/data/cq/crx-quickstart/repository-segment-tar-20171030-112401/segmentstore]
java.lang.IllegalStateException: already shut down
    at org.apache.jackrabbit.oak.segment.file.ShutDown.keepAlive(ShutDown.java:42)
    at org.apache.jackrabbit.oak.segment.file.FileStore.size(FileStore.java:302)
    at org.apache.jackrabbit.oak.segment.file.FileStore.checkDiskSpace(FileStore.java:541)
    at org.apache.jackrabbit.oak.segment.file.FileStore.access$300(FileStore.java:102)
    at org.apache.jackrabbit.oak.segment.file.FileStore$3.run(FileStore.java:237)
    at org.apache.jackrabbit.oak.segment.file.SafeRunnable.run(SafeRunnable.java:67)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)