Details
Major Description
We have quite a big project with custom karaf disctribution. Recently we have begun the migration from 3.x to 4.x karaf.
Our core feature is placed into <installedFeatures> section.
Additional features are placed into deploy folder with maven-dependency-plugin (copy-to-deploy) as KAR files.
There are 3 karaf archives in the deploy folder after our assembly is done.
After startup the karaf (with karaf.bat file), 1 kar is usually being deployed well, 2 rest fails.
In karaf.log we can see following exceptions:
2016-12-15 14:36:55,759 | INFO | \assembly/deploy | FeaturesServiceImpl | 8 - org.apache.karaf.features.core - 4.0.7 | Adding features: snamp-http-acceptor/[2.0.0,2.0.0]
2016-12-15 14:36:55,759 | WARN | \assembly/deploy | KarServiceImpl | 36 - org.apache.karaf.kar.core - 4.0.7 | Unable to install Kar feature snamp-http-acceptor/2.0.0
java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@5fd6d173 rejected from java.util.concurrent.ThreadPoolExecutor@37659c9[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 5]
at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047)[:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)[:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)[:1.8.0_111]
at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134)[:1.8.0_111]
at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:681)[:1.8.0_111]
at org.apache.karaf.features.internal.service.FeaturesServiceImpl.doProvisionInThread(FeaturesServiceImpl.java:1071)[8:org.apache.karaf.features.core:4.0.7]
at org.apache.karaf.features.internal.service.FeaturesServiceImpl.installFeatures(FeaturesServiceImpl.java:954)[8:org.apache.karaf.features.core:4.0.7]
at org.apache.karaf.features.internal.service.FeaturesServiceImpl.installFeatures(FeaturesServiceImpl.java:847)[8:org.apache.karaf.features.core:4.0.7]
at org.apache.karaf.features.internal.service.FeaturesServiceImpl.installFeature(FeaturesServiceImpl.java:832)[8:org.apache.karaf.features.core:4.0.7]
at org.apache.karaf.features.internal.service.FeaturesServiceImpl.installFeature(FeaturesServiceImpl.java:842)[8:org.apache.karaf.features.core:4.0.7]
at org.apache.karaf.kar.internal.KarServiceImpl.installFeatures(KarServiceImpl.java:283)[36:org.apache.karaf.kar.core:4.0.7]
at org.apache.karaf.kar.internal.KarServiceImpl.install(KarServiceImpl.java:112)[36:org.apache.karaf.kar.core:4.0.7]
at org.apache.karaf.kar.internal.KarServiceImpl.install(KarServiceImpl.java:94)[36:org.apache.karaf.kar.core:4.0.7]
at org.apache.karaf.deployer.kar.KarArtifactInstaller.install(KarArtifactInstaller.java:50)[27:org.apache.karaf.deployer.kar:4.0.7]
at org.apache.felix.fileinstall.internal.DirectoryWatcher.install(DirectoryWatcher.java:937)[4:org.apache.felix.fileinstall:3.5.4]
at org.apache.felix.fileinstall.internal.DirectoryWatcher.install(DirectoryWatcher.java:871)[4:org.apache.felix.fileinstall:3.5.4]
at org.apache.felix.fileinstall.internal.DirectoryWatcher.doProcess(DirectoryWatcher.java:485)[4:org.apache.felix.fileinstall:3.5.4]
at org.apache.felix.fileinstall.internal.DirectoryWatcher.process(DirectoryWatcher.java:361)[4:org.apache.felix.fileinstall:3.5.4]
at org.apache.felix.fileinstall.internal.DirectoryWatcher.run(DirectoryWatcher.java:312)[4:org.apache.felix.fileinstall:3.5.4]
After that we are able to install our features with console commands (feature:install) - it never causes such exceptions.
Please note, that this defect can be reproduced not always, but almost always.
Our proposition - something goes wrong in class FeaturesServiceImpl (https://github.com/apache/karaf/blob/karaf-4.0.7/features/core/src/main/java/org/apache/karaf/features/internal/service/FeaturesServiceImpl.java) - executor is closed for some reasons at the moment when next feature is being deployed. The main reason is permanent refreshing of bundle org.apache.karaf.features.core during installation of our features. But out features have no explicit dependency on this bundle. Method FeaturesServiceImpl.doProvisionInThread relies on private field 'executor'. The executor can be terminated using method FeaturesServiceImpl.stop(). Theoretically, refreshing of bundle org.apache.karaf.features.core causes concurrency when service is stopping and new provisioning task arriving at the same time.
We also have set noAutoRefreshBundles=true in org.apache.karaf.kar.cfg, but without success.