[BROOKLYN-144] Log flooded with errors: Problem in HA-poller... No such file or directory

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.7.0
• Fix Version/s: 0.9.0
• Assignee: Unassigned
• Reporter: Aled Sage
• Created: 13/May/15 09:27
• Updated: 27/Apr/17 22:58
• Resolved: 27/Apr/17 22:58

Description

A customer sent through a log that is flooded with ERROR stacktraces, one per second, such as:

2015-05-13 06:12:19,145 ERROR b.m.h.HighAvailabilityManagerImpl [brooklyn-execmanager-hDhPL5ud-867]: Problem in HA-poller: brooklyn.util.exceptions.PropagatedRuntimeException: java.io.IOException: No such file or directory
brooklyn.util.exceptions.PropagatedRuntimeException: 
        at brooklyn.util.exceptions.Exceptions.propagate(Exceptions.java:93) ~[brooklyn-utils-common-0.7.0-20150509.1751.jar:0.7.0-20150509.1751]
        at brooklyn.entity.rebind.persister.FileBasedStoreObjectAccessor.put(FileBasedStoreObjectAccessor.java:89) ~[brooklyn-core-0.7.0-20150509.1751.jar:0.7.0-20150509.1751]
        at brooklyn.entity.rebind.persister.StoreObjectAccessorLocking.put(StoreObjectAccessorLocking.java:136) ~[brooklyn-core-0.7.0-20150509.1751.jar:0.7.0-20150509.1751]
        at brooklyn.management.ha.ManagementPlaneSyncRecordPersisterToObjectStore.persist(ManagementPlaneSyncRecordPersisterToObjectStore.java:321) ~[brooklyn-core-0.7.0-20150509.1751.jar:0.7.0-20150509.1751]
        at brooklyn.management.ha.ManagementPlaneSyncRecordPersisterToObjectStore.delta(ManagementPlaneSyncRecordPersisterToObjectStore.java:252) ~[brooklyn-core-0.7.0-20150509.1751.jar:0.7.0-20150509.1751]
        at brooklyn.management.ha.HighAvailabilityManagerImpl.publishHealth(HighAvailabilityManagerImpl.java:591) ~[brooklyn-core-0.7.0-20150509.1751.jar:0.7.0-20150509.1751]
        at brooklyn.management.ha.HighAvailabilityManagerImpl.publishAndCheck(HighAvailabilityManagerImpl.java:577) ~[brooklyn-core-0.7.0-20150509.1751.jar:0.7.0-20150509.1751]
        at brooklyn.management.ha.HighAvailabilityManagerImpl$2.run(HighAvailabilityManagerImpl.java:540) ~[brooklyn-core-0.7.0-20150509.1751.jar:0.7.0-20150509.1751]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_75]
        at brooklyn.util.task.BasicExecutionManager$ScheduledTaskCallable$1.call(BasicExecutionManager.java:410) [brooklyn-core-0.7.0-20150509.1751.jar:0.7.0-20150509.1751]
        at brooklyn.util.task.BasicExecutionManager$SubmissionCallable.call(BasicExecutionManager.java:469) [brooklyn-core-0.7.0-20150509.1751.jar:0.7.0-20150509.1751]
        at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_75]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_75]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_75]
        at java.lang.Thread.run(Thread.java:745) [na:1.7.0_75]
Caused by: java.io.IOException: No such file or directory
        at java.io.UnixFileSystem.createFileExclusively(Native Method) ~[na:1.7.0_75]
        at java.io.File.createNewFile(File.java:1006) ~[na:1.7.0_75]
        at brooklyn.util.io.FileUtil.setFilePermissionsTo600(FileUtil.java:78) ~[brooklyn-utils-common-0.7.0-20150509.1751.jar:0.7.0-20150509.1751]
        at brooklyn.entity.rebind.persister.FileBasedStoreObjectAccessor.put(FileBasedStoreObjectAccessor.java:85) ~[brooklyn-core-0.7.0-20150509.1751.jar:0.7.0-20150509.1751]
        ... 13 common frames omitted

I would have expected one ERROR message to be logged to report the problem, and subsequent (identical) problems to be logged at just DEBUG. If we log this many errors, it will cause our log files to fill up and roll over too quickly (hiding other more interesting info).

The underlying exception might be caused by a mis-configuration for persistence, but that is not the point of this jira issue.