[ACCUMULO-4080] TabletServers should be less aggressively "monitoring RO filesystems"

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.6.4, 1.7.0
• Fix Version/s: 1.6.5, 1.7.1, 1.8.0
• Component/s: tserver

Environment

uname -a
Linux 3.10.0-123.9.3.el7.x86_64 #1 SMP Thu Nov 6 15:06:03 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
cat /etc/redhat-release
CentOS Linux release 7.0.1406 (Core)

• Assignee: Josh Elser
• Reporter: Josh Elser
• Created: 17/Dec/15 17:54
• Updated: 03/Jun/16 18:12
• Resolved: 18/Dec/15 19:09

Description

Ran into an automated test case where all of the tservers killed themselves on Centos7.

2015-12-17 14:51:30,164 [util.FileSystemMonitor] FATAL: Exception while checking mount points, halting process
java.lang.Exception: Filesystem /sys/fs/cgroup switched to read only
        at org.apache.accumulo.server.util.FileSystemMonitor.checkMounts(FileSystemMonitor.java:123)
        at org.apache.accumulo.server.util.FileSystemMonitor$1.run(FileSystemMonitor.java:90)
        at java.util.TimerThread.mainLoop(Timer.java:555)
        at java.util.TimerThread.run(Timer.java:505)

I'm not quite sure what exactly happened that caused /sys/fs/cgroup to suddenly be mounted as ro (my hunch is that it was an updated package).

A workaround is to set tserver.monitor.fs to false in accumulo-site.xml and restart Accumulo.

Issue Links

• links to: GitHub Pull Request #58