Uploaded image for project: 'Accumulo'
  1. Accumulo
  2. ACCUMULO-4080

TabletServers should be less aggressively "monitoring RO filesystems"

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.6.4, 1.7.0
    • Fix Version/s: 1.6.5, 1.7.1, 1.8.0
    • Component/s: tserver
    • Labels:
      None
    • Environment:

      uname -a
      Linux 3.10.0-123.9.3.el7.x86_64 #1 SMP Thu Nov 6 15:06:03 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
      cat /etc/redhat-release
      CentOS Linux release 7.0.1406 (Core)

      Description

      Ran into an automated test case where all of the tservers killed themselves on Centos7.

      2015-12-17 14:51:30,164 [util.FileSystemMonitor] FATAL: Exception while checking mount points, halting process
      java.lang.Exception: Filesystem /sys/fs/cgroup switched to read only
              at org.apache.accumulo.server.util.FileSystemMonitor.checkMounts(FileSystemMonitor.java:123)
              at org.apache.accumulo.server.util.FileSystemMonitor$1.run(FileSystemMonitor.java:90)
              at java.util.TimerThread.mainLoop(Timer.java:555)
              at java.util.TimerThread.run(Timer.java:505)
      

      I'm not quite sure what exactly happened that caused /sys/fs/cgroup to suddenly be mounted as ro (my hunch is that it was an updated package).

      A workaround is to set tserver.monitor.fs to false in accumulo-site.xml and restart Accumulo.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                elserj Josh Elser
                Reporter:
                elserj Josh Elser
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1h 20m
                  1h 20m