Cassandra
  1. Cassandra
  2. CASSANDRA-809

Full disk can result in being marked down

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Duplicate
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      We had a node file up the disk under one of two data directories. The result was that the node stopped making progress. The problem appears to be this (I'll update with more details as we find them):

      When new tasks are put onto most queues in Cassandra, if there isn't a thread in the pool to handle the task immediately, the task in run in the caller's thread
      (org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor:69 sets the caller-runs policy). The queue in question here is the queue that manages flushes, which is enqueued to from various places in our code (and therefore likely from multiple threads). Assuming that the full disk meant that no threads doing flushing could make progress (it appears that way) eventually any thread that calls the flush code would become stalled.

      Assuming our analysis is right (and we're still looking into it) we need to make a change. Here's a proposal so far:

      SHORT TERM:

      • change the TheadPoolExecutor policy to not be caller runs. This will let other threads make progress in the event that one pool is stalled

      LONG TERM

      • It appears that there are n threads for n data directories that we flush to, but they're not dedicated to a data directory. We should have a thread per data directory and have that thread dedicated to that directory
      • Perhaps we could use the failure detector on disks?

        Activity

        Ryan King created issue -
        Jonathan Ellis made changes -
        Field Original Value New Value
        Fix Version/s 0.7 [ 12314533 ]
        Affects Version/s 0.5 [ 12314040 ]
        Affects Version/s 0.6 [ 12314361 ]
        Affects Version/s 0.7 [ 12314533 ]
        Jonathan Ellis made changes -
        Fix Version/s 0.7 [ 12314533 ]
        Fix Version/s 0.8 [ 12314820 ]
        Jonathan Ellis made changes -
        Priority Major [ 3 ] Minor [ 4 ]
        Ryan King made changes -
        Fix Version/s 1.0 [ 12316349 ]
        Fix Version/s 0.8 [ 12314820 ]
        Jonathan Ellis made changes -
        Fix Version/s 1.1 [ 12317615 ]
        Fix Version/s 1.0 [ 12316349 ]
        Jonathan Ellis made changes -
        Fix Version/s 1.2 [ 12319262 ]
        Fix Version/s 1.1 [ 12317615 ]
        Jonathan Ellis made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Fix Version/s 1.2 [ 12319262 ]
        Resolution Duplicate [ 3 ]
        Gavin made changes -
        Workflow no-reopen-closed, patch-avail [ 12499449 ] patch-available, re-open possible [ 12749787 ]
        Gavin made changes -
        Workflow patch-available, re-open possible [ 12749787 ] reopen-resolved, no closed status, patch-avail, testing [ 12757309 ]

          People

          • Assignee:
            Unassigned
            Reporter:
            Ryan King
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development