Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-4032

memtable.updateLiveRatio() is blocking, causing insane latencies for writes

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Normal
    • Resolution: Fixed
    • Fix Version/s: 1.1.0
    • Component/s: None
    • Labels:
      None
    • Severity:
      Normal

      Description

      Reproduce by just starting a fresh cassandra with a heap large enough for live ratio calculation (which is O) to be insanely slow, and then running ./bin/stress -d host -n100000000 -t10. With a large enough heap and default flushing behavior this is bad enough that stress gets timeouts.

      Example ("blocked for" is my debug log added around submit()):

       INFO [MemoryMeter:1] 2012-03-09 15:07:30,857 Memtable.java (line 198) CFS(Keyspace='Keyspace1', ColumnFamily='Standard1') liveRatio is 8.89014894083727 (just-counted was 8.89014894083727).  calculation took 28273ms for 1320245 columns
       WARN [MutationStage:8] 2012-03-09 15:07:30,857 Memtable.java (line 209) submit() blocked for: 231135
      

      The calling code was written assuming a RejectedExecutionException is thrown, but it's not because DebuggableThreadPoolExecutor installs a blocking rejection handler.

        Attachments

        1. 4032-v4.txt
          7 kB
          Jonathan Ellis
        2. 4032-v3.txt
          8 kB
          Jonathan Ellis
        3. CASSANDRA-4032-1.1.0-v2.txt
          8 kB
          Peter Schuller
        4. CASSANDRA-4032-1.1.0-v1.txt
          4 kB
          Peter Schuller

          Activity

            People

            • Assignee:
              scode Peter Schuller
              Reporter:
              scode Peter Schuller
              Authors:
              Peter Schuller
              Reviewers:
              Sylvain Lebresne
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: