Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-7242

More compaction visibility into thread pool and per CF

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Low
    • Resolution: Fixed
    • 2.0.8, 2.1 rc1
    • None
    • None

    Description

      Two parts to this to help diagnose compactions issues/bottlenecks. Could be two different issues but pretty closely related.

      First is adding per column family pending compactions. When theres a lot of backed up compactions but multiple ones currently being compacted its hard to identify which CF is causing the backlog. In patch provided this doesnt cover the compactions in the thread pools queue like compactionstats does but not sure how big that gets ever or if needs to be... which brings me to the second idea.

      Second is to change compactionExecutor to extend the JMXEnabledThreadPoolExecutor. Big difference there would be the blocking rejection handler. With a 2^31 pending queue the blocking becoming an issue is a pretty extreme case in itself that would most likely OOM the server. So the different rejection policy shouldn't cause much of an issue but if it does can always override it to use default behavior. Would help identify scenarios where corrupted sstables or unhandled exceptions etc killing the compactions lead to a large backlog with nothing actively working. Also just for added visibility into this from tpstats.

      Attachments

        1. 7242_jmxify_compactionpool.txt
          2 kB
          Chris Lohfink
        2. 7242_per_cf_compactionstats.txt
          1 kB
          Chris Lohfink

        Activity

          People

            cnlwsu Chris Lohfink
            cnlwsu Chris Lohfink
            Chris Lohfink
            Marcus Eriksson
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: