Uploaded image for project: 'Apache Cassandra'
  1. Apache Cassandra
  2. CASSANDRA-11063

Unable to compute ceiling for max when histogram overflowed

Agile BoardAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Normal
    • Resolution: Fixed
    • None
    • Local/Compaction
    • Cassandra 2.1.9 on RHEL

    • Normal

    Description

      Issue https://issues.apache.org/jira/browse/CASSANDRA-8028 seems related with error we are getting. But we are getting this with Cassandra 2.1.9 when autocompaction is running it keeps throwing following errors, we are unsure if its a bug or can be resolved, please suggest.

      WARN  [CompactionExecutor:3] 2016-01-23 13:30:40,907 SSTableWriter.java:240 - Compacting large partition gccatlgsvcks/category_name_dedup:66611300 (138152195 bytes)
      ERROR [CompactionExecutor:1] 2016-01-23 13:30:50,267 CassandraDaemon.java:223 - Exception in thread Thread[CompactionExecutor:1,1,main]
      java.lang.IllegalStateException: Unable to compute ceiling for max when histogram overflowed
              at org.apache.cassandra.utils.EstimatedHistogram.mean(EstimatedHistogram.java:203) ~[apache-cassandra-2.1.9.jar:2.1.9]
              at org.apache.cassandra.io.sstable.metadata.StatsMetadata.getEstimatedDroppableTombstoneRatio(StatsMetadata.java:98) ~[apache-cassandra-2.1.9.jar:2.1.9]
              at org.apache.cassandra.io.sstable.SSTableReader.getEstimatedDroppableTombstoneRatio(SSTableReader.java:1987) ~[apache-cassandra-2.1.9.jar:2.1.9]
              at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.worthDroppingTombstones(AbstractCompactionStrategy.java:370) ~[apache-cassandra-2.1.9.jar:2.1.9]
              at org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy.getNextBackgroundSSTables(SizeTieredCompactionStrategy.java:96) ~[apache-cassandra-2.1.9.jar:2.1.9]
              at org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy.getNextBackgroundTask(SizeTieredCompactionStrategy.java:179) ~[apache-cassandra-2.1.9.jar:2.1.9]
              at org.apache.cassandra.db.compaction.WrappingCompactionStrategy.getNextBackgroundTask(WrappingCompactionStrategy.java:84) ~[apache-cassandra-2.1.9.jar:2.1.9]
              at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:230) ~[apache-cassandra-2.1.9.jar:2.1.9]
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.7.0_51]
              at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_51]
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_51]
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_51]
              at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51]
      

      Additional info:

      cfstats is running fine for that table...

      ~ $ nodetool cfstats gccatlgsvcks.category_name_dedup
      Keyspace: gccatlgsvcks
              Read Count: 0
              Read Latency: NaN ms.
              Write Count: 0
              Write Latency: NaN ms.
              Pending Flushes: 0
                      Table: category_name_dedup
                      SSTable count: 6
                      Space used (live): 836314727
                      Space used (total): 836314727
                      Space used by snapshots (total): 3621519
                      Off heap memory used (total): 6930368
                      SSTable Compression Ratio: 0.03725358753117693
                      Number of keys (estimate): 3004
                      Memtable cell count: 0
                      Memtable data size: 0
                      Memtable off heap memory used: 0
                      Memtable switch count: 0
                      Local read count: 0
                      Local read latency: NaN ms
                      Local write count: 0
                      Local write latency: NaN ms
                      Pending flushes: 0
                      Bloom filter false positives: 0
                      Bloom filter false ratio: 0.00000
                      Bloom filter space used: 5240
                      Bloom filter off heap memory used: 5192
                      Index summary off heap memory used: 1200
                      Compression metadata off heap memory used: 6923976
                      Compacted partition minimum bytes: 125
                      Compacted partition maximum bytes: 30753941057
                      Compacted partition mean bytes: 8352388
                      Average live cells per slice (last five minutes): 0.0
                      Maximum live cells per slice (last five minutes): 0.0
                      Average tombstones per slice (last five minutes): 0.0
                      Maximum tombstones per slice (last five minutes): 0.0
      
      

      cfhistograms is also running fine...

      ~ $ nodetool cfhistograms gccatlgsvcks category_name_dedup
      gccatlgsvcks/category_name_dedup histograms
      Percentile  SSTables     Write Latency      Read Latency    Partition Size        Cell Count
                                    (micros)          (micros)           (bytes)
      50%             0.00              0.00              0.00              1109                20
      75%             0.00              0.00              0.00              2299                42
      95%             0.00              0.00              0.00             11864               215
      98%             0.00              0.00              0.00             35425               642
      99%             0.00              0.00              0.00             51012               924
      Min             0.00              0.00              0.00               125                 4
      Max             0.00              0.00              0.00       30753941057         268650950
      

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned Assign to me
            nnishant Navjyot Nishant
            Votes:
            1 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment