Cassandra
  1. Cassandra
  2. CASSANDRA-3763

compactionstats throws ArithmeticException: / by zero

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Trivial Trivial
    • Resolution: Cannot Reproduce
    • Fix Version/s: None
    • Component/s: Core, Tools
    • Labels:
      None
    • Environment:

      debian linux - openvz kernel, oracle java 1.6.0.26

      Description

      compactionstats looks like this:

      1. nodetool -h localhost compactionstats
        Exception in thread "main" java.lang.ArithmeticException: / by zero
        at org.apache.cassandra.db.compaction.LeveledManifest.getEstimatedTasks(LeveledManifest.java:435)
        at org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getEstimatedRemainingTasks(LeveledCompactionStrategy.java:128)
        at org.apache.cassandra.db.compaction.CompactionManager.getPendingTasks(CompactionManager.java:1060)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93)
        at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27)
        at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208)
        at com.sun.jmx.mbeanserver.PerInterface.getAttribute(PerInterface.java:65)
        at com.sun.jmx.mbeanserver.MBeanSupport.getAttribute(MBeanSupport.java:216)
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:666)
        at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:638)
        at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1404)
        at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72)
        at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265)
        at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1360)
        at javax.management.remote.rmi.RMIConnectionImpl.getAttribute(RMIConnectionImpl.java:600)
        at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305)
        at sun.rmi.transport.Transport$1.run(Transport.java:159)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
        at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
        #

      nodetool is working fine in other actions:

      1. nodetool -h localhost netstats
        Mode: NORMAL
        Not sending any streams.
        Not receiving any streams.
        Pool Name Active Pending Completed
        Commands n/a 0 2
        Responses n/a 0 1810
        #

        Activity

        Hide
        Zenek Kraweznik added a comment -

        IT doesn't happens with sstable about 1GB, but with 4GB it occurs always

        Show
        Zenek Kraweznik added a comment - IT doesn't happens with sstable about 1GB, but with 4GB it occurs always
        Hide
        Zenek Kraweznik added a comment -

        LeveledCompacionStrategy seems to be broken. ArithmeticExceptions is thrown only with LeveledCompactionStrategy. Table compression has no influence on this.

        Show
        Zenek Kraweznik added a comment - LeveledCompacionStrategy seems to be broken. ArithmeticExceptions is thrown only with LeveledCompactionStrategy. Table compression has no influence on this.
        Hide
        Zenek Kraweznik added a comment -

        bug has to be linke with CASSANDRA-3693, I think fix for negative values on pending tasks causes this problem (maybe fix is like

        if pending_tasks < 0 then print('pending tasks: 0')

        Show
        Zenek Kraweznik added a comment - bug has to be linke with CASSANDRA-3693 , I think fix for negative values on pending tasks causes this problem (maybe fix is like if pending_tasks < 0 then print('pending tasks: 0')
        Hide
        Zenek Kraweznik added a comment - - edited

        this bug may be linked with CASSANDRA-3693 and CASSANDRA-3731

        Show
        Zenek Kraweznik added a comment - - edited this bug may be linked with CASSANDRA-3693 and CASSANDRA-3731
        Hide
        Zenek Kraweznik added a comment - - edited

        Problem appears after ugprading 1.0.6 to 1.0.7. I've create only one keyspace with 4 column family. All of my CF are:
        with column_type = 'Standard'
        and comparator = 'AsciiType'
        and default_validation_class = 'AsciiType'
        and key_validation_class = 'AsciiType'
        and rows_cached = 0.0
        and row_cache_save_period = 0
        and row_cache_keys_to_save = 2147483647
        and keys_cached = 200000.0
        and key_cache_save_period = 14400
        and read_repair_chance = 1.0
        and gc_grace = 864000
        and min_compaction_threshold = 2
        and max_compaction_threshold = 4
        and replicate_on_write = true
        and row_cache_provider = 'ConcurrentLinkedHashCacheProvider'
        and compaction_strategy = 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'
        and compaction_strategy_options =

        {'sstable_size_in_mb' : '4096'}

        and compression_options =

        {'chunk_length_kb' : '64', 'sstable_compression' : 'org.apache.cassandra.io.compress.SnappyCompressor'}

        ;

        system keyspace is untouched, I didn't change it.

        After upgrading node try to use 'nodetool -h $IP compactionstats'

        And that's all.

        Show
        Zenek Kraweznik added a comment - - edited Problem appears after ugprading 1.0.6 to 1.0.7. I've create only one keyspace with 4 column family. All of my CF are: with column_type = 'Standard' and comparator = 'AsciiType' and default_validation_class = 'AsciiType' and key_validation_class = 'AsciiType' and rows_cached = 0.0 and row_cache_save_period = 0 and row_cache_keys_to_save = 2147483647 and keys_cached = 200000.0 and key_cache_save_period = 14400 and read_repair_chance = 1.0 and gc_grace = 864000 and min_compaction_threshold = 2 and max_compaction_threshold = 4 and replicate_on_write = true and row_cache_provider = 'ConcurrentLinkedHashCacheProvider' and compaction_strategy = 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy' and compaction_strategy_options = {'sstable_size_in_mb' : '4096'} and compression_options = {'chunk_length_kb' : '64', 'sstable_compression' : 'org.apache.cassandra.io.compress.SnappyCompressor'} ; system keyspace is untouched, I didn't change it. After upgrading node try to use 'nodetool -h $IP compactionstats' And that's all.
        Hide
        Jonathan Ellis added a comment -

        I think you're going to need to tell us how to reproduce, because we haven't had any luck yet with the Zen Mind Powers approach.

        Show
        Jonathan Ellis added a comment - I think you're going to need to tell us how to reproduce, because we haven't had any luck yet with the Zen Mind Powers approach.
        Hide
        Zenek Kraweznik added a comment -

        nodetool from all affected version can read compactionstats from cassandra 1.0.6.

        nodetool from 1.0.6, 1.0.7, 1.0.8 cannot read compacionstats from affected versions of cassandra

        Show
        Zenek Kraweznik added a comment - nodetool from all affected version can read compactionstats from cassandra 1.0.6. nodetool from 1.0.6, 1.0.7, 1.0.8 cannot read compacionstats from affected versions of cassandra
        Hide
        Zenek Kraweznik added a comment - - edited

        No, i set sstable_size_in_mb to 4096 on all CF with LeveledCompactionStrategy.

        Also I have LeveledCompactionStrategy on my databases, but system databases has default comaction stragedy (tiered?).

        Show
        Zenek Kraweznik added a comment - - edited No, i set sstable_size_in_mb to 4096 on all CF with LeveledCompactionStrategy. Also I have LeveledCompactionStrategy on my databases, but system databases has default comaction stragedy (tiered?).
        Hide
        Dave Brosius added a comment -

        any chance you're specifying a compaction strategy option of sstable_size_in_mb = 0 ?

        Show
        Dave Brosius added a comment - any chance you're specifying a compaction strategy option of sstable_size_in_mb = 0 ?

          People

          • Assignee:
            Unassigned
            Reporter:
            Zenek Kraweznik
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development