Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-694

Failure to flush commit log

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Normal
    • Resolution: Fixed
    • Fix Version/s: 0.5
    • Component/s: None
    • Labels:
      None
    • Environment:

      Linux 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:39:36 EST 2008 i686 GNU/Linux, ec2 small instance

    • Severity:
      Normal

      Description

      The following exception occurs consistently on at least node (note did not occur on other same-configured nodes) during startup:

      INFO - Replaying /var/lib/cassandra/commitlog/CommitLog-1262855754427.log, /var/lib/cassandra/commitlog/CommitLog-1262832689989.log, /var/lib/cassandra/commitlog/CommitLog-1262885833186.log, /var/lib/cassandra/commitlog/CommitLog-1262900845019.log, /var/lib/cassandra/commitlog/CommitLog-1262913267844.log, /var/lib/cassandra/commitlog/CommitLog-1262927898170.log, /var/lib/cassandra/commitlog/CommitLog-1262961421039.log, /var/lib/cassandra/commitlog/CommitLog-1262977175175.log, /var/lib/cassandra/commitlog/CommitLog-1262989588783.log, /var/lib/cassandra/commitlog/CommitLog-1263000573676.log, /var/lib/cassandra/commitlog/CommitLog-1263013691393.log, /var/lib/cassandra/commitlog/CommitLog-1263044706108.log, /var/lib/cassandra/commitlog/CommitLog-1263060004191.log, /var/lib/cassandra/commitlog/CommitLog-1263071446342.log, /var/lib/cassandra/commitlog/CommitLog-1263082950154.log, /var/lib/cassandra/commitlog/CommitLog-1263095400814.log, /var/lib/cassandra/commitlog/CommitLog-1263118331046.log, /var/lib/cassandra/commitlog/CommitLog-1263143402963.log, /var/lib/cassandra/commitlog/CommitLog-1263155294308.log, /var/lib/cassandra/commitlog/CommitLog-1263166154352.log, /var/lib/cassandra/commitlog/CommitLog-1263178359247.log, /var/lib/cassandra/commitlog/CommitLog-1263202112017.log, /var/lib/cassandra/commitlog/CommitLog-1263230932274.log, /var/lib/cassandra/commitlog/CommitLog-1263250726505.log, /var/lib/cassandra/commitlog/CommitLog-1263264159438.log, /var/lib/cassandra/commitlog/CommitLog-1263289964249.log, /var/lib/cassandra/commitlog/CommitLog-1263317974387.log, /var/lib/cassandra/commitlog/CommitLog-1263331989090.log, /var/lib/cassandra/commitlog/CommitLog-1263344147667.log, /var/lib/cassandra/commitlog/CommitLog-1263359751527.log, /var/lib/cassandra/commitlog/CommitLog-1263395707008.log, /var/lib/cassandra/commitlog/CommitLog-1263397833524.log, /var/lib/cassandra/commitlog/CommitLog-1263398736183.log, /var/lib/cassandra/commitlog/CommitLog-1263399753707.log, /var/lib/cassandra/commitlog/CommitLog-1263401667504.log, /var/lib/cassandra/commitlog/CommitLog-1263404640782.log, /var/lib/cassandra/commitlog/CommitLog-1263405827234.log, /var/lib/cassandra/commitlog/CommitLog-1263406901115.log
      INFO - LocationInfo has reached its threshold; switching in a fresh Memtable
      INFO - Enqueuing flush of Memtable(LocationInfo)@25934689
      INFO - HintsColumnFamily has reached its threshold; switching in a fresh Memtable
      INFO - Enqueuing flush of Memtable(HintsColumnFamily)@4766820
      INFO - AdXRequestStatistics has reached its threshold; switching in a fresh Memtable
      INFO - Enqueuing flush of Memtable(AdXRequestStatistics)@21521158
      INFO - TokenGoogleIDCF has reached its threshold; switching in a fresh Memtable
      INFO - Enqueuing flush of Memtable(TokenGoogleIDCF)@22889075
      java.lang.reflect.InvocationTargetException
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.apache.commons.daemon.support.DaemonLoader.load(DaemonLoader.java:160)
      Caused by: java.lang.AssertionError: Blocking serialized executor is not yet implemented
      at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:84)
      at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
      at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
      at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:78)
      at org.apache.cassandra.db.ColumnFamilyStore.submitFlush(ColumnFamilyStore.java:1045)
      at org.apache.cassandra.db.ColumnFamilyStore.switchMemtable(ColumnFamilyStore.java:395)
      at org.apache.cassandra.db.ColumnFamilyStore.forceFlush(ColumnFamilyStore.java:448)
      at org.apache.cassandra.db.Table.flush(Table.java:464)
      at org.apache.cassandra.db.CommitLog.recover(CommitLog.java:397)
      at org.apache.cassandra.db.RecoveryManager.doRecovery(RecoveryManager.java:65)
      at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:90)
      at org.apache.cassandra.service.CassandraDaemon.init(CassandraDaemon.java:135)
      ... 5 more

      And the same exception occurs intermittently on other node (running) nodes during 'nodeprobe flush':

      root@domU-12-31-38-00-26-31:~# nodeprobe -host localhost -port 8080 flush Logger
      Exception in thread "main" java.lang.AssertionError: Blocking serialized executor is not yet implemented
      at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:84)
      at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
      at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
      at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:78)
      at org.apache.cassandra.db.ColumnFamilyStore.submitFlush(ColumnFamilyStore.java:1045)
      at org.apache.cassandra.db.ColumnFamilyStore.switchMemtable(ColumnFamilyStore.java:395)
      at org.apache.cassandra.db.ColumnFamilyStore.forceFlush(ColumnFamilyStore.java:448)
      at org.apache.cassandra.service.StorageService.forceTableFlush(StorageService.java:984)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93)
      at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27)
      at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208)
      at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120)
      at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262)
      at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836)
      at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761)
      at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1426)
      at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72)
      at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1264)
      at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1359)
      at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305)
      at sun.rmi.transport.Transport$1.run(Transport.java:159)
      at java.security.AccessController.doPrivileged(Native Method)
      at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
      at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
      at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
      at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      at java.lang.Thread.run(Thread.java:619)

        Attachments

        1. 694-0.5.txt
          3 kB
          Jonathan Ellis
        2. 694-trunk.txt
          5 kB
          Jonathan Ellis

          Activity

            People

            • Assignee:
              jbellis Jonathan Ellis
              Reporter:
              ryandaum Ryan Daum
              Authors:
              Jonathan Ellis
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: