Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Fix Version/s: 1.0.9
    • Component/s: None
    • Labels:
    • Environment:

      Ubuntu LTS 10.04, OpenJDK 1.6.0_20

      Description

      We just recently started using version 1.0.9, previously we were using tiered compaction because of a bug in 1.0.8 (not letting us use leveled compaction) and now since moving to 1.0.9 we have started using leveled compaction.

      Trying to do a cleanup we are getting the following exception:

      root@test:~# nodetool -h localhost cleanup
      Error occured during cleanup
      java.util.concurrent.ExecutionException: java.util.NoSuchElementException
      at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
      at java.util.concurrent.FutureTask.get(FutureTask.java:111)
      at org.apache.cassandra.db.compaction.CompactionManager.performAllSSTableOperation(CompactionManager.java:204)
      at org.apache.cassandra.db.compaction.CompactionManager.performCleanup(CompactionManager.java:240)
      at org.apache.cassandra.db.ColumnFamilyStore.forceCleanup(ColumnFamilyStore.java:988)
      at org.apache.cassandra.service.StorageService.forceTableCleanup(StorageService.java:1639)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:616)
      at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
      at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
      at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
      at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
      at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251)
      at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)
      at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
      at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1450)
      at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90)
      at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1285)
      at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1383)
      at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807)
      at sun.reflect.GeneratedMethodAccessor31.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:616)
      at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
      at sun.rmi.transport.Transport$1.run(Transport.java:177)
      at java.security.AccessController.doPrivileged(Native Method)
      at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
      at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
      at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
      at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
      at java.lang.Thread.run(Thread.java:636)
      Caused by: java.util.NoSuchElementException
      at java.util.ArrayList$Itr.next(ArrayList.java:757)
      at org.apache.cassandra.db.compaction.LeveledManifest.replace(LeveledManifest.java:196)
      at org.apache.cassandra.db.compaction.LeveledCompactionStrategy.handleNotification(LeveledCompactionStrategy.java:147)
      at org.apache.cassandra.db.DataTracker.notifySSTablesChanged(DataTracker.java:495)
      at org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:235)
      at org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:1010)
      at org.apache.cassandra.db.compaction.CompactionManager.doCleanupCompaction(CompactionManager.java:802)
      at org.apache.cassandra.db.compaction.CompactionManager.access$300(CompactionManager.java:64)
      at org.apache.cassandra.db.compaction.CompactionManager$5.perform(CompactionManager.java:244)
      at org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:183)
      at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
      at java.util.concurrent.FutureTask.run(FutureTask.java:166)

      cheers,
      Shoaib

      1. 4112.txt
        2 kB
        Jonathan Ellis
      2. 4112_v2.txt
        2 kB
        Sylvain Lebresne

        Activity

        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Patch Available Patch Available
        2h 5m 1 Jonathan Ellis 03/Apr/12 03:24
        Patch Available Patch Available Resolved Resolved
        12h 11m 1 Sylvain Lebresne 03/Apr/12 15:35
        Gavin made changes -
        Workflow patch-available, re-open possible [ 12749004 ] reopen-resolved, no closed status, patch-avail, testing [ 12756778 ]
        Gavin made changes -
        Workflow no-reopen-closed, patch-avail [ 12660777 ] patch-available, re-open possible [ 12749004 ]
        Sylvain Lebresne made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Hide
        Sylvain Lebresne added a comment -

        Committed, thanks

        Show
        Sylvain Lebresne added a comment - Committed, thanks
        Hide
        Jonathan Ellis added a comment -

        +1 on v2

        Show
        Jonathan Ellis added a comment - +1 on v2
        Sylvain Lebresne made changes -
        Attachment 4112_v2.txt [ 12521114 ]
        Hide
        Sylvain Lebresne added a comment -

        Agreed on the diagnostic, but the patch changes the behavior a bit in that it adds the new sstable to level 0 (which is probably ok in the sense of not introducing a bug, but is less efficient for no good reason that I can see). Attaching v2 that correct that.

        Show
        Sylvain Lebresne added a comment - Agreed on the diagnostic, but the patch changes the behavior a bit in that it adds the new sstable to level 0 (which is probably ok in the sense of not introducing a bug, but is less efficient for no good reason that I can see). Attaching v2 that correct that.
        Jonathan Ellis made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Labels compaction
        Assignee Jonathan Ellis [ jbellis ]
        Reviewer slebresne
        Fix Version/s 1.0.9 [ 12319856 ]
        Jonathan Ellis made changes -
        Attachment 4112.txt [ 12521096 ]
        Hide
        Jonathan Ellis added a comment -

        LCS incorrectly assumed that cleanup would always result in the same number of sstables as before, which is not the case (if no keys in an sstable belong post-cleanup, it will be left out entirely).

        fix attached.

        Show
        Jonathan Ellis added a comment - LCS incorrectly assumed that cleanup would always result in the same number of sstables as before, which is not the case (if no keys in an sstable belong post-cleanup, it will be left out entirely). fix attached.
        Shoaib made changes -
        Description We just recently started using version 1.0.9, previously we were using tiered compaction because of a bug in 1.0.8 (not letting us use leveled compaction) and now since moving to 1.0.9 we have started using leveled compaction.

        Trying to do a cleanup we are getting the following exception:

        root@hk1adsdbp001:~# nodetool -h localhost cleanup
        Error occured during cleanup
        java.util.concurrent.ExecutionException: java.util.NoSuchElementException
                at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
                at java.util.concurrent.FutureTask.get(FutureTask.java:111)
                at org.apache.cassandra.db.compaction.CompactionManager.performAllSSTableOperation(CompactionManager.java:204)
                at org.apache.cassandra.db.compaction.CompactionManager.performCleanup(CompactionManager.java:240)
                at org.apache.cassandra.db.ColumnFamilyStore.forceCleanup(ColumnFamilyStore.java:988)
                at org.apache.cassandra.service.StorageService.forceTableCleanup(StorageService.java:1639)
                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                at java.lang.reflect.Method.invoke(Method.java:616)
                at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
                at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
                at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
                at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
                at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251)
                at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)
                at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
                at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1450)
                at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90)
                at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1285)
                at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1383)
                at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807)
                at sun.reflect.GeneratedMethodAccessor31.invoke(Unknown Source)
                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                at java.lang.reflect.Method.invoke(Method.java:616)
                at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
                at sun.rmi.transport.Transport$1.run(Transport.java:177)
                at java.security.AccessController.doPrivileged(Native Method)
                at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
                at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
                at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
                at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
                at java.lang.Thread.run(Thread.java:636)
        Caused by: java.util.NoSuchElementException
                at java.util.ArrayList$Itr.next(ArrayList.java:757)
                at org.apache.cassandra.db.compaction.LeveledManifest.replace(LeveledManifest.java:196)
                at org.apache.cassandra.db.compaction.LeveledCompactionStrategy.handleNotification(LeveledCompactionStrategy.java:147)
                at org.apache.cassandra.db.DataTracker.notifySSTablesChanged(DataTracker.java:495)
                at org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:235)
                at org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:1010)
                at org.apache.cassandra.db.compaction.CompactionManager.doCleanupCompaction(CompactionManager.java:802)
                at org.apache.cassandra.db.compaction.CompactionManager.access$300(CompactionManager.java:64)
                at org.apache.cassandra.db.compaction.CompactionManager$5.perform(CompactionManager.java:244)
                at org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:183)
                at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
                at java.util.concurrent.FutureTask.run(FutureTask.java:166)

        cheers,
        Shoaib
        We just recently started using version 1.0.9, previously we were using tiered compaction because of a bug in 1.0.8 (not letting us use leveled compaction) and now since moving to 1.0.9 we have started using leveled compaction.

        Trying to do a cleanup we are getting the following exception:

        root@test:~# nodetool -h localhost cleanup
        Error occured during cleanup
        java.util.concurrent.ExecutionException: java.util.NoSuchElementException
                at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
                at java.util.concurrent.FutureTask.get(FutureTask.java:111)
                at org.apache.cassandra.db.compaction.CompactionManager.performAllSSTableOperation(CompactionManager.java:204)
                at org.apache.cassandra.db.compaction.CompactionManager.performCleanup(CompactionManager.java:240)
                at org.apache.cassandra.db.ColumnFamilyStore.forceCleanup(ColumnFamilyStore.java:988)
                at org.apache.cassandra.service.StorageService.forceTableCleanup(StorageService.java:1639)
                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                at java.lang.reflect.Method.invoke(Method.java:616)
                at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
                at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
                at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
                at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
                at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251)
                at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)
                at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
                at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1450)
                at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90)
                at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1285)
                at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1383)
                at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807)
                at sun.reflect.GeneratedMethodAccessor31.invoke(Unknown Source)
                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                at java.lang.reflect.Method.invoke(Method.java:616)
                at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
                at sun.rmi.transport.Transport$1.run(Transport.java:177)
                at java.security.AccessController.doPrivileged(Native Method)
                at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
                at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
                at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
                at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
                at java.lang.Thread.run(Thread.java:636)
        Caused by: java.util.NoSuchElementException
                at java.util.ArrayList$Itr.next(ArrayList.java:757)
                at org.apache.cassandra.db.compaction.LeveledManifest.replace(LeveledManifest.java:196)
                at org.apache.cassandra.db.compaction.LeveledCompactionStrategy.handleNotification(LeveledCompactionStrategy.java:147)
                at org.apache.cassandra.db.DataTracker.notifySSTablesChanged(DataTracker.java:495)
                at org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:235)
                at org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:1010)
                at org.apache.cassandra.db.compaction.CompactionManager.doCleanupCompaction(CompactionManager.java:802)
                at org.apache.cassandra.db.compaction.CompactionManager.access$300(CompactionManager.java:64)
                at org.apache.cassandra.db.compaction.CompactionManager$5.perform(CompactionManager.java:244)
                at org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:183)
                at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
                at java.util.concurrent.FutureTask.run(FutureTask.java:166)

        cheers,
        Shoaib
        Hide
        Shoaib added a comment - - edited

        Yes, we have assertions disabled.

        Show
        Shoaib added a comment - - edited Yes, we have assertions disabled.
        Jonathan Ellis made changes -
        Field Original Value New Value
        Priority Blocker [ 1 ] Major [ 3 ]
        Hide
        Jonathan Ellis added a comment -

        Are you running with assertions disabled?

        Show
        Jonathan Ellis added a comment - Are you running with assertions disabled?
        Shoaib created issue -

          People

          • Assignee:
            Jonathan Ellis
            Reporter:
            Shoaib
            Reviewer:
            Sylvain Lebresne
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development