Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-12979

checkAvailableDiskSpace doesn't update expectedWriteSize when reducing thread scope

    Details

      Description

      If a compaction occurs that looks like it'll take up more space than remaining disk available, the compaction manager attempts to reduce the scope of the compaction by calling reduceScopeForLimitedSpace() repeatedly.

      Unfortunately, the while loop passes the estimatedWriteSize calculated from the original call to hasAvailableDiskSpace, so the comparisons that are done will always be against the size of the original compaction, rather than the reduced scope one.

      Full method below:

          protected void checkAvailableDiskSpace(long estimatedSSTables, long expectedWriteSize)
          {
              if(!cfs.isCompactionDiskSpaceCheckEnabled() && compactionType == OperationType.COMPACTION)
              {
                  logger.info("Compaction space check is disabled");
                  return;
              }
      
              while (!getDirectories().hasAvailableDiskSpace(estimatedSSTables, expectedWriteSize))
              {
                  if (!reduceScopeForLimitedSpace())
                      throw new RuntimeException(String.format("Not enough space for compaction, estimated sstables = %d, expected write size = %d", estimatedSSTables, expectedWriteSize));
      
            
              }
          }
      

      I'm proposing to recalculate the estimatedSSTables and expectedWriteSize after each iteration of reduceScopeForLimitedSpace.

      1. 3.0.patch
        6 kB
        Jon Haddad
      2. 2.2.patch
        6 kB
        Jon Haddad
      3. trunk-12979.patch
        6 kB
        Jon Haddad

        Activity

        Show
        rustyrazorblade Jon Haddad added a comment - 3.11 patch https://github.com/rustyrazorblade/cassandra/commit/44d60e68576945d882fd62aa3d34524b3ab70f0e.patch
        Hide
        rustyrazorblade Jon Haddad added a comment -

        Updated NEWS.txt and CHANGES.txt on all 3 patches. Changed the log error to warn. Not sure about the spacing nit though - there's 1 line above the single line comment, which is consistent with the other comments in the file. Maybe the patch had some weird formatting?

        Show
        rustyrazorblade Jon Haddad added a comment - Updated NEWS.txt and CHANGES.txt on all 3 patches. Changed the log error to warn. Not sure about the spacing nit though - there's 1 line above the single line comment, which is consistent with the other comments in the file. Maybe the patch had some weird formatting?
        Hide
        zznate Nate McCall added a comment -

        Jon Haddad Also, let's add a line to NEWS.txt since this is different (better, but different) runtime behavior.

        Show
        zznate Nate McCall added a comment - Jon Haddad Also, let's add a line to NEWS.txt since this is different (better, but different) runtime behavior.
        Hide
        zznate Nate McCall added a comment -

        Some nits:

        • (super minor) Looks like an extra line break before this comment // Note that we have removed files that are still marked as compacting. on the trunk patch
        • let's do WARN for the message about no space remaining so it's on the same output as the initial warning message

        Otherwise, LGTM (particularly since Sotirios Delimanolis is running it in production already

        Show
        zznate Nate McCall added a comment - Some nits: (super minor) Looks like an extra line break before this comment // Note that we have removed files that are still marked as compacting. on the trunk patch let's do WARN for the message about no space remaining so it's on the same output as the initial warning message Otherwise, LGTM (particularly since Sotirios Delimanolis is running it in production already
        Hide
        rustyrazorblade Jon Haddad added a comment -

        Ready for review.

        Show
        rustyrazorblade Jon Haddad added a comment - Ready for review.
        Hide
        rustyrazorblade Jon Haddad added a comment -

        Trunk functions correctly:

        DEBUG [MemtableFlushWriter:1] 2017-01-05 15:07:58,036 ColumnFamilyStore.java:1194 - Flushed to [BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-411-big-Data.db')] (1 sstables, 964.442KiB), biggest 964.442KiB, smallest 964.442KiB
        DEBUG [CompactionExecutor:2] 2017-01-05 15:07:59,051 CompactionTask.java:255 - Compacted (c4056ba0-d39b-11e6-b610-ef0dd3682585) 32 sstables to [/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-339-big,] to level=0.  27.586MiB to 28.058MiB (~101% of original) in 14,704ms.  Read Throughput = 1.876MiB/s, Write Throughput = 1.908MiB/s, Row Throughput = ~8,580/s.  128,704 total partitions merged to 128,704.  Partition merge counts were {1:128704, }
        DEBUG [CompactionExecutor:1] 2017-01-05 15:08:01,743 CompactionTask.java:255 - Compacted (c64bb220-d39b-11e6-b610-ef0dd3682585) 29 sstables to [/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-360-big,] to level=0.  63.751MiB to 64.450MiB (~101% of original) in 13,580ms.  Read Throughput = 4.694MiB/s, Write Throughput = 4.746MiB/s, Row Throughput = ~20,991/s.  293,883 total partitions merged to 293,883.  Partition merge counts were {1:293883, }
        WARN  [CompactionExecutor:2] 2017-01-05 15:08:01,743 CompactionTask.java:94 - insufficient space to compact all requested files. 27.529812MB required, BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-406-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-385-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-387-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-389-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-381-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-370-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-372-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-375-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-393-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-391-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-401-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-411-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-409-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-403-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-384-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-407-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-363-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-382-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-365-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-361-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-388-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-378-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-369-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-380-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-371-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-374-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-394-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-396-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-400-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-398-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-408-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-404-big-Data.db')
        WARN  [CompactionExecutor:2] 2017-01-05 15:08:01,744 CompactionTask.java:354 - Not enough space for compaction, 27.529812MB estimated.  Reducing scope.
        WARN  [CompactionExecutor:2] 2017-01-05 15:08:01,744 CompactionTask.java:94 - insufficient space to compact all requested files. 26.669067MB required, BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-385-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-387-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-389-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-381-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-370-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-372-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-375-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-393-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-391-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-401-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-411-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-409-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-403-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-384-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-407-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-363-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-382-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-365-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-361-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-388-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-378-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-369-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-380-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-371-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-374-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-394-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-396-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-400-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-398-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-408-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-404-big-Data.db')
        WARN  [CompactionExecutor:2] 2017-01-05 15:08:01,745 CompactionTask.java:354 - Not enough space for compaction, 26.669067MB estimated.  Reducing scope.
        WARN  [CompactionExecutor:2] 2017-01-05 15:08:01,745 CompactionTask.java:94 - insufficient space to compact all requested files. 25.808325MB required, BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-385-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-387-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-389-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-381-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-370-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-372-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-375-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-393-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-391-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-401-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-411-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-409-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-403-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-384-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-407-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-363-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-382-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-365-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-361-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-388-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-378-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-369-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-371-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-374-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-394-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-396-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-400-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-398-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-408-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-404-big-Data.db')
        WARN  [CompactionExecutor:2] 2017-01-05 15:08:01,745 CompactionTask.java:354 - Not enough space for compaction, 25.808325MB estimated.  Reducing scope.
        WARN  [CompactionExecutor:2] 2017-01-05 15:08:01,746 CompactionTask.java:94 - insufficient space to compact all requested files. 24.947624MB required, BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-385-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-387-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-389-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-381-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-370-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-372-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-375-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-393-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-391-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-401-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-411-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-409-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-403-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-384-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-407-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-363-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-382-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-365-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-361-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-388-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-378-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-369-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-371-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-374-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-394-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-396-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-400-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-408-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-404-big-Data.db')
        WARN  [CompactionExecutor:2] 2017-01-05 15:08:01,746 CompactionTask.java:354 - Not enough space for compaction, 24.947624MB estimated.  Reducing scope.
        WARN  [CompactionExecutor:2] 2017-01-05 15:08:01,746 CompactionTask.java:94 - insufficient space to compact all requested files. 24.086933MB required, BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-385-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-387-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-389-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-381-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-370-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-372-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-375-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-393-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-391-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-401-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-411-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-409-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-403-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-384-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-407-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-363-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-382-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-365-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-361-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-388-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-378-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-369-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-371-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-374-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-394-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-396-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-408-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-404-big-Data.db')
        WARN  [CompactionExecutor:2] 2017-01-05 15:08:01,747 CompactionTask.java:354 - Not enough space for compaction, 24.086933MB estimated.  Reducing scope.
        WARN  [CompactionExecutor:2] 2017-01-05 15:08:01,747 CompactionTask.java:94 - insufficient space to compact all requested files. 23.226252MB required, BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-385-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-387-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-389-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-381-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-370-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-372-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-375-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-393-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-391-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-401-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-411-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-409-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-403-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-384-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-407-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-363-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-382-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-365-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-361-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-388-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-378-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-369-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-371-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-374-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-394-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-408-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-404-big-Data.db')
        WARN  [CompactionExecutor:2] 2017-01-05 15:08:01,747 CompactionTask.java:354 - Not enough space for compaction, 23.226252MB estimated.  Reducing scope.
        WARN  [CompactionExecutor:2] 2017-01-05 15:08:01,748 CompactionTask.java:94 - insufficient space to compact all requested files. 22.36557MB required, BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-385-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-387-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-389-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-381-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-370-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-372-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-375-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-393-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-391-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-401-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-411-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-409-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-403-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-384-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-407-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-363-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-382-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-365-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-361-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-388-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-378-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-369-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-371-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-374-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-394-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-408-big-Data.db')
        WARN  [CompactionExecutor:2] 2017-01-05 15:08:01,748 CompactionTask.java:354 - Not enough space for compaction, 22.36557MB estimated.  Reducing scope.
        WARN  [CompactionExecutor:2] 2017-01-05 15:08:01,748 CompactionTask.java:94 - insufficient space to compact all requested files. 21.504894MB required, BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-385-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-387-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-389-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-381-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-370-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-372-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-375-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-393-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-401-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-411-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-409-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-403-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-384-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-407-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-363-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-382-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-365-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-361-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-388-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-378-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-369-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-371-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-374-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-394-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-408-big-Data.db')
        WARN  [CompactionExecutor:2] 2017-01-05 15:08:01,748 CompactionTask.java:354 - Not enough space for compaction, 21.504894MB estimated.  Reducing scope.
        
        Show
        rustyrazorblade Jon Haddad added a comment - Trunk functions correctly: DEBUG [MemtableFlushWriter:1] 2017-01-05 15:07:58,036 ColumnFamilyStore.java:1194 - Flushed to [BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-411-big-Data.db')] (1 sstables, 964.442KiB), biggest 964.442KiB, smallest 964.442KiB DEBUG [CompactionExecutor:2] 2017-01-05 15:07:59,051 CompactionTask.java:255 - Compacted (c4056ba0-d39b-11e6-b610-ef0dd3682585) 32 sstables to [/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-339-big,] to level=0. 27.586MiB to 28.058MiB (~101% of original) in 14,704ms. Read Throughput = 1.876MiB/s, Write Throughput = 1.908MiB/s, Row Throughput = ~8,580/s. 128,704 total partitions merged to 128,704. Partition merge counts were {1:128704, } DEBUG [CompactionExecutor:1] 2017-01-05 15:08:01,743 CompactionTask.java:255 - Compacted (c64bb220-d39b-11e6-b610-ef0dd3682585) 29 sstables to [/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-360-big,] to level=0. 63.751MiB to 64.450MiB (~101% of original) in 13,580ms. Read Throughput = 4.694MiB/s, Write Throughput = 4.746MiB/s, Row Throughput = ~20,991/s. 293,883 total partitions merged to 293,883. Partition merge counts were {1:293883, } WARN [CompactionExecutor:2] 2017-01-05 15:08:01,743 CompactionTask.java:94 - insufficient space to compact all requested files. 27.529812MB required, BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-406-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-385-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-387-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-389-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-381-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-370-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-372-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-375-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-393-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-391-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-401-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-411-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-409-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-403-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-384-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-407-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-363-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-382-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-365-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-361-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-388-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-378-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-369-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-380-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-371-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-374-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-394-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-396-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-400-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-398-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-408-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-404-big-Data.db') WARN [CompactionExecutor:2] 2017-01-05 15:08:01,744 CompactionTask.java:354 - Not enough space for compaction, 27.529812MB estimated. Reducing scope. WARN [CompactionExecutor:2] 2017-01-05 15:08:01,744 CompactionTask.java:94 - insufficient space to compact all requested files. 26.669067MB required, BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-385-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-387-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-389-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-381-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-370-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-372-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-375-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-393-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-391-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-401-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-411-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-409-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-403-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-384-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-407-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-363-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-382-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-365-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-361-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-388-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-378-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-369-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-380-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-371-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-374-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-394-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-396-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-400-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-398-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-408-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-404-big-Data.db') WARN [CompactionExecutor:2] 2017-01-05 15:08:01,745 CompactionTask.java:354 - Not enough space for compaction, 26.669067MB estimated. Reducing scope. WARN [CompactionExecutor:2] 2017-01-05 15:08:01,745 CompactionTask.java:94 - insufficient space to compact all requested files. 25.808325MB required, BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-385-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-387-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-389-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-381-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-370-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-372-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-375-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-393-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-391-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-401-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-411-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-409-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-403-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-384-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-407-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-363-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-382-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-365-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-361-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-388-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-378-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-369-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-371-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-374-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-394-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-396-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-400-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-398-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-408-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-404-big-Data.db') WARN [CompactionExecutor:2] 2017-01-05 15:08:01,745 CompactionTask.java:354 - Not enough space for compaction, 25.808325MB estimated. Reducing scope. WARN [CompactionExecutor:2] 2017-01-05 15:08:01,746 CompactionTask.java:94 - insufficient space to compact all requested files. 24.947624MB required, BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-385-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-387-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-389-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-381-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-370-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-372-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-375-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-393-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-391-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-401-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-411-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-409-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-403-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-384-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-407-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-363-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-382-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-365-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-361-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-388-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-378-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-369-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-371-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-374-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-394-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-396-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-400-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-408-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-404-big-Data.db') WARN [CompactionExecutor:2] 2017-01-05 15:08:01,746 CompactionTask.java:354 - Not enough space for compaction, 24.947624MB estimated. Reducing scope. WARN [CompactionExecutor:2] 2017-01-05 15:08:01,746 CompactionTask.java:94 - insufficient space to compact all requested files. 24.086933MB required, BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-385-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-387-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-389-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-381-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-370-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-372-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-375-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-393-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-391-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-401-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-411-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-409-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-403-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-384-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-407-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-363-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-382-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-365-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-361-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-388-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-378-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-369-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-371-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-374-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-394-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-396-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-408-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-404-big-Data.db') WARN [CompactionExecutor:2] 2017-01-05 15:08:01,747 CompactionTask.java:354 - Not enough space for compaction, 24.086933MB estimated. Reducing scope. WARN [CompactionExecutor:2] 2017-01-05 15:08:01,747 CompactionTask.java:94 - insufficient space to compact all requested files. 23.226252MB required, BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-385-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-387-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-389-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-381-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-370-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-372-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-375-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-393-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-391-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-401-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-411-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-409-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-403-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-384-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-407-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-363-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-382-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-365-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-361-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-388-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-378-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-369-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-371-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-374-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-394-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-408-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-404-big-Data.db') WARN [CompactionExecutor:2] 2017-01-05 15:08:01,747 CompactionTask.java:354 - Not enough space for compaction, 23.226252MB estimated. Reducing scope. WARN [CompactionExecutor:2] 2017-01-05 15:08:01,748 CompactionTask.java:94 - insufficient space to compact all requested files. 22.36557MB required, BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-385-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-387-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-389-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-381-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-370-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-372-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-375-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-393-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-391-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-401-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-411-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-409-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-403-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-384-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-407-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-363-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-382-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-365-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-361-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-388-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-378-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-369-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-371-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-374-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-394-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-408-big-Data.db') WARN [CompactionExecutor:2] 2017-01-05 15:08:01,748 CompactionTask.java:354 - Not enough space for compaction, 22.36557MB estimated. Reducing scope. WARN [CompactionExecutor:2] 2017-01-05 15:08:01,748 CompactionTask.java:94 - insufficient space to compact all requested files. 21.504894MB required, BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-385-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-387-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-389-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-381-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-370-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-372-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-375-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-393-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-401-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-411-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-409-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-403-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-384-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-407-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-363-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-382-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-365-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-361-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-388-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-378-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-369-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-371-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-374-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-394-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-771c8c10d39b11e6b610ef0dd3682585/mc-408-big-Data.db') WARN [CompactionExecutor:2] 2017-01-05 15:08:01,748 CompactionTask.java:354 - Not enough space for compaction, 21.504894MB estimated. Reducing scope.
        Hide
        rustyrazorblade Jon Haddad added a comment -

        Similar output for 3.0:

        DEBUG [SlabPoolCleaner] 2017-01-05 14:42:49,576 ColumnFamilyStore.java:854 - Enqueuing flush of standard1: 3393696 (32%) on-heap, 0 (0%) off-heap
        DEBUG [MemtableFlushWriter:3] 2017-01-05 14:42:49,577 Memtable.java:364 - Writing Memtable-standard1@1306908314(1.065MiB serialized bytes, 4002 ops, 32%/0% of on/off-heap limit)
        DEBUG [MemtableFlushWriter:3] 2017-01-05 14:42:49,628 Memtable.java:397 - Completed flushing /Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-325-big-Data.db (878.008KiB) for commitlog position ReplayPosition(segmentId=1483655869016, position=16570861)
        DEBUG [CompactionExecutor:4] 2017-01-05 14:42:52,205 CompactionTask.java:235 - Compacted (48323b00-d398-11e6-a3e2-7b0b736bccfe) 32 sstables to [/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-319-big,] to level=0.  28,817,720 bytes to 29,327,595 (~101% of original) in 4,091ms = 6.836708MB/s.  128,262 total partitions merged to 128,262.  Partition merge counts were {1:128262, }
        DEBUG [CompactionExecutor:3] 2017-01-05 14:42:52,264 CompactionTask.java:235 - Compacted (4347af80-d398-11e6-a3e2-7b0b736bccfe) 26 sstables to [/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-284-big,] to level=0.  44,583,630 bytes to 45,115,065 (~101% of original) in 12,399ms = 3.470044MB/s.  196,716 total partitions merged to 196,716.  Partition merge counts were {1:196716, }
        WARN  [CompactionExecutor:4] 2017-01-05 14:42:52,264 CompactionTask.java:98 - insufficient space to compact all requested files. 112.973495MB required, BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-284-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-318-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-319-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-320-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-323-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-293-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-325-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-324-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-322-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-300-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-321-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-258-big-Data.db')
        WARN  [CompactionExecutor:4] 2017-01-05 14:42:52,265 CompactionTask.java:319 - Not enough space for compaction, 112.973495MB estimated.  Reducing scope.
        DEBUG [CompactionExecutor:4] 2017-01-05 14:42:52,265 CompactionTask.java:158 - Compacting (4aabc681-d398-11e6-a3e2-7b0b736bccfe) [/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-318-big-Data.db:level=0, /Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-319-big-Data.db:level=0, /Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-320-big-Data.db:level=0, /Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-323-big-Data.db:level=0, /Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-293-big-Data.db:level=0, /Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-325-big-Data.db:level=0, /Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-324-big-Data.db:level=0, /Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-322-big-Data.db:level=0, /Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-300-big-Data.db:level=0, /Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-321-big-Data.db:level=0, /Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-258-big-Data.db:level=0, ]
        DEBUG [CompactionExecutor:4] 2017-01-05 14:42:56,395 CompactionTask.java:235 - Compacted (4aabc681-d398-11e6-a3e2-7b0b736bccfe) 11 sstables to [/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-326-big,] to level=0.  73,346,235 bytes to 73,709,675 (~100% of original) in 4,129ms = 17.024709MB/s.  320,961 total partitions merged to 320,961.  Partition merge counts were {1:320961, }
        WARN  [CompactionExecutor:4] 2017-01-05 14:42:56,395 CompactionTask.java:98 - insufficient space to compact all requested files. 227.1553MB required, BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-284-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-185-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-326-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-103-big-Data.db')
        WARN  [CompactionExecutor:4] 2017-01-05 14:42:56,395 CompactionTask.java:319 - Not enough space for compaction, 227.1553MB estimated.  Reducing scope.
        WARN  [CompactionExecutor:4] 2017-01-05 14:42:56,395 CompactionTask.java:98 - insufficient space to compact all requested files. 156.86029MB required, BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-284-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-185-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-103-big-Data.db')
        WARN  [CompactionExecutor:4] 2017-01-05 14:42:56,395 CompactionTask.java:319 - Not enough space for compaction, 156.86029MB estimated.  Reducing scope.
        WARN  [CompactionExecutor:4] 2017-01-05 14:42:56,395 CompactionTask.java:98 - insufficient space to compact all requested files. 97.03554MB required, BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-284-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-185-big-Data.db')
        WARN  [CompactionExecutor:4] 2017-01-05 14:42:56,396 CompactionTask.java:319 - Not enough space for compaction, 97.03554MB estimated.  Reducing scope.
        DEBUG [CompactionExecutor:4] 2017-01-05 14:42:56,396 CompactionTask.java:158 - Compacting (4d221db0-d398-11e6-a3e2-7b0b736bccfe) [/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-284-big-Data.db:level=0, ]
        DEBUG [CompactionExecutor:4] 2017-01-05 14:42:59,066 CompactionTask.java:235 - Compacted (4d221db0-d398-11e6-a3e2-7b0b736bccfe) 1 sstables to [/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-327-big,] to level=0.  45,115,065 bytes to 45,115,065 (~100% of original) in 2,670ms = 16.114262MB/s.  196,716 total partitions merged to 196,716.  Partition merge counts were {1:196716, }
        WARN  [CompactionExecutor:3] 2017-01-05 14:42:59,067 CompactionTask.java:98 - insufficient space to compact all requested files. 227.1553MB required, BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-327-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-185-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-326-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-103-big-Data.db')
        WARN  [CompactionExecutor:3] 2017-01-05 14:42:59,067 CompactionTask.java:319 - Not enough space for compaction, 227.1553MB estimated.  Reducing scope.
        WARN  [CompactionExecutor:3] 2017-01-05 14:42:59,068 CompactionTask.java:98 - insufficient space to compact all requested files. 156.86029MB required, BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-327-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-185-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-103-big-Data.db')
        WARN  [CompactionExecutor:3] 2017-01-05 14:42:59,068 CompactionTask.java:319 - Not enough space for compaction, 156.86029MB estimated.  Reducing scope.
        DEBUG [CompactionExecutor:3] 2017-01-05 14:42:59,068 CompactionTask.java:158 - Compacting (4eb9d4b0-d398-11e6-a3e2-7b0b736bccfe) [/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-327-big-Data.db:level=0, /Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-185-big-Data.db:level=0, ]
        DEBUG [SlabPoolCleaner] 2017-01-05 14:43:04,830 ColumnFamilyStore.java:1152 - Flushing largest CFS(Keyspace='keyspace1', ColumnFamily='standard1') to free up room. Used total: 0.33/0.00, live: 0.33/0.00, flushing: 0.00/0.00, this: 0.32/0.00
        DEBUG [SlabPoolCleaner] 2017-01-05 14:43:04,830 ColumnFamilyStore.java:854 - Enqueuing flush of standard1: 3378432 (32%) on-heap, 0 (0%) off-heap
        DEBUG [MemtableFlushWriter:4] 2017-01-05 14:43:04,831 Memtable.java:364 - Writing Memtable-standard1@279129348(1.060MiB serialized bytes, 3985 ops, 32%/0% of on/off-heap limit)
        DEBUG [MemtableFlushWriter:4] 2017-01-05 14:43:04,897 Memtable.java:397 - Completed flushing /Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-329-big-Data.db (877.544KiB) for commitlog position ReplayPosition(segmentId=1483655869016, position=17624611)
        Show
        rustyrazorblade Jon Haddad added a comment - Similar output for 3.0: DEBUG [SlabPoolCleaner] 2017-01-05 14:42:49,576 ColumnFamilyStore.java:854 - Enqueuing flush of standard1: 3393696 (32%) on-heap, 0 (0%) off-heap DEBUG [MemtableFlushWriter:3] 2017-01-05 14:42:49,577 Memtable.java:364 - Writing Memtable-standard1@1306908314(1.065MiB serialized bytes, 4002 ops, 32%/0% of on/off-heap limit) DEBUG [MemtableFlushWriter:3] 2017-01-05 14:42:49,628 Memtable.java:397 - Completed flushing /Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-325-big-Data.db (878.008KiB) for commitlog position ReplayPosition(segmentId=1483655869016, position=16570861) DEBUG [CompactionExecutor:4] 2017-01-05 14:42:52,205 CompactionTask.java:235 - Compacted (48323b00-d398-11e6-a3e2-7b0b736bccfe) 32 sstables to [/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-319-big,] to level=0. 28,817,720 bytes to 29,327,595 (~101% of original) in 4,091ms = 6.836708MB/s. 128,262 total partitions merged to 128,262. Partition merge counts were {1:128262, } DEBUG [CompactionExecutor:3] 2017-01-05 14:42:52,264 CompactionTask.java:235 - Compacted (4347af80-d398-11e6-a3e2-7b0b736bccfe) 26 sstables to [/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-284-big,] to level=0. 44,583,630 bytes to 45,115,065 (~101% of original) in 12,399ms = 3.470044MB/s. 196,716 total partitions merged to 196,716. Partition merge counts were {1:196716, } WARN [CompactionExecutor:4] 2017-01-05 14:42:52,264 CompactionTask.java:98 - insufficient space to compact all requested files. 112.973495MB required, BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-284-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-318-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-319-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-320-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-323-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-293-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-325-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-324-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-322-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-300-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-321-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-258-big-Data.db') WARN [CompactionExecutor:4] 2017-01-05 14:42:52,265 CompactionTask.java:319 - Not enough space for compaction, 112.973495MB estimated. Reducing scope. DEBUG [CompactionExecutor:4] 2017-01-05 14:42:52,265 CompactionTask.java:158 - Compacting (4aabc681-d398-11e6-a3e2-7b0b736bccfe) [/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-318-big-Data.db:level=0, /Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-319-big-Data.db:level=0, /Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-320-big-Data.db:level=0, /Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-323-big-Data.db:level=0, /Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-293-big-Data.db:level=0, /Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-325-big-Data.db:level=0, /Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-324-big-Data.db:level=0, /Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-322-big-Data.db:level=0, /Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-300-big-Data.db:level=0, /Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-321-big-Data.db:level=0, /Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-258-big-Data.db:level=0, ] DEBUG [CompactionExecutor:4] 2017-01-05 14:42:56,395 CompactionTask.java:235 - Compacted (4aabc681-d398-11e6-a3e2-7b0b736bccfe) 11 sstables to [/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-326-big,] to level=0. 73,346,235 bytes to 73,709,675 (~100% of original) in 4,129ms = 17.024709MB/s. 320,961 total partitions merged to 320,961. Partition merge counts were {1:320961, } WARN [CompactionExecutor:4] 2017-01-05 14:42:56,395 CompactionTask.java:98 - insufficient space to compact all requested files. 227.1553MB required, BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-284-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-185-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-326-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-103-big-Data.db') WARN [CompactionExecutor:4] 2017-01-05 14:42:56,395 CompactionTask.java:319 - Not enough space for compaction, 227.1553MB estimated. Reducing scope. WARN [CompactionExecutor:4] 2017-01-05 14:42:56,395 CompactionTask.java:98 - insufficient space to compact all requested files. 156.86029MB required, BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-284-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-185-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-103-big-Data.db') WARN [CompactionExecutor:4] 2017-01-05 14:42:56,395 CompactionTask.java:319 - Not enough space for compaction, 156.86029MB estimated. Reducing scope. WARN [CompactionExecutor:4] 2017-01-05 14:42:56,395 CompactionTask.java:98 - insufficient space to compact all requested files. 97.03554MB required, BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-284-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-185-big-Data.db') WARN [CompactionExecutor:4] 2017-01-05 14:42:56,396 CompactionTask.java:319 - Not enough space for compaction, 97.03554MB estimated. Reducing scope. DEBUG [CompactionExecutor:4] 2017-01-05 14:42:56,396 CompactionTask.java:158 - Compacting (4d221db0-d398-11e6-a3e2-7b0b736bccfe) [/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-284-big-Data.db:level=0, ] DEBUG [CompactionExecutor:4] 2017-01-05 14:42:59,066 CompactionTask.java:235 - Compacted (4d221db0-d398-11e6-a3e2-7b0b736bccfe) 1 sstables to [/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-327-big,] to level=0. 45,115,065 bytes to 45,115,065 (~100% of original) in 2,670ms = 16.114262MB/s. 196,716 total partitions merged to 196,716. Partition merge counts were {1:196716, } WARN [CompactionExecutor:3] 2017-01-05 14:42:59,067 CompactionTask.java:98 - insufficient space to compact all requested files. 227.1553MB required, BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-327-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-185-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-326-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-103-big-Data.db') WARN [CompactionExecutor:3] 2017-01-05 14:42:59,067 CompactionTask.java:319 - Not enough space for compaction, 227.1553MB estimated. Reducing scope. WARN [CompactionExecutor:3] 2017-01-05 14:42:59,068 CompactionTask.java:98 - insufficient space to compact all requested files. 156.86029MB required, BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-327-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-185-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-103-big-Data.db') WARN [CompactionExecutor:3] 2017-01-05 14:42:59,068 CompactionTask.java:319 - Not enough space for compaction, 156.86029MB estimated. Reducing scope. DEBUG [CompactionExecutor:3] 2017-01-05 14:42:59,068 CompactionTask.java:158 - Compacting (4eb9d4b0-d398-11e6-a3e2-7b0b736bccfe) [/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-327-big-Data.db:level=0, /Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-185-big-Data.db:level=0, ] DEBUG [SlabPoolCleaner] 2017-01-05 14:43:04,830 ColumnFamilyStore.java:1152 - Flushing largest CFS(Keyspace='keyspace1', ColumnFamily='standard1') to free up room. Used total: 0.33/0.00, live: 0.33/0.00, flushing: 0.00/0.00, this : 0.32/0.00 DEBUG [SlabPoolCleaner] 2017-01-05 14:43:04,830 ColumnFamilyStore.java:854 - Enqueuing flush of standard1: 3378432 (32%) on-heap, 0 (0%) off-heap DEBUG [MemtableFlushWriter:4] 2017-01-05 14:43:04,831 Memtable.java:364 - Writing Memtable-standard1@279129348(1.060MiB serialized bytes, 3985 ops, 32%/0% of on/off-heap limit) DEBUG [MemtableFlushWriter:4] 2017-01-05 14:43:04,897 Memtable.java:397 - Completed flushing /Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-329-big-Data.db (877.544KiB) for commitlog position ReplayPosition(segmentId=1483655869016, position=17624611)
        Hide
        rustyrazorblade Jon Haddad added a comment -

        Patch for 2.2

        Show
        rustyrazorblade Jon Haddad added a comment - Patch for 2.2
        Hide
        rustyrazorblade Jon Haddad added a comment -

        Posting patch files for 2.2, 3.0 and trunk tonight. I created a 500MB partition and put my data directory on it. I changed memtable_heap_space_in_mb to 10 and ran a cassandra-stress write workload with defaults. The writes proceed as normal till disk space is almost full, then correctly drop sstables out of the CompactionTask until there's enough space to run the compaction.

        INFO  04:16:52 Writing Memtable-standard1@453545543(994.383KiB serialized bytes, 22330 ops, 32%/0% of on/off-heap limit)
        INFO  04:16:52 Writing Memtable-standard1@1954443352(995.941KiB serialized bytes, 22365 ops, 32%/0% of on/off-heap limit)
        INFO  04:16:52 Writing Memtable-standard1@430003003(995.496KiB serialized bytes, 22355 ops, 32%/0% of on/off-heap limit)
        INFO  04:16:52 Writing Memtable-standard1@546357542(993.938KiB serialized bytes, 22320 ops, 32%/0% of on/off-heap limit)
        INFO  04:16:53 Writing Memtable-standard1@483193746(995.273KiB serialized bytes, 22350 ops, 32%/0% of on/off-heap limit)
        INFO  04:16:53 Writing Memtable-standard1@477708562(995.941KiB serialized bytes, 22365 ops, 32%/0% of on/off-heap limit)
        INFO  04:16:53 Writing Memtable-standard1@198137254(993.270KiB serialized bytes, 22305 ops, 32%/0% of on/off-heap limit)
        INFO  04:16:53 Writing Memtable-compactions_in_progress@2062736652(0.008KiB serialized bytes, 1 ops, 0%/0% of on/off-heap limit)
        INFO  04:16:53 Writing Memtable-compactions_in_progress@967481321(0.008KiB serialized bytes, 1 ops, 0%/0% of on/off-heap limit)
        WARN  04:16:53 insufficient space to compact all requested files. 121.46146MB required, BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-263-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-282-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-268-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-283-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-277-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-262-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-273-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-264-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-281-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-267-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-272-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-255-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-276-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-280-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-270-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-265-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-271-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-260-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-275-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-269-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-279-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-266-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-261-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-278-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-274-big-Data.db')
        WARN  04:16:53 Not enough space for compaction, 121.46146MB estimated.  Reducing scope.
        WARN  04:16:53 insufficient space to compact all requested files. 72.13821MB required, BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-263-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-282-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-268-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-283-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-277-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-262-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-273-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-264-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-281-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-267-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-272-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-276-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-280-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-270-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-265-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-271-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-260-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-275-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-269-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-279-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-266-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-261-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-278-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-274-big-Data.db')
        WARN  04:16:53 Not enough space for compaction, 72.13821MB estimated.  Reducing scope.
        INFO  04:16:53 Writing Memtable-compactions_in_progress@2049879518(0.482KiB serialized bytes, 28 ops, 0%/0% of on/off-heap limit)
        INFO  04:16:54 Writing Memtable-standard1@1622288656(992.602KiB serialized bytes, 22290 ops, 32%/0% of on/off-heap limit)
        INFO  04:16:54 Writing Memtable-standard1@1956423559(992.379KiB serialized bytes, 22285 ops, 32%/0% of on/off-heap limit)
        INFO  04:16:54 Writing Memtable-compactions_in_progress@1018281242(0.148KiB serialized bytes, 9 ops, 0%/0% of on/off-heap limit)
        
        Show
        rustyrazorblade Jon Haddad added a comment - Posting patch files for 2.2, 3.0 and trunk tonight. I created a 500MB partition and put my data directory on it. I changed memtable_heap_space_in_mb to 10 and ran a cassandra-stress write workload with defaults. The writes proceed as normal till disk space is almost full, then correctly drop sstables out of the CompactionTask until there's enough space to run the compaction. INFO 04:16:52 Writing Memtable-standard1@453545543(994.383KiB serialized bytes, 22330 ops, 32%/0% of on/off-heap limit) INFO 04:16:52 Writing Memtable-standard1@1954443352(995.941KiB serialized bytes, 22365 ops, 32%/0% of on/off-heap limit) INFO 04:16:52 Writing Memtable-standard1@430003003(995.496KiB serialized bytes, 22355 ops, 32%/0% of on/off-heap limit) INFO 04:16:52 Writing Memtable-standard1@546357542(993.938KiB serialized bytes, 22320 ops, 32%/0% of on/off-heap limit) INFO 04:16:53 Writing Memtable-standard1@483193746(995.273KiB serialized bytes, 22350 ops, 32%/0% of on/off-heap limit) INFO 04:16:53 Writing Memtable-standard1@477708562(995.941KiB serialized bytes, 22365 ops, 32%/0% of on/off-heap limit) INFO 04:16:53 Writing Memtable-standard1@198137254(993.270KiB serialized bytes, 22305 ops, 32%/0% of on/off-heap limit) INFO 04:16:53 Writing Memtable-compactions_in_progress@2062736652(0.008KiB serialized bytes, 1 ops, 0%/0% of on/off-heap limit) INFO 04:16:53 Writing Memtable-compactions_in_progress@967481321(0.008KiB serialized bytes, 1 ops, 0%/0% of on/off-heap limit) WARN 04:16:53 insufficient space to compact all requested files. 121.46146MB required, BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-263-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-282-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-268-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-283-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-277-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-262-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-273-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-264-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-281-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-267-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-272-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-255-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-276-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-280-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-270-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-265-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-271-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-260-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-275-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-269-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-279-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-266-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-261-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-278-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-274-big-Data.db') WARN 04:16:53 Not enough space for compaction, 121.46146MB estimated. Reducing scope. WARN 04:16:53 insufficient space to compact all requested files. 72.13821MB required, BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-263-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-282-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-268-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-283-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-277-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-262-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-273-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-264-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-281-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-267-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-272-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-276-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-280-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-270-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-265-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-271-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-260-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-275-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-269-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-279-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-266-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-261-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-278-big-Data.db'), BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-274-big-Data.db') WARN 04:16:53 Not enough space for compaction, 72.13821MB estimated. Reducing scope. INFO 04:16:53 Writing Memtable-compactions_in_progress@2049879518(0.482KiB serialized bytes, 28 ops, 0%/0% of on/off-heap limit) INFO 04:16:54 Writing Memtable-standard1@1622288656(992.602KiB serialized bytes, 22290 ops, 32%/0% of on/off-heap limit) INFO 04:16:54 Writing Memtable-standard1@1956423559(992.379KiB serialized bytes, 22285 ops, 32%/0% of on/off-heap limit) INFO 04:16:54 Writing Memtable-compactions_in_progress@1018281242(0.148KiB serialized bytes, 9 ops, 0%/0% of on/off-heap limit)
        Show
        rustyrazorblade Jon Haddad added a comment - https://github.com/apache/cassandra/pull/87
        Hide
        s_delima Sotirios Delimanolis added a comment - - edited

        +1

        We hit this issue recently. A huge set of sstables couldn't get compacted. We've been running a version of this patch (in 2.2 and just some added logging) in production for a couple of days and it unblocks these compactions.

        I suggest you open a separate ticket for the RuntimeException, though. Nothing is currently set up to handle it right now. It doesn't even get logged. I assume that's why this issue wasn't identified sooner.

        Show
        s_delima Sotirios Delimanolis added a comment - - edited +1 We hit this issue recently. A huge set of sstables couldn't get compacted. We've been running a version of this patch (in 2.2 and just some added logging) in production for a couple of days and it unblocks these compactions. I suggest you open a separate ticket for the RuntimeException , though. Nothing is currently set up to handle it right now. It doesn't even get logged. I assume that's why this issue wasn't identified sooner.
        Show
        rustyrazorblade Jon Haddad added a comment - See https://github.com/apache/cassandra/compare/trunk...rustyrazorblade:12979_fix_check_disk_space?expand=1
        Hide
        rustyrazorblade Jon Haddad added a comment -

        I've written a small patch against trunk to deal with this issue. A couple notes.

        • I moved the calculation for the space taken up by this compaction into checkAvailableDiskSpace()
        • I improved the logging in reduceScopeForLimitedSpace to print out the disk space that was estimated / required. This is what I had initially done that lead to finding this issue
        • checkAvailableDiskSpace() no longer accepts the estimates as args since they're recalculated internally
        • as far as I can tell there's no way to test this (it wasn't tested before). I'm creating follow up tickets for refactorings that will help test these sorts of edge cases
        • this bug has existed at least since 2.2, and probably before that. the patch is trivial enough to be backported.

        I consider this a critical fix for anyone who's running low on disk space, as compactions completely freeze up without this patch, so I'd be happy to supply patches for 2.1 -> 3.x.

        For initial review:

        https://github.com/apache/cassandra/compare/trunk...rustyrazorblade:12979_fix_check_disk_space?expand=1

        Show
        rustyrazorblade Jon Haddad added a comment - I've written a small patch against trunk to deal with this issue. A couple notes. I moved the calculation for the space taken up by this compaction into checkAvailableDiskSpace() I improved the logging in reduceScopeForLimitedSpace to print out the disk space that was estimated / required. This is what I had initially done that lead to finding this issue checkAvailableDiskSpace() no longer accepts the estimates as args since they're recalculated internally as far as I can tell there's no way to test this (it wasn't tested before). I'm creating follow up tickets for refactorings that will help test these sorts of edge cases this bug has existed at least since 2.2, and probably before that. the patch is trivial enough to be backported. I consider this a critical fix for anyone who's running low on disk space, as compactions completely freeze up without this patch, so I'd be happy to supply patches for 2.1 -> 3.x. For initial review: https://github.com/apache/cassandra/compare/trunk...rustyrazorblade:12979_fix_check_disk_space?expand=1
        Hide
        s_delima Sotirios Delimanolis added a comment -

        Slightly related: that RuntimeException seems to get ignored and eventually swallowed by the current executor.

        Show
        s_delima Sotirios Delimanolis added a comment - Slightly related: that RuntimeException seems to get ignored and eventually swallowed by the current executor.

          People

          • Assignee:
            rustyrazorblade Jon Haddad
            Reporter:
            rustyrazorblade Jon Haddad
            Reviewer:
            Nate McCall
          • Votes:
            2 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development