[CASSANDRA-12183] compaction_history system table does not capture all historical compaction sessions - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Bug
Status: Resolved
Priority: Normal
Resolution: Fixed
Fix Version/s: 3.0.30, 3.11.16, 4.0.11, 4.1.3, 5.0-alpha1, 5.0
Component/s: Legacy/Observability, Local/Compaction
Labels:
None

Severity:
Normal
Since Version:

3.0.0
Source Control Link:

https://github.com/apache/cassandra/commit/a00d1de4414bb5ab57a27cb463af0e33af6e36c8
Test and Documentation Plan:

Hide

ci

Show
ci

Description

It appears that some compaction sessions are not recorded in system.compaction_history table after the compaction session successfully finishes.

The following is an example (test by simply running cassandra-stress write n=100000000):

automaton@wdengdse50google-98425b985-3:~$ nodetool compactionstats
pending tasks: 46
                                     id   compaction type    keyspace       table    completed        total    unit   progress
   fa8a4f30-4884-11e6-b916-1dbd340a212f        Compaction   keyspace1   standard1   4233184044   4774209875   bytes     88.67%
   91e30d21-4887-11e6-b916-1dbd340a212f        Compaction   keyspace1   standard1    836983029    889773060   bytes     94.07%
Active compaction remaining time :   0h00m35s
automaton@wdengdse50google-98425b985-3:~$ nodetool compactionstats
pending tasks: 47
                                     id   compaction type    keyspace       table    completed        total    unit   progress
   fa8a4f30-4884-11e6-b916-1dbd340a212f        Compaction   keyspace1   standard1   4353251539   4774209875   bytes     91.18%
   28359094-4888-11e6-b916-1dbd340a212f        Compaction   keyspace1   standard1     49732274   4071652280   bytes      1.22%
Active compaction remaining time :   0h04m24s

At this point you know the previous compaction session 91e30d21-4887-11e6-b916-1dbd340a212f finished and confirmation can be found from debug.log

automaton@wdengdse50google-98425b985-3:~$ grep 91e30d21-4887-11e6-b916-1dbd340a212f /var/log/cassandra/debug.log
DEBUG [CompactionExecutor:4] 2016-07-12 23:22:58,674  CompactionTask.java:146 - Compacting (91e30d21-4887-11e6-b916-1dbd340a212f) [/var/lib/cassandra/data/keyspace1/standard1-9c02e9c1487c11e6b9161dbd340a212f/mb-290-big-Data.db:level=0, /var/lib/cassandra/data/keyspace1/standard1-9c02e9c1487c11e6b9161dbd340a212f/mb-279-big-Data.db:level=0, /var/lib/cassandra/data/keyspace1/standard1-9c02e9c1487c11e6b9161dbd340a212f/mb-281-big-Data.db:level=0, /var/lib/cassandra/data/keyspace1/standard1-9c02e9c1487c11e6b9161dbd340a212f/mb-280-big-Data.db:level=0, /var/lib/cassandra/data/keyspace1/standard1-9c02e9c1487c11e6b9161dbd340a212f/mb-284-big-Data.db:level=0, /var/lib/cassandra/data/keyspace1/standard1-9c02e9c1487c11e6b9161dbd340a212f/mb-283-big-Data.db:level=0, /var/lib/cassandra/data/keyspace1/standard1-9c02e9c1487c11e6b9161dbd340a212f/mb-287-big-Data.db:level=0, /var/lib/cassandra/data/keyspace1/standard1-9c02e9c1487c11e6b9161dbd340a212f/mb-292-big-Data.db:level=0, /var/lib/cassandra/data/keyspace1/standard1-9c02e9c1487c11e6b9161dbd340a212f/mb-286-big-Data.db:level=0, /var/lib/cassandra/data/keyspace1/standard1-9c02e9c1487c11e6b9161dbd340a212f/mb-289-big-Data.db:level=0, ]
DEBUG [CompactionExecutor:4] 2016-07-12 23:26:56,054  CompactionTask.java:217 - Compacted (91e30d21-4887-11e6-b916-1dbd340a212f) 10 sstables to [/var/lib/cassandra/data/keyspace1/standard1-9c02e9c1487c11e6b9161dbd340a212f/mb-293-big,] to level=0.  889,773,060 bytes to 890,473,350 (~100% of original) in 237,365ms = 3.577703MB/s.  0 total partitions merged to 3,871,921.  Partition merge counts were {1:3871921, }

However, if you query system.compaction_history table or run "nodetool compactionhistory | grep 91e30d21-4887-11e6-b916-1dbd340a212f" you will get nothing:

automaton@wdengdse50google-98425b985-3:~$ cqlsh -u cassandra
Password:
Connected to dse50 at 127.0.0.1:9042.
[cqlsh 5.0.1 | Cassandra 3.0.7.1158 | DSE 5.0.0 | CQL spec 3.4.0 | Native protocol v4]
Use HELP for help.
cassandra@cqlsh> select * from system.compaction_history where id=91e30d21-4887-11e6-b916-1dbd340a212f;

 id | bytes_in | bytes_out | columnfamily_name | compacted_at | keyspace_name | rows_merged
----+----------+-----------+-------------------+--------------+---------------+-------------

(0 rows)

automaton@wdengdse50google-98425b985-3:~$ nodetool flush system
automaton@wdengdse50google-98425b985-3:~$ nodetool compactionhistory | grep 91e30d21-4887-11e6-b916-1dbd340a212f