Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Fix Version/s: 1.1.5
    • Component/s: Core
    • Labels:
      None

      Description

      [default@test] show schema;
      create column family Messages
      with column_type = 'Standard'
      and comparator = 'AsciiType'
      and default_validation_class = 'BytesType'
      and key_validation_class = 'AsciiType'
      and read_repair_chance = 0.1
      and dclocal_read_repair_chance = 0.0
      and gc_grace = 864000
      and min_compaction_threshold = 2
      and max_compaction_threshold = 4
      and replicate_on_write = true
      and compaction_strategy = 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'
      and caching = 'KEYS_ONLY'
      and compaction_strategy_options =

      {'sstable_size_in_mb' : '1024'}

      and compression_options =

      {'chunk_length_kb' : '64', 'sstable_compression' : 'org.apache.cassandra.io.compress.DeflateCompressor'}

      ;

      [default@test] update column family Messages with min_compaction_threshold = 4 and max_compaction_threshold = 32;
      a5b7544e-1ef5-3bfd-8770-c09594e37ec2
      Waiting for schema agreement...
      ... schemas agree across the cluster

      [default@test] show schema;
      create column family Messages
      with column_type = 'Standard'
      and comparator = 'AsciiType'
      and default_validation_class = 'BytesType'
      and key_validation_class = 'AsciiType'
      and read_repair_chance = 0.1
      and dclocal_read_repair_chance = 0.0
      and gc_grace = 864000
      and min_compaction_threshold = 2
      and max_compaction_threshold = 4
      and replicate_on_write = true
      and compaction_strategy = 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'
      and caching = 'KEYS_ONLY'
      and compaction_strategy_options =

      {'sstable_size_in_mb' : '1024'}

      and compression_options =

      {'chunk_length_kb' : '64', 'sstable_compression' : 'org.apache.cassandra.io.compress.DeflateCompressor'}

      ;

      1. CASSANDRA-4561-CS.patch
        1 kB
        Pavel Yaskevich
      2. CASSANDRA-4561.patch
        4 kB
        Pavel Yaskevich

        Activity

        Hide
        Zenek Kraweznik added a comment -

        in logfile I see only this:
        INFO [MigrationStage:1] 2012-08-21 11:27:55,560 ColumnFamilyStore.java (line 659) Enqueuing flush of Memtable-schema_columnfamilies@970905946(1266/1582 serialized/live bytes, 20 ops)
        INFO [FlushWriter:5] 2012-08-21 11:27:55,561 Memtable.java (line 264) Writing Memtable-schema_columnfamilies@970905946(1266/1582 serialized/live bytes, 20 ops)
        INFO [FlushWriter:5] 2012-08-21 11:27:55,587 Memtable.java (line 305) Completed flushing /var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-he-196-Data.db (1336 bytes) for commitlog position ReplayPosition(segmentId=4914817711083622, position=333055)

        Show
        Zenek Kraweznik added a comment - in logfile I see only this: INFO [MigrationStage:1] 2012-08-21 11:27:55,560 ColumnFamilyStore.java (line 659) Enqueuing flush of Memtable-schema_columnfamilies@970905946(1266/1582 serialized/live bytes, 20 ops) INFO [FlushWriter:5] 2012-08-21 11:27:55,561 Memtable.java (line 264) Writing Memtable-schema_columnfamilies@970905946(1266/1582 serialized/live bytes, 20 ops) INFO [FlushWriter:5] 2012-08-21 11:27:55,587 Memtable.java (line 305) Completed flushing /var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-he-196-Data.db (1336 bytes) for commitlog position ReplayPosition(segmentId=4914817711083622, position=333055)
        Hide
        Zenek Kraweznik added a comment -

        oh, and it obviously happens when I'm trying to change anything else, not only compaction treshhold, ex: sstable_size_in_mb

        Show
        Zenek Kraweznik added a comment - oh, and it obviously happens when I'm trying to change anything else, not only compaction treshhold, ex: sstable_size_in_mb
        Hide
        Suguru Namura added a comment -

        I saw a similar case. In my case, columns of schema_keyspaces and schema_columnfamilies in the system keyspace had future timestamp such as 2052-08-01 11:22:33. All operations of changing schemas were not affected. I do not know why timestamps had future values. I fixed it with rebuilding system keyspace.

        Show
        Suguru Namura added a comment - I saw a similar case. In my case, columns of schema_keyspaces and schema_columnfamilies in the system keyspace had future timestamp such as 2052-08-01 11:22:33. All operations of changing schemas were not affected. I do not know why timestamps had future values. I fixed it with rebuilding system keyspace.
        Hide
        Zenek Kraweznik added a comment - - edited

        Hmmm, it's possible. How did you rebuilt system keyspace?

        Show
        Zenek Kraweznik added a comment - - edited Hmmm, it's possible. How did you rebuilt system keyspace?
        Hide
        Pavel Yaskevich added a comment -

        Zenek, what version are you using? It's possible if you are on version less than 1.1.3 or if you have created your schema before 1.1.3 because of CASSANDRA-4432...

        Show
        Pavel Yaskevich added a comment - Zenek, what version are you using? It's possible if you are on version less than 1.1.3 or if you have created your schema before 1.1.3 because of CASSANDRA-4432 ...
        Hide
        Zenek Kraweznik added a comment -

        I'm using 1.1.4, but schema was created in 0.8.5

        Could you help me solve my problem step by step?

        Show
        Zenek Kraweznik added a comment - I'm using 1.1.4, but schema was created in 0.8.5 Could you help me solve my problem step by step?
        Hide
        Pavel Yaskevich added a comment -

        Did you upgrade directly to 1.1.4 or one of the previous 1.1 versions first? Please check timestamps in your schema_* ColumnFamilies and see if they have any future dates. The easiest way I see to fix this would be to re-create your schema if you have timestamp problems from CASSANDRA-4432

        Show
        Pavel Yaskevich added a comment - Did you upgrade directly to 1.1.4 or one of the previous 1.1 versions first? Please check timestamps in your schema_* ColumnFamilies and see if they have any future dates. The easiest way I see to fix this would be to re-create your schema if you have timestamp problems from CASSANDRA-4432
        Hide
        Zenek Kraweznik added a comment -

        I've upgraded cassandra from 8.7 to 1.0, then to 1.0.5 -> 1.0.6 -> 1.1.0 -> 1.1.1 -> 1.1.2 -> 1.1.0 -> 1.1.3 -> 1.1.4.

        Show
        Zenek Kraweznik added a comment - I've upgraded cassandra from 8.7 to 1.0, then to 1.0.5 -> 1.0.6 -> 1.1.0 -> 1.1.1 -> 1.1.2 -> 1.1.0 -> 1.1.3 -> 1.1.4.
        Hide
        Pavel Yaskevich added a comment -

        aha, so check system.schema_* CFs column's timestamp values I bet they are from the future. The simplest fix for you would be to delete system/schema_* SSTables and re-create a schema after restart.

        Show
        Pavel Yaskevich added a comment - aha, so check system.schema_* CFs column's timestamp values I bet they are from the future. The simplest fix for you would be to delete system/schema_* SSTables and re-create a schema after restart.
        Hide
        Jonathan Ellis added a comment -

        check system.schema_* CFs column's timestamp values I bet they are from the future

        Can we fix that on startup for other upgraders?

        Show
        Jonathan Ellis added a comment - check system.schema_* CFs column's timestamp values I bet they are from the future Can we fix that on startup for other upgraders?
        Hide
        Pavel Yaskevich added a comment -

        Yes, I was about too

        Show
        Pavel Yaskevich added a comment - Yes, I was about too
        Hide
        Arya Goudarzi added a comment -

        +1 I just ended up seeing this problem in our cluster which was upgraded from 1.1.2 to 1.1.3. Will probably have to find a workaround since I have to change schema now.

        Show
        Arya Goudarzi added a comment - +1 I just ended up seeing this problem in our cluster which was upgraded from 1.1.2 to 1.1.3. Will probably have to find a workaround since I have to change schema now.
        Hide
        Arya Goudarzi added a comment -

        I took my cluster down for 10 minutes. I took a snapshot of 'show schema' into a text file and removed system KS from it so I will only have my KS schema definition in it. Then I stop cassandra on all nodes. On each node, I removed system/schema_* folders from system's keyspace folder in cassandra data dir. I started all cassandra nodes. When I tried to reload the schema file using cli to recreate my CFs, I kept getting the message that CFs already exist. When I listed schema_columnfamilies in one of the node, I saws the same long timestamps like 2705487066780774 on columns of that CF. So, the procedure didn't quiet work out for me. What could have gone wrong here?

        Please advice.

        Show
        Arya Goudarzi added a comment - I took my cluster down for 10 minutes. I took a snapshot of 'show schema' into a text file and removed system KS from it so I will only have my KS schema definition in it. Then I stop cassandra on all nodes. On each node, I removed system/schema_* folders from system's keyspace folder in cassandra data dir. I started all cassandra nodes. When I tried to reload the schema file using cli to recreate my CFs, I kept getting the message that CFs already exist. When I listed schema_columnfamilies in one of the node, I saws the same long timestamps like 2705487066780774 on columns of that CF. So, the procedure didn't quiet work out for me. What could have gone wrong here? Please advice.
        Hide
        Pavel Yaskevich added a comment -

        What cassandra version are you running? Did you try to recreate a schema on all of the nodes or just one of them?

        Show
        Pavel Yaskevich added a comment - What cassandra version are you running? Did you try to recreate a schema on all of the nodes or just one of them?
        Hide
        Arya Goudarzi added a comment - - edited

        I am running 1.1.3 which was upgraded from 1.1.2. When nodes came up, I created the schemas in one node only.

        Show
        Arya Goudarzi added a comment - - edited I am running 1.1.3 which was upgraded from 1.1.2. When nodes came up, I created the schemas in one node only.
        Hide
        Arya Goudarzi added a comment -

        As part of my process, I also removed saved_caches/system-*.

        Show
        Arya Goudarzi added a comment - As part of my process, I also removed saved_caches/system-*.
        Hide
        Pavel Yaskevich added a comment -

        if you still have such timestamps and messages that CF already exists means that schema wasn't empty on restart and was reloaded from one of the nodes. I'm currently working on the patch which would fix timestamp situation on node's start up.

        Show
        Pavel Yaskevich added a comment - if you still have such timestamps and messages that CF already exists means that schema wasn't empty on restart and was reloaded from one of the nodes. I'm currently working on the patch which would fix timestamp situation on node's start up.
        Hide
        Arya Goudarzi added a comment -

        Do you have an ETA? I'll be happy to monkey apply it on my cluster and be the guinea pig for it.

        Show
        Arya Goudarzi added a comment - Do you have an ETA? I'll be happy to monkey apply it on my cluster and be the guinea pig for it.
        Hide
        Pavel Yaskevich added a comment -

        Not yet but I'm working on it and soon as it's ready I will attach it here.

        Show
        Pavel Yaskevich added a comment - Not yet but I'm working on it and soon as it's ready I will attach it here.
        Hide
        Jonathan Ellis added a comment -

        Don't you want to call cfs.truncate().get() to block for the truncate to finish? Rest LGTM.

        Show
        Jonathan Ellis added a comment - Don't you want to call cfs.truncate().get() to block for the truncate to finish? Rest LGTM.
        Hide
        Pavel Yaskevich added a comment -

        good point, I would add that and commit, thanks!

        Show
        Pavel Yaskevich added a comment - good point, I would add that and commit, thanks!
        Hide
        Pavel Yaskevich added a comment -

        Committed.

        Show
        Pavel Yaskevich added a comment - Committed.
        Hide
        Hudson added a comment -

        Integrated in Cassandra #2035 (See https://builds.apache.org/job/Cassandra/2035/)
        change SYSTEM_TABLE to SYSTEM_KS related to CASSANDRA-4561 merge (Revision cf23bd0a0fd192d991e971bcae94c2447126f873)

        Result = FAILURE
        xedin :
        Files :

        • src/java/org/apache/cassandra/db/DefsTable.java
        Show
        Hudson added a comment - Integrated in Cassandra #2035 (See https://builds.apache.org/job/Cassandra/2035/ ) change SYSTEM_TABLE to SYSTEM_KS related to CASSANDRA-4561 merge (Revision cf23bd0a0fd192d991e971bcae94c2447126f873) Result = FAILURE xedin : Files : src/java/org/apache/cassandra/db/DefsTable.java
        Hide
        Anton Winter added a comment -

        After applying this patch (1.1.5-tentative) I see the future timestamps being detected and fixed:

        INFO [main] 2012-09-07 02:24:00,090 DefsTable.java (line 202) Fixing timestamps of schema ColumnFamily schema_keyspaces...
        INFO [main] 2012-09-07 02:24:00,168 DefsTable.java (line 202) Fixing timestamps of schema ColumnFamily schema_columnfamilies...
        

        If I list schema_keyspaces I still see the old (future) timestamps.

        Given the timestamps weren't actually updated I did go back and found the following error shortly after the restarted node joined the ring (this may or not be related):

        ERROR [InternalResponseStage:1] 2012-09-07 02:24:16,555 AbstractCassandraDaemon.java (line 135) Exception in thread Thread[InternalResponseStage:1,5,main]
        java.lang.NullPointerException
                at org.apache.cassandra.db.DefsTable.mergeColumnFamilies(DefsTable.java:468)
                at org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:346)
                at org.apache.cassandra.db.DefsTable.mergeRemoteSchema(DefsTable.java:324)
                at org.apache.cassandra.service.MigrationManager$MigrationTask$1.response(MigrationManager.java:416)
                at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:45)
                at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
                at java.lang.Thread.run(Thread.java:679)
        

        I have however assumed that the patch automatically fixes this across all servers in a multi dc ring once this patch is applied. If there is a specific series of steps required to fix this, such as the full ring outage, deletion of system/schema_* etc can it please be documented here, the wiki or in the NEWS.txt file under upgrading notes?

        Show
        Anton Winter added a comment - After applying this patch (1.1.5-tentative) I see the future timestamps being detected and fixed: INFO [main] 2012-09-07 02:24:00,090 DefsTable.java (line 202) Fixing timestamps of schema ColumnFamily schema_keyspaces... INFO [main] 2012-09-07 02:24:00,168 DefsTable.java (line 202) Fixing timestamps of schema ColumnFamily schema_columnfamilies... If I list schema_keyspaces I still see the old (future) timestamps. Given the timestamps weren't actually updated I did go back and found the following error shortly after the restarted node joined the ring (this may or not be related): ERROR [InternalResponseStage:1] 2012-09-07 02:24:16,555 AbstractCassandraDaemon.java (line 135) Exception in thread Thread [InternalResponseStage:1,5,main] java.lang.NullPointerException at org.apache.cassandra.db.DefsTable.mergeColumnFamilies(DefsTable.java:468) at org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:346) at org.apache.cassandra.db.DefsTable.mergeRemoteSchema(DefsTable.java:324) at org.apache.cassandra.service.MigrationManager$MigrationTask$1.response(MigrationManager.java:416) at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:45) at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang. Thread .run( Thread .java:679) I have however assumed that the patch automatically fixes this across all servers in a multi dc ring once this patch is applied. If there is a specific series of steps required to fix this, such as the full ring outage, deletion of system/schema_* etc can it please be documented here, the wiki or in the NEWS.txt file under upgrading notes?
        Hide
        Pavel Yaskevich added a comment -

        Yes it does, but it assumes that you have schema in agreement across the nodes before it tries to fix timestamps. I can put that into NEWS.txt

        Show
        Pavel Yaskevich added a comment - Yes it does, but it assumes that you have schema in agreement across the nodes before it tries to fix timestamps. I can put that into NEWS.txt
        Hide
        Anton Winter added a comment -

        My schema is in agreement, but the timestamps aren't fixed, even though the log suggests otherwise (is that previously mentioned NPE related?).

        [default@unknown] describe cluster;
        Cluster Information:
           Snitch: org.apache.cassandra.locator.PropertyFileSnitch
           Partitioner: org.apache.cassandra.dht.RandomPartitioner
           Schema versions: 
                89b22434-5e34-381d-83d1-2a3cde1482fe: [x, x, x, x, x, x, x, x, x, x, x, x, x, x, x, x, x, x, x, x, x, x, x, x, x]
        
        [default@unknown]
        

        Schema updates continue to silently fail.

        Show
        Anton Winter added a comment - My schema is in agreement, but the timestamps aren't fixed, even though the log suggests otherwise (is that previously mentioned NPE related?). [ default @unknown] describe cluster; Cluster Information: Snitch: org.apache.cassandra.locator.PropertyFileSnitch Partitioner: org.apache.cassandra.dht.RandomPartitioner Schema versions: 89b22434-5e34-381d-83d1-2a3cde1482fe: [x, x, x, x, x, x, x, x, x, x, x, x, x, x, x, x, x, x, x, x, x, x, x, x, x] [ default @unknown] Schema updates continue to silently fail.
        Hide
        Pavel Yaskevich added a comment -

        Interesting, why does it try to merge remote schema if all of your nodes are in agreement, can you run in debug mode as attach the log? The thing is timestamps are fixed long before the storage server is initialized and the process involves truncate of the system.schema_* CFs so I don't see any other reason why your schema still has old timestamps except some other node sent it migration request...

        NPE that you experience is also interesting, cfMetaData() is always not null as CFs are copied from one map to another on KS initialization, and KS itself could not be null or you would have the assertion error instead of NPE.

        Show
        Pavel Yaskevich added a comment - Interesting, why does it try to merge remote schema if all of your nodes are in agreement, can you run in debug mode as attach the log? The thing is timestamps are fixed long before the storage server is initialized and the process involves truncate of the system.schema_* CFs so I don't see any other reason why your schema still has old timestamps except some other node sent it migration request... NPE that you experience is also interesting, cfMetaData() is always not null as CFs are copied from one map to another on KS initialization, and KS itself could not be null or you would have the assertion error instead of NPE.
        Hide
        Sylvain Lebresne added a comment -

        Maybe we should reopen this if that didn't fully fixed the problem? I'd better hold on a bit on 1.1.5 if this is not fixed yet as this seem to hit quite a few people.

        Show
        Sylvain Lebresne added a comment - Maybe we should reopen this if that didn't fully fixed the problem? I'd better hold on a bit on 1.1.5 if this is not fixed yet as this seem to hit quite a few people.
        Hide
        Pavel Yaskevich added a comment -

        We are not yet sure what is the problem so I think we should hold on reopening for a bit.

        Show
        Pavel Yaskevich added a comment - We are not yet sure what is the problem so I think we should hold on reopening for a bit.
        Hide
        Pavel Yaskevich added a comment -

        I just thought that a good addition to existing patch could be change in ColumnSerializer.deserialize to fix timestamps from the future, that way migrations from the remote locations would be deserialized with correct timestamp even if they were sent with the wrong one...

        Show
        Pavel Yaskevich added a comment - I just thought that a good addition to existing patch could be change in ColumnSerializer.deserialize to fix timestamps from the future, that way migrations from the remote locations would be deserialized with correct timestamp even if they were sent with the wrong one...
        Hide
        Sylvain Lebresne added a comment -

        Well, Anton still sees timestamp in the future and is still unable to do schema updates, it does sound a lot like there is a problem and it's related to this ticket. I just meant that I prefer keeping that in mind before releasing 1.1.5 into the wild too quickly.

        Show
        Sylvain Lebresne added a comment - Well, Anton still sees timestamp in the future and is still unable to do schema updates, it does sound a lot like there is a problem and it's related to this ticket. I just meant that I prefer keeping that in mind before releasing 1.1.5 into the wild too quickly.
        Hide
        Pavel Yaskevich added a comment -

        I understand, but we don't know if that is caused by the fix not working or by something else as exception indicates that something was send to the node, this could be a separate issue. I will work on the ColumnSerializer change I mentioned and we'll see if it helps.

        Show
        Pavel Yaskevich added a comment - I understand, but we don't know if that is caused by the fix not working or by something else as exception indicates that something was send to the node, this could be a separate issue. I will work on the ColumnSerializer change I mentioned and we'll see if it helps.
        Hide
        Anton Winter added a comment -

        Debug log sent to Pavel Yaskevich privately.

        Show
        Anton Winter added a comment - Debug log sent to Pavel Yaskevich privately.
        Hide
        Pavel Yaskevich added a comment -

        adding a patch that fixes column timestamp on deserialization. Anton, can you please apply and see if that fixes the problem?

        Show
        Pavel Yaskevich added a comment - adding a patch that fixes column timestamp on deserialization. Anton, can you please apply and see if that fixes the problem?
        Hide
        Pavel Yaskevich added a comment -

        As I see from the log - timestamps were actually fixed to "7 september 2012 cl 10:58 GMT" but then overriten by remote migration, latest patch should help with that.

        Show
        Pavel Yaskevich added a comment - As I see from the log - timestamps were actually fixed to "7 september 2012 cl 10:58 GMT" but then overriten by remote migration, latest patch should help with that.
        Hide
        Zenek Kraweznik added a comment -

        When cassandra 1.1.5 will be available?

        Show
        Zenek Kraweznik added a comment - When cassandra 1.1.5 will be available?
        Hide
        Anton Winter added a comment -

        I've patched & deployed to a couple of nodes where I now see the corrected timestamps. The NPE appears to also be gone.

        It will take me some time to deploy to the entire ring but it looks promising so far.

        Show
        Anton Winter added a comment - I've patched & deployed to a couple of nodes where I now see the corrected timestamps. The NPE appears to also be gone. It will take me some time to deploy to the entire ring but it looks promising so far.
        Hide
        Anton Winter added a comment -

        Ring upgraded with the second patch and I am now able to perform schema updates. Thanks!

        Show
        Anton Winter added a comment - Ring upgraded with the second patch and I am now able to perform schema updates. Thanks!
        Hide
        Jonathan Ellis added a comment -

        +1 lgtm

        Show
        Jonathan Ellis added a comment - +1 lgtm
        Hide
        Pavel Yaskevich added a comment -

        Committed to cassandra-1.1 branch.

        Show
        Pavel Yaskevich added a comment - Committed to cassandra-1.1 branch.
        Hide
        Hudson added a comment -

        Integrated in Cassandra #2084 (See https://builds.apache.org/job/Cassandra/2084/)
        change ColumnSerializer.deserialize to be able fix timestamps from future, related to CASSANDRA-4561 (Revision 2c69e2ea757be9492a095aa22b5d51234c4b4102)
        change ColumnSerializer.deserialize to be able fix timestamps from future, related to CASSANDRA-4561 (Revision 429fa7a80e22757a55c03e99c27c157824a666af)

        Result = ABORTED
        xedin :
        Files :

        • src/java/org/apache/cassandra/db/ColumnSerializer.java

        xedin :
        Files :

        • src/java/org/apache/cassandra/db/ColumnSerializer.java
        Show
        Hudson added a comment - Integrated in Cassandra #2084 (See https://builds.apache.org/job/Cassandra/2084/ ) change ColumnSerializer.deserialize to be able fix timestamps from future, related to CASSANDRA-4561 (Revision 2c69e2ea757be9492a095aa22b5d51234c4b4102) change ColumnSerializer.deserialize to be able fix timestamps from future, related to CASSANDRA-4561 (Revision 429fa7a80e22757a55c03e99c27c157824a666af) Result = ABORTED xedin : Files : src/java/org/apache/cassandra/db/ColumnSerializer.java xedin : Files : src/java/org/apache/cassandra/db/ColumnSerializer.java

          People

          • Assignee:
            Pavel Yaskevich
            Reporter:
            Zenek Kraweznik
            Reviewer:
            Jonathan Ellis
          • Votes:
            2 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development