Uploaded image for project: 'Ignite'
  1. Ignite
  2. IGNITE-10644

CorruptedTreeException might occur after force node kill during transaction

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Duplicate
    • None
    • None
    • None
    • None
    • Docs Required

    Description

      Partition eviction process on the other hand:

       

      2018-12-10 20:59:24.426 [ERROR]sys-#204%_GRID%GridNodeName%[o.a.i.i.p.c.d.d.t.PartitionsEvictManager] Partition eviction failed, this can cause grid hang.
      org.h2.message.DbException: General error: "class org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException: Runtime failure on search row: Row@3580787f[ key: 4071535538120363041, val: X.common.dpl.model.backstream.DBackStreamMessage_DPL_PROXY [idHash=1961442513, hash=529139710, colocationKey=14465, entityType=I, lastChangeDate=1544464745135, errorMessage=No api [X.scripts.ucp.retail.propagate.publicapi.ClientPropagateService] services available for route: [*][*][kbt] (zone-node-module).IP: [*].
      List of services violations:
      NODE MODULE FILTER VIOLATIONS
      No services or violations were found for routing, partition_X_id=5, messageId=1211871172446406939, entityId=1211871174131851324, ownerId=ucp, responseDate=null, entityVersion=1, isDeleted=false, requestDate=Mon Dec 10 20:59:05 MSK 2018, id=4071535538120363041], ver: GridCacheVersion [topVer=155940834, order=1544596983071, nodeOrder=114] ][ I, null, 1211871172446406939, 1211871174131851324, null, 1, 2018-12-10 20:59:05.115, No api [X.scripts.ucp.retail.propagate.publicapi.ClientPropagateService] services available for route: [*][*][kbt] (zone-node-module).IP: [*].
      List of services violations:
      NODE MODULE FILTER VIOLATIONS
      No services or violations were found for routing, 4071535538120363041, FALSE, 5 ]" [50000-195]
      at org.h2.message.DbException.get(DbException.java:168)
      at org.h2.message.DbException.convert(DbException.java:295)
      at org.apache.ignite.internal.processors.query.h2.database.H2TreeIndex.removex(H2TreeIndex.java:293)
      at org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.remove(GridH2Table.java:515)
      at org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.remove(IgniteH2Indexing.java:738)
      at org.apache.ignite.internal.processors.query.GridQueryProcessor.remove(GridQueryProcessor.java:2487)
      at org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.remove(GridCacheQueryManager.java:433)
      at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.finishRemove(IgniteCacheOffheapManagerImpl.java:1465)
      at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.remove(IgniteCacheOffheapManagerImpl.java:1435)
      at org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.remove(GridCacheOffheapManager.java:1633)
      at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.remove(IgniteCacheOffheapManagerImpl.java:383)
      at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.removeValue(GridCacheMapEntry.java:3706)
      at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheEntry.clearInternal(GridDhtCacheEntry.java:652)
      at org.apache.ignite.internal.processors.cache.distributed.dht.topology.GridDhtLocalPartition.clearAll(GridDhtLocalPartition.java:1079)
      at org.apache.ignite.internal.processors.cache.distributed.dht.topology.GridDhtLocalPartition.tryClear(GridDhtLocalPartition.java:915)
      at org.apache.ignite.internal.processors.cache.distributed.dht.topology.PartitionsEvictManager$PartitionEvictionTask.run(PartitionsEvictManager.java:423)
      at org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6782)
      at org.apache.ignite.internal.processors.closure.GridClosureProcessor$1.body(GridClosureProcessor.java:827)
      at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      at java.lang.Thread.run(Thread.java:745)
      Caused by: org.h2.jdbc.JdbcSQLException: General error: "class org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException: Runtime failure on search row: Row@3580787f[ key: 4071535538120363041, val: X.common.X.model.backstream.DBackStreamMessage_X_PROXY [idHash=1961442513, hash=529139710, colocationKey=14465, entityType=I, lastChangeDate=1544464745135, errorMessage=No api [X.scripts.ucp.retail.propagate.publicapi.ClientPropagateService] services available for route: [*][*][kbt] (zone-node-module).IP: [*].
      List of services violations:
      NODE MODULE FILTER VIOLATIONS
      No services or violations were found for routing, partition_X_id=5, messageId=1211871172446406939, entityId=1211871174131851324, ownerId=ucp, responseDate=null, entityVersion=1, isDeleted=false, requestDate=Mon Dec 10 20:59:05 MSK 2018, id=4071535538120363041], ver: GridCacheVersion [topVer=155940834, order=1544596983071, nodeOrder=114] ][ I, null, 1211871172446406939, 1211871174131851324, null, 1, 2018-12-10 20:59:05.115, No api [X.scripts.ucp.retail.propagate.publicapi.ClientPropagateService] services available for route: [*][*][kbt] (zone-node-module).IP: [*].
      List of services violations:
      NODE MODULE FILTER VIOLATIONS
      No services or violations were found for routing, 4071535538120363041, FALSE, 5 ]" [50000-195]
      at org.h2.message.DbException.getJdbcSQLException(DbException.java:345)
      ... 22 common frames omitted
      Caused by: org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException: Runtime failure on search row: Row@3580787f[ key: 4071535538120363041, val: X.common.X.model.backstream.DBackStreamMessage_X_PROXY [idHash=1961442513, hash=529139710, colocationKey=14465, entityType=I, lastChangeDate=1544464745135, errorMessage=No api [X.scripts.ucp.retail.propagate.publicapi.ClientPropagateService] services available for route: [*][*][kbt] (zone-node-module).IP: [*].
      List of services violations:
      NODE MODULE FILTER VIOLATIONS
      No services or violations were found for routing, partition_X_id=5, messageId=1211871172446406939, entityId=1211871174131851324, ownerId=ucp, responseDate=null, entityVersion=1, isDeleted=false, requestDate=Mon Dec 10 20:59:05 MSK 2018, id=4071535538120363041], ver: GridCacheVersion [topVer=155940834, order=1544596983071, nodeOrder=114] ][ I, null, 1211871172446406939, 1211871174131851324, null, 1, 2018-12-10 20:59:05.115, No api [ru.sbt.integration.orchestration.scripts.ucp.retail.propagate.publicapi.ClientPropagateService] services available for route: [*][*][kbt] (zone-node-module).IP: [*].
      List of services violations:
      NODE MODULE FILTER VIOLATIONS
      No services or violations were found for routing, 4071535538120363041, FALSE, 5 ]
      at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doRemove(BPlusTree.java:1817)
      at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.removex(BPlusTree.java:1612)
      at org.apache.ignite.internal.processors.query.h2.database.H2TreeIndex.removex(H2TreeIndex.java:290)
      ... 19 common frames omitted
      Caused by: java.lang.IllegalStateException: Item not found: 9
      at org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.findIndirectItemIndex(AbstractDataPageIO.java:341)
      at org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.getDataOffset(AbstractDataPageIO.java:450)
      at org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.readPayload(AbstractDataPageIO.java:492)
      at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:150)
      at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102)
      at org.apache.ignite.internal.processors.query.h2.database.H2RowFactory.getRow(H2RowFactory.java:61)
      at org.apache.ignite.internal.processors.query.h2.database.H2Tree.createRowFromLink(H2Tree.java:152)
      at org.apache.ignite.internal.processors.query.h2.database.io.H2LeafIO.getLookupRow(H2LeafIO.java:67)
      at org.apache.ignite.internal.processors.query.h2.database.io.H2LeafIO.getLookupRow(H2LeafIO.java:33)
      at org.apache.ignite.internal.processors.query.h2.database.H2Tree.getRow(H2Tree.java:170)
      at org.apache.ignite.internal.processors.query.h2.database.H2Tree.getRow(H2Tree.java:47)
      at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.getRow(BPlusTree.java:4524)
      at org.apache.ignite.internal.processors.query.h2.database.H2Tree.compare(H2Tree.java:212)
      at org.apache.ignite.internal.processors.query.h2.database.H2Tree.compare(H2Tree.java:47)
      at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.compare(BPlusTree.java:4511)
      at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findInsertionPoint(BPlusTree.java:4431)
      at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.access$1300(BPlusTree.java:90)
      at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search.run0(BPlusTree.java:291)
      at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:4858)
      at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search.run(BPlusTree.java:271)
      at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:4843)
      at org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler.readPage(PageHandler.java:161)
      at org.apache.ignite.internal.processors.cache.persistence.DataStructure.read(DataStructure.java:332)
      at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.removeDown(BPlusTree.java:1852)
      at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.removeDown(BPlusTree.java:1871)
      at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.removeDown(BPlusTree.java:1871)
      at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.removeDown(BPlusTree.java:1871)
      at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doRemove(BPlusTree.java:1782)
      ... 21 common frames omitted

       

      And transaction on the other:

      2018-12-10 20:59:34.881 [ERROR]sys-stripe-33-#34%X_GRID%XGridNodeName%[org.apache.ignite.Ignite] Critical system error detected. Will be handled accordingly to configured handler [hnd=class o.a.i.failure.StopNodeOrHaltFailureHandler, failureCtx=FailureContext [type=CRITICAL_ERROR, err=class o.a.i.i.transactions.IgniteTxHeuristicCheckedException: Committing a transaction has produced runtime exception]]
      org.apache.ignite.internal.transactions.IgniteTxHeuristicCheckedException: Committing a transaction has produced runtime exception
      at org.apache.ignite.internal.processors.cache.transactions.IgniteTxAdapter.heuristicException(IgniteTxAdapter.java:720)
      at org.apache.ignite.internal.processors.cache.distributed.GridDistributedTxRemoteAdapter.commitIfLocked(GridDistributedTxRemoteAdapter.java:769)
      at org.apache.ignite.internal.processors.cache.distributed.GridDistributedTxRemoteAdapter.commitRemoteTx(GridDistributedTxRemoteAdapter.java:822)
      at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finish(IgniteTxHandler.java:1380)
      at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processDhtTxFinishRequest(IgniteTxHandler.java:1292)
      at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.access$600(IgniteTxHandler.java:101)
      at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$7.apply(IgniteTxHandler.java:217)
      at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$7.apply(IgniteTxHandler.java:215)
      at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1061)
      at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:586)
      at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:385)
      at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:311)
      at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
      at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:300)
      at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1556)
      at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1184)
      at org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:125)
      at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1091)
      at org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:496)
      at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
      at java.lang.Thread.run(Thread.java:745)
      Caused by: org.h2.message.DbException: General error: "class org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException: Runtime failure on row: Row@7651aeee[ key: 4071535538120364041, val: com.sbt.bm.ucp.common.X.model.backstream.DBackStreamMessage_X_PROXY [idHash=360032889, hash=-247223174, colocationKey=14465, entityType=I, lastChangeDate=1544464774859, errorMessage=null, partition_X_id=5, messageId=1211871297000585341, entityId=1211871298687462152, ownerId=ucp, responseDate=null, entityVersion=1, isDeleted=false, requestDate=Mon Dec 10 20:59:34 MSK 2018, id=4071535538120364041], ver: GridCacheVersion [topVer=155940834, order=1544597515244, nodeOrder=114] ][ I, null, 1211871297000585341, 1211871298687462152, null, 1, 2018-12-10 20:59:34.856, null, 4071535538120364041, FALSE, 5 ]" [50000-195]
      at org.h2.message.DbException.get(DbException.java:168)
      at org.h2.message.DbException.convert(DbException.java:295)
      at org.apache.ignite.internal.processors.query.h2.database.H2TreeIndex.putx(H2TreeIndex.java:251)
      at org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.update(GridH2Table.java:466)
      at org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.store(IgniteH2Indexing.java:709)
      at org.apache.ignite.internal.processors.query.GridQueryProcessor.store(GridQueryProcessor.java:1866)
      at org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.store(GridCacheQueryManager.java:403)
      at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.finishUpdate(IgniteCacheOffheapManagerImpl.java:1402)
      at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.invoke(IgniteCacheOffheapManagerImpl.java:1263)
      at org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.invoke(GridCacheOffheapManager.java:1625)
      at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.invoke(IgniteCacheOffheapManagerImpl.java:358)
      at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.storeValue(GridCacheMapEntry.java:3629)
      at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.storeValue(GridCacheMapEntry.java:3605)
      at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerSet(GridCacheMapEntry.java:1043)
      at org.apache.ignite.internal.processors.cache.distributed.GridDistributedTxRemoteAdapter.commitIfLocked(GridDistributedTxRemoteAdapter.java:617)
      ... 19 common frames omitted
      Caused by: org.h2.jdbc.JdbcSQLException: General error: "class org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException: Runtime failure on row: Row@7651aeee[ key: 4071535538120364041, val: com.sbt.bm.ucp.common.X.model.backstream.DBackStreamMessage_X_PROXY [idHash=360032889, hash=-247223174, colocationKey=14465, entityType=I, lastChangeDate=1544464774859, errorMessage=null, partition_X_id=5, messageId=1211871297000585341, entityId=1211871298687462152, ownerId=ucp, responseDate=null, entityVersion=1, isDeleted=false, requestDate=Mon Dec 10 20:59:34 MSK 2018, id=4071535538120364041], ver: GridCacheVersion [topVer=155940834, order=1544597515244, nodeOrder=114] ][ I, null, 1211871297000585341, 1211871298687462152, null, 1, 2018-12-10 20:59:34.856, null, 4071535538120364041, FALSE, 5 ]" [50000-195]
      at org.h2.message.DbException.getJdbcSQLException(DbException.java:345)
      ... 34 common frames omitted
      Caused by: org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException: Runtime failure on row: Row@7651aeee[ key: 4071535538120364041, val: com.sbt.bm.ucp.common.X.model.backstream.DBackStreamMessage_X_PROXY [idHash=360032889, hash=-247223174, colocationKey=14465, entityType=I, lastChangeDate=1544464774859, errorMessage=null, partition_X_id=5, messageId=1211871297000585341, entityId=1211871298687462152, ownerId=ucp, responseDate=null, entityVersion=1, isDeleted=false, requestDate=Mon Dec 10 20:59:34 MSK 2018, id=4071535538120364041], ver: GridCacheVersion [topVer=155940834, order=1544597515244, nodeOrder=114] ][ I, null, 1211871297000585341, 1211871298687462152, null, 1, 2018-12-10 20:59:34.856, null, 4071535538120364041, FALSE, 5 ]
      at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doPut(BPlusTree.java:2133)
      at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putx(BPlusTree.java:2080)
      at org.apache.ignite.internal.processors.query.h2.database.H2TreeIndex.putx(H2TreeIndex.java:248)
      ... 31 common frames omitted
      Caused by: java.lang.IllegalStateException: Item not found: 9
      at org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.findIndirectItemIndex(AbstractDataPageIO.java:341)
      at org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.getDataOffset(AbstractDataPageIO.java:450)
      at org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.readPayload(AbstractDataPageIO.java:492)
      at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:150)
      at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102)
      at org.apache.ignite.internal.processors.query.h2.database.H2RowFactory.getRow(H2RowFactory.java:61)
      at org.apache.ignite.internal.processors.query.h2.database.H2Tree.createRowFromLink(H2Tree.java:152)
      at org.apache.ignite.internal.processors.query.h2.database.io.H2LeafIO.getLookupRow(H2LeafIO.java:67)
      at org.apache.ignite.internal.processors.query.h2.database.io.H2LeafIO.getLookupRow(H2LeafIO.java:33)
      at org.apache.ignite.internal.processors.query.h2.database.H2Tree.getRow(H2Tree.java:170)
      at org.apache.ignite.internal.processors.query.h2.database.H2Tree.getRow(H2Tree.java:47)
      at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.getRow(BPlusTree.java:4524)
      at org.apache.ignite.internal.processors.query.h2.database.H2Tree.compare(H2Tree.java:212)
      at org.apache.ignite.internal.processors.query.h2.database.H2Tree.compare(H2Tree.java:47)
      at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.compare(BPlusTree.java:4511)
      at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findInsertionPoint(BPlusTree.java:4431)
      at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.access$1300(BPlusTree.java:90)
      at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search.run0(BPlusTree.java:291)
      at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:4858)
      at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search.run(BPlusTree.java:271)
      at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:4843)
      at org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler.readPage(PageHandler.java:161)
      at org.apache.ignite.internal.processors.cache.persistence.DataStructure.read(DataStructure.java:332)
      at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2347)
      at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2359)
      at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2359)
      at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2359)
      at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doPut(BPlusTree.java:2100)
      ... 33 common frames omitted

       

      Are failing to remove and put the same index entry on findIndirectItemIndex(). Because of node force (kill -9) stop index can be corrupted.

       

       

       

       

      Attachments

        Issue Links

          duplicates

          Bug - A problem which impairs or prevents the functions of the product. IGNITE-10873 CorruptedTreeException during simultaneous cache put operations

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved
          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. IGNITE-9303 PageSnapshot can contain wrong pageId tag when not dirty page is recycling

          • Major - Major loss of function.
          • Resolved
          relates to

          Bug - A problem which impairs or prevents the functions of the product. IGNITE-6580 Cluster can fail during concurrent re-balancing and cache destruction

          • Major - Major loss of function.
          • Open

          Activity

            People

              Unassigned Unassigned
              voropava Pavel Voronkin
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: