Uploaded image for project: 'Ignite'
  1. Ignite
  2. IGNITE-14244

Performance degradation of entire ignite because of indexed persisted cache

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Blocker
    • Resolution: Invalid
    • 2.9.1
    • None
    • cache, data structures
    • 2 nodes

      56 cpu per node

      600gb ram per node

      ssd(wal)/hdd(storage)

    • Docs Required, Release Notes Required

    Description

      We have multiple not persisted caches and one with persistenc on (needed for events registration)
       
      Ignite nodes starting up inside Business module (share single jvm)
      All caches are transactional.
       
      Persisted cache key, value:
       
      key:
      long recordId; (unique)
      Int type;.       \ affinity
      int entityid;  /
       
       
      Value:
      long evtDate;
      String system;
      ... and other not necessary fields
       
      Indexes:
      1) EntityId, entityType, system
      2) evtDate, system
       
      Case:
       
      1) Put value on not persisted cache,
      2) Put new value on persisted cache (register event of creation)
      3) remove value on not persisted cache
      4) Put new value on persisted cache (register event of delete)
       
      On start I start to send 4000 tps of that case.
      so 4k put/remove from not persisted cache and 8k values added to persisted cache per second.
       
      It running okay until 10 mil of entries added in persisted cache.
      After that performance start to slowdown. Transaction timeouts begin showing up (and not only on persisted cache but on NOT persisted caches TOO!)
      And throughput lowers to 3k tps of 4k tps input (business threads reject 1k tps cause internal threads cant process more because of ignite)
       
      On 20 mil - handled tps dropping to 100 tps. And so on.
      All business threads waiting on commit (and then after ignite response - failing with timeout)
       
      1) Is that because of index update?
      2) All requests to not persisted caches slowdown too accordingly (waiting on commit too, even tho its simple get)
       
      CPU utilization only around 30%. Memory not swapping, gc is good, cant say about IO but it looks good (any tip how to check that out will be nice)
       
      I tried to remove all indexes and it slowdowns too overtime.
       
      I attached thread dumps and ignite log of 1 node.
      PropagationTaskExecutorThread-poo[]-thread[%d] - business threads that struggling with ignite call, all sit on LockSupport.park.
       
      Would be nice to see any help or explanation why persisted cache is so slow 

      Attachments

        Activity

          People

            Unassigned Unassigned
            FDoKE Vyacheslav Puzakov
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: