Uploaded image for project: 'Jackrabbit Oak'
  1. Jackrabbit Oak
  2. OAK-2683

the "hitting the observation queue limit" problem



    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • core, mongomk


      There are several tickets in this area:

      • OAK-2587: threading with observation being too eagar causing observation queue to grow
      • OAK-2669: avoiding diffing from mongo by using persistent cache instead.
      • OAK-2349: which might be a duplicate or at least similar to 2669..
      • OAK-2562: diffcache is inefficient

      Yet I think it makes sense to create this summarizing ticket, about describing again what happens when the observation queue hits the limit - and eventually about how this can be improved

      Consider the following scenario (also compare with OAK-2587 - but that one focused more on eagerness of threading):

      • rate of incoming commits is large and starts to generate many changes into the observation queues, hence those queue become somewhat filled/loaded
      • depending on the underlying nodestore used the calculation of diffs is more or less expensive - but at least for mongomk it is important that the diff can be served from the cache
        • in case of mongomk it can happen that diffs are no longer found in the cache and thus require a round-trip to mongo - which is magnitudes slower than via cache of course. this would result in the queue to start increasing even faster as dequeuing becomes slower now.
        • not sure about tarmk - I believe it should always be fast there
      • so based on the above, there can be a situation where the queue grows and hits the configured limit
      • if this limit is reached, the current mechanism is to collapse any subsequent change into one-big-marked-as-external-event change, lets call this a collapsed-change.
      • this collapsed-change now becomes part of the normal queue and eventually would 'walk down the queue' and be processed normally - hence opening a high chance that yet a new collapsed-change is created should the queue just hit the limit again. and this game can now be played for a while, resulting in the queue to contain many/mostly such collapse-changes.
      • there is now an additional assumption in that the diffing of such collapses is more expensive than normal diffing - plus it is almost guaranteed that the diff cannot for example be shared between observation listeners, since the exact 'collapse borders' depends on timing of each of the listeners' queues - ie the collapse diffs are unique thus not cachable..
      • so as a result: once you have those collapse-diffs you can almost not get rid of them - they are heavy to process - hence dequeuing is very slow
      • at the same time, there is always likely some commits happening in a typical system, eg with sling on top you have sling discovery which does heartbeats every now and then. So there's always new commits that add to the load.
      • this will hence create a situation where quite a small additional commit rate can keep all the queues filled - due to the fact that the queue is full with 'heavy collapse diffs' that have to be calculated for each and every listener (of which you could have eg 150-200) individually.

      So again, possible solutions for this:

      • OAK-2669: tune diffing via persistent cache
      • OAK-2587: have more threads to remain longer 'in the cache zone'
      • tune your input speed explicitly to avoid filling the observation queues (this would be specific to your use-case of course, but can be seen as explicitly throttling on the input side)
      • increase the relevant caches to the max
      • but I think we will come up with yet a broader improvement of this observation queue limit problem by either
        • doing flow control - eg via the commit rate limiter (also see OAK-1659)
        • moving out handling of observation changes to a messaging subsystem - be it to handle local events only (since handling external events makes the system problematic wrt scalability if not done right) - also see corresponding suggestion on dev list


        Issue Links



              Unassigned Unassigned
              stefanegli Stefan Egli
              0 Vote for this issue
              10 Start watching this issue