Uploaded image for project: 'ManifoldCF'
  1. ManifoldCF
  2. CONNECTORS-1100

Improve ManifoldCF scalability by adopting dynamic reprioritization




      The single greatest impediment to MCF scalability at this point is the time that is required to reprioritize documents. This can be approved by adopting dynamic reprioritization.

      This process involves the following changes to the reprioritization process:

      • At the time of reprioritization, wipe the docpriority field to its null value. (Q: How long does this take?)
      • A background thread picks up documents on the queue with null priorities, and prioritizes them based on current conditions, doing 1000 or 10000 at a time
      • We need a new field to properly manage this; a simple boolean field which I'll call "needpriority", and an index based on it too.
      • ReprioritizationTracker currently has a notion of "reprioritization cycles". Cycles are interesting insofar as the minimum depth gets reset at the beginning of them. We will need to redefine what a cycle is, if we wish to maintain the current logic. Specifically, a "cycle" looks like this:
        (1) Remove all current document priorities (set to nullDocumentPriority), while also setting "needpriority" field to "true"
        (2) Reset priority calculation values (e.g. minimum depth)
        (3) Done with first part of cycle
        (4) Reprioritize via thread over time
        (5) When no more to do, done with second part of cycle
      • But, the whole "priorityset" logic is not needed anymore, if there's a persistent thread and a "needpriority" index. So we only will need the first part of the cycle, above, and no priorityset field is needed anymore at all. We just need a global write lock to coordinate the priority setting threads cross-cluster. These also have to synchronize, though, with any "reprioritization" cycles.
      • We can simply change the contract of IReprioritizationTracker to include entrance and exit for the reprioritization threads, and manage any locking internally. This would be PROVIDED the threads exercised good cleanup hygiene.




            kwright@metacarta.com Karl Wright
            kwright@metacarta.com Karl Wright
            0 Vote for this issue
            1 Start watching this issue