[AURORA-1090] Optimize or remove shard uniqueness check from StorageBackfill

• Type: Task
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 0.8.0
• Component/s: Scheduler, Technical Debt
• Sprint: Twitter Aurora Q1'15 Sprint 2
• Story Points: 2
• Assignee: Bill Farner
• Reporter: Bill Farner
• Created: 02/Feb/15 23:52
• Updated: 04/May/15 14:02
• Resolved: 03/Feb/15 01:00

Description

We have noticed that during scheduler startup, the operation, there can be a significant amount of time spent between the following log lines:

Performing shard uniqueness sanity check.
storage state machine transition PREPARED -> READY

Looking at what happens in the scheduler between those points, the expensive operation seems to be guaranteeShardUniqueness.

This operation aims to validate the integrity of the storage, but its value is dubious. There are many other things that could be done to validate integrity, but they should probably not be done every time the scheduler loads its database.

If the operation is kept, it can be dramatically optimized. It currently performs an O(n^2) scan of tasks, and this could trivially be reduced to O(n).