Details
-
Improvement
-
Status: Open
-
Major
-
Resolution: Unresolved
-
4.15.0
-
None
-
None
Description
We can split SYSTEM.TASK into 2 tables:
- SYSTEM.TASK_QUEUE: used as an ephemeral task queue with potentially a smaller TTL which contains tasks that are new and yet to be picked up.
- SYSTEM.TASK_HISTORY: used to store tasks that have already been picked up and may or may not have succeeded, along with more data about the task (stack trace, job IDs, etc.). This table can have a longer TTL and can be used for auditing/debugging purposes.
The advantages of doing this are:
- Makes SYSTEM.TASK_QUEUE more lightweight.
- Less data scanned when picking up new tasks.
- Combined with a PK change, we can introduce a natural priority queue using the TASK_PRIORITY field.
- Different TTL value on the TASK_HISTORY table so we can maintain past tasks for a longer period of time.
- Potentially makes this more in-line with other use-cases that want to maintain a similar kind of task and task history table (for ex: View TTL's async delete functionality).
Attachments
Issue Links
- links to