Falcon should also have pipeline entity.
- Pipeline entity,can comprise of the complete DAG for given set of process and feeds, within cluster or across clusters.
- How this helps,
- setting up a pipeline, should take care of relevant feeds and process
to be submitted.
- in case of cluster having issue, a particular pipeline can be processed
on another cluster
- to build monitoring system for a pipeline system
- run a particular pipeline for given time-window
- cases like, backlog and catch can be handled easily
- for Pipeline(A) to complete, we can suspend Pipeline(B),
if they have dependency