Details
-
New Feature
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
-
None
Description
Design and implement a declarative query language over fine-grained lineage traces to enable forward and backward lineage queries, ordered lookups, change detection, compare and versioning (inspired by Dagger [1]), and in addition to that, enable fairness-specific analysis [2]. One step forward, combine with lineage cache (materialized) and re-execute from lineage to allow partial execution of phases, model debugging, and play-pause-based debugging.
[1] Samuel Madden, Mourad Ouzzani, Nan Tang, and Michael Stonebraker. 2020. Dagger: A Data (not code) Debugger. CIDR.
[2] Stefan Grafberger, Julia Stoyanovich, Sebastian Schelter. 2021. Lightweight Inspection of Data Preprocessing in Native Machine Learning Pipelines. CIDR.
Attachments
Issue Links
- links to