[SYSTEMDS-3087] Memory management and lazy evaluation in dynamic environments - ASF JIRA

XML

Word

Printable

JSON

Epic Link:
Multi-location & Multi-device Lineage Tracing and Reuse of Intermediates

Prefetch, broadcast, persist, related memory management, and other features in dynamic environments (spark, gpu)

1.	Prefetch instruction	Resolved	Arnab Phani
2.	Broadcast instruction	Closed	Arnab Phani
3.	Asynchronous Execution and Persist Spark Transformations	Resolved	Arnab Phani
4.	Future-based asynchronous execution of Spark actions	Resolved	Arnab Phani
5.	New operator linearization order to maximize inter-operator parallelism	Closed	Arnab Phani
6.	Lineage-based reuse of Spark actions	Closed	Arnab Phani
7.	Push down rmvar instructions for asynchronous instructions	Resolved	Arnab Phani
8.	Lineage-based reuse of asynchronous operators	Closed	Arnab Phani
9.	Persist and reuse Spark RDDs	Closed	Arnab Phani
10.	Compile-time chkpoint placement for shared Spark operators	Closed	Arnab Phani
11.	Lineage-based reuse of all RDDs	Open	Arnab Phani
12.	Refactor to add LOP rewrite step in compilation	Closed	Arnab Phani
13.	Multi-statementblock LOP rewrites	Open	Arnab Phani
14.	Analytical costing of Spark operators	Open	Arnab Phani
15.	Combined compile- and runtime selection of operators for reusing	Open	Arnab Phani
16.	Eviction policy for cached intermediates in Spark storage region	Open	Arnab Phani
17.	Compile-time chkpoint placement for Spark operators in loop	Closed	Arnab Phani
18.	Heuristic-based caching location (local/distributed)	Open	Arnab Phani
19.	Cost-based operator ordering	Reopened	Arnab Phani
20.	Partial reuse for Spark operators	Open	Arnab Phani
21.	Multi-level reuse of RDDs	Closed	Arnab Phani
22.	Clean up child RDDs, broadcasts from driver and executors	Open	Arnab Phani