[MAHOUT-1802] Capture attached checkpoints (if cached)

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.11.1
• Fix Version/s: 0.11.2
• Assignee: Andrew Palumbo
• Reporter: Andrew Palumbo
• Created: 08/Mar/16 22:45
• Updated: 11/Apr/16 08:41
• Resolved: 09/Mar/16 04:01

Description

Currently, the optimizer generates checkpoints and attaches them to actual logical elements of the DAG via CheckpointAction$cp.

the way it worsk today is as follows:

drmC = drmA+ drmB

val cp1 = drmC.checkpoint() // checkpoint
val cp2 = drmC.checkpoint() // cp2 == cp1

drmD = cp1 + drmE // cp1 + drmE

but, in:

drmD = drmC + drmE // computes drmA + drmB + drmC all over

drmC already has cp1 attached to it so we should assume the common computational path is the intent here regardless and should be used, instead of building plans that recompute it. That is,

drmD = drmC + drmE should imply cp1 + drmE as well even if checkpoint is not used explicitly.