Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
-
None
-
None
Description
I have a problem where a Crunch client can potentially get killed through no fault of its own (e.g., an Airflow task failing a heartbeat check), which will kill the client, but leave the MR job itself still running. The problem is that the finalize() operation can delete the temp directory that contains the config info for Crunch, which will make any subsequent tasks fail to run to completion b/c their config info has been removed, or because some temp output in the directory that would have been useful can be deleted. I'd like to add an option to override the automatic delete of the temp directory during the finalize() operation for these situations.