[SYSTEMDS-1127] Synchronize creation of buffer pool on remote workers

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: SystemML 0.12
• Component/s: ParFor
• Epic Link: Enable parfor on remote Spark
• Assignee: Felix Schueler
• Reporter: Felix Schueler
• Created: 23/Nov/16 02:03
• Updated: 02/Mar/17 18:22
• Resolved: 29/Nov/16 19:01

Description

When executing a Spark parfor, the SparkParforWorker throws an exception which states that the localtmpdir could not be created. This is due to the fact that multiple executors are running multithreaded on the same worker.

The createDistributedUniqueID() method in the IDHander.java creates unique IDs only per pid and host, not per thread. This could potentially be solved by adding the threadID to the unique ID. The question is if every thread should have its own cache or if the logic should be changed so that the first creation will be successful and then the threads share one cache.

Issue Links

• relates to: SYSTEMDS-1130 Parfor remote_spark misses a proper cleanup