[SYSTEMDS-2420] Communication between ps and workers

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: SystemML 1.2
• Assignee: LI Guobao
• Reporter: LI Guobao
• Created: 24/Jun/18 18:59
• Updated: 30/Jul/18 07:20
• Resolved: 30/Jul/18 07:20

Description

It aims to implement the parameter exchange between ps and workers. We could leverage netty framework to implement our own Rpc framework. In general, the netty TransportClient and TransportServer provides the sending and receiving service for ps and workers. Extending the RpcHandler allows to invoke the corresponding ps method (i.e., push/pull method) by handling the different input Rpc call object. And then the SparkPsProxy wrapping TransportClient allows the workers to execute the push/pull call to server. At the same time, the ps netty server also provides the file repository service which allows the workers to download the partitioned training data, so that the workers could rebuild the matrix object with the transfered file instead of broadcasting all the files with spark which are not all necessary for each worker.

Issue Links

• duplicates: SYSTEMDS-2423 Implementation of spark ps
• Is contained by: SYSTEMDS-2087 Initial version of distributed spark backend