[SYSTEMDS-2083] Language and runtime for parameter servers

• Type: Epic
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Labels: gsoc2018
• Epic Name: Language and runtime for parameter servers
• Assignee: LI Guobao
• Reporter: Matthias Boehm
• Created: 27/Jan/18 03:12
• Updated: 02/Aug/18 13:15

Description

SystemML already provides a rich set of execution strategies ranging from local operations to large-scale computation on MapReduce or Spark. In this context, we support both data-parallel (multi-threaded or distributed operations) as well as task-parallel computation (multi-threaded or distributed parfor loops). This epic aims to complement the existing execution strategies by language and runtime primitives for parameter servers, i.e., model-parallel execution. We use the terminology of model-parallel execution with distributed data and distributed model to differentiate them from the existing data-parallel operations. Target applications are distributed deep learning and mini-batch algorithms in general. These new abstractions will help making SystemML a unified framework for small- and large-scale machine learning that supports all three major execution strategies in a single framework.

 

A major challenge is the integration of stateful parameter servers and their common push/pull primitives into an otherwise functional (and thus, stateless) language. We will approach this challenge via a new builtin function paramserv which internally maintains state but at the same time fits into the runtime framework of stateless operations.

Furthermore, we are interested in providing (1) different runtime backends (local and distributed), (2) different parameter server modes (synchronous, asynchronous, hogwild!, stale-synchronous), (3) different update frequencies (batch, multi-batch, epoch), as well as (4) different architectures for distributed data (1 parameter server, k workers) and distributed model (k1 parameter servers, k2 workers). 

 

Note for GSOC students: This is large project which will be broken down into sub projects, so everybody will be having their share of pie.

Prerequistes: Java, machine learning experience is a plus but not required.

Issue Links

• is duplicated by: SYSTEMDS-739 Explore model-parallel constructs in DML
• is related to: SYSTEMDS-2379 Performance sample operations

Sub-Tasks

• Preparation of dev environment: Closed
• API design of the paramserv function: Resolved
• Implementation of a script with paramserv func: Closed
• Implementation of language and compiler extension: Closed
• Implementation of language extension: Closed
• Hops, lops, instruction generation: Closed
• IPA integration: Closed
• Parfor integration: Closed
• Initial version of local backend: Resolved
• Local server: Closed
• Data partition: Closed
• Push/pull service: Closed
• Local workers: Closed
• Aggregation service: Closed
• Determine the level of parallelism: Closed
• Add the mnist-lenet test: Closed
• Add some optional argument: Closed
• Synchronization: Closed
• Checkpointing: Open
• Rewrite the ps using BlockingQueue: Closed
• Error handling: Closed
• Inline the agg service thread in ps: Closed
• Passing function pointers: Open
• Extend the update strategy ASP: Closed
• Extend update per EPOCH: Closed
• Extend scheme Disjoint_Round_Robin: Closed
• Extend scheme Disjoint_Random: Closed
• Extend scheme Overlap_Reshuffle: Closed
• Extend the update strategy SSP: Open
• Reorganise the test: Closed
• Preparation of baseline experiments: Resolved
• Add unit test for data partitioner: Closed
• Shutdown the thread pool of agg service: Closed
• Reorganise the api of data partitioner: Closed
• Concurrent problem about LibMatrixMult: Closed
• Forget to fetch the last iteration in local workers: Closed
• Extend statistics for paramserv func: Closed
• First evaluation: Closed
• Batch pre-fetching per workers: Open
• Rework the function block recompilation: Closed
• Use synchronized method instead of single thread pool in agg service: Closed
• Initial version of distributed spark backend: Resolved
• Determine the level of par: Closed
• Spark data partitioner: Resolved
• Setup and cleanup of remote workers: Resolved
• Implementation of remote worker: Resolved
• Implementation of spark ps: Resolved
• Communication between ps and workers: Resolved
• Task error and preemption handles: Open
• Second evaluation: Closed
• Add experiments varied on optimizers: Resolved
• Error handling and add statistics for spark backend: Resolved
• Add experiment on spark paramserv: Resolved
• Keep data consistency for a pre-trained model: Open
• Add java doc: Resolved
• Documentation of language extension: Resolved
• Submit final product: Resolved
• Second version of execution backend: Closed
• Integration test, implementation of samples and documentation: Open
• Performance features batching and parameter transfers: Open
• Extended Caffe2DML and Keras2DML script generators: Open