Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: SystemML 1.2
    • Component/s: None
    • Labels:
      None

      Description

      It aims to implement the parameter exchange between ps and workers. We could leverage netty framework to implement our own Rpc framework. In general, the netty TransportClient and TransportServer provides the sending and receiving service for ps and workers. Extending the RpcHandler allows to invoke the corresponding ps method (i.e., push/pull method) by handling the different input Rpc call object. And then the SparkPsProxy wrapping TransportClient allows the workers to execute the push/pull call to server. At the same time, the ps netty server also provides the file repository service which allows the workers to download the partitioned training data, so that the workers could rebuild the matrix object with the transfered file instead of broadcasting all the files with spark which are not all necessary for each worker.

        Attachments

        1. systemml_rpc_class_diagram.png
          175 kB
          LI Guobao
        2. systemml_rpc_2_seq_diagram.png
          47 kB
          LI Guobao
        3. systemml_rpc_sequence_diagram.png
          86 kB
          LI Guobao

          Issue Links

            Activity

              People

              • Assignee:
                Guobao LI Guobao
                Reporter:
                Guobao LI Guobao
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: