XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      In distributed spark backend, pass a given pre-trained model to the paramserv function may cause the data inconsistency. Because the pre-trained model would be cached in driver's memory. In this case, when kicking off the paramserv func, the workers firstly will try to read the data from HDFS where the dirty data in pre-trained model has not been persisted. This leads to a inconsistency. So the idea is to export the dirty data to HDFS before kicking off the remote workers.

        Attachments

          Activity

            People

            • Assignee:
              Guobao LI Guobao
              Reporter:
              Guobao LI Guobao
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: