Uploaded image for project: 'Singa'
  1. Singa
  2. SINGA-130

Implement a layer subclass for data prefetching

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    Description

      Data prefetching is important for training with GPU, because the IO would become the bottleneck when the computation is very fast.

      One idea is to create a general prefetch layer which embeds the application specific data loading layers.

      PrefetchLayer::ComptueFeature() {
        wait until the pretch thread finishes.
        swap the prefeth_data_ and data_ blobs.
        if (first time)
           load data into data_ blobs
        spawn a new thread to call functions from data loading layers for loading data into prefetch_data_.
      }
      

      If the prefetch layer has multiple loading layers and is connected to multiple destination layers, then different destination layer may want data loaded by different loading layers. This case should be handled properly.

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            ug93tad Anh Dinh
            wangwei.cs wangwei
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment