Uploaded image for project: 'Libcloud'
  1. Libcloud
  2. LIBCLOUD-101

Add support for uploading storage objects using an iterator even though driver might not support chunked encoding

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Minor
    • Resolution: Fixed
    • 0.5.0
    • 0.6.1
    • Storage

    Description

      Currently there are two options for uploading objects to any given storage; StorageDriver.upload_object() and StorageDriver.upload_object_via_stream(). The former strictly requires the object to be written to disk prior to upload since it takes the file path as argument. The latter does support upload given an iterator, but does not work for Amazon S3 since they do not support chunked encoding. After discussing this with Tomaž he and I came to the conclusion that a third method is in order which will take an iterator as argument while not requiring the provider to support chunked encoding.

      The solution is to support an iterator as argument and generate a request by iterating through it directly. However, these will require the object to be kept in memory and depending on the size and available resources it might lead to exhaustion. So it should be noted in the methods documentation that the implementation is responsible for preventing such outcomes; which can be done by preventing the allowed upload size for example.

      Attachments

        Activity

          People

            kami Tomaz Muraus
            birk Birk Nilson
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - 48h
                48h
                Remaining:
                Remaining Estimate - 48h
                48h
                Logged:
                Time Spent - Not Specified
                Not Specified