Major Description
This is the original description from Anirban:
There are quite a few people that are attempting to write plugins that download data from an origin server (or other
HTTP sites) and then optionally modify the contents like the transformation plugins allow for.
However, the downloading of the pages is cumbersome and a repeated task as this, should be wrapped in an API.
The input into the function would be
1. provide the headers passed into YTS as part of the request
- the header should obviously include the url to fetch
2. provide the moment when to wake up the plugin's continuation. Options include
0 - as soon as the request is made - no attempt is made to wait for the download to happen
1 - as soon as the headers are downloaded
2 - as soon as a portion of the message is downloaded
3 - after the full download is complete
3. ability to not cache the data (optional) (only used if one would like to override the data
4. ability to not run plugins (optional)
The output of the function would be
1. for each of the results
- error codes that define the download status of each of the requests
- IOBuffer ptr (or char*) to the results (the caller of the fxn has to release references to the IOBuffers or if char* is used free the chunk)
- time to download each object (not so critical)
Note: This fxn doesnt support streaming the answer back to the requesting client.
The fetches should be asynchronous and simultaneous.
A possible model would be to have the user call INKHttpFetchPages which then in turn calls INKHttpFetchPage per request.