[MESOS-1794] allow CURL download into a pipe

• Type: Improvement
• Status: Accepted
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.20.0
• Component/s: stout
• Labels: features patch performance
• Target Version/s: 1.0.0
• Assignee: Kamil Domański
• Reporter: Kamil Domański
• Shepherd: Benjamin Hindman
• Created: 14/Sep/14 17:46
• Updated: 26/Jun/15 15:17

Description

This divides Try<int> net::download(const std::string& url, const std::string& path) into three overloads:

1. Try<int> net::download(const std::string& url, FILE* file) downloads a file and sends it to FILE stream
2. Try<int> net::download(const std::string& url, int fdout) opens a file stream based on a file descriptor and calls 1)
3. Try<int> net::download(const std::string& url, const std::string& path) opens a file at path and passes the descriptor to 2), effectively working as before this change

This will allow to download into any file descriptor, such as a pipe, e.g. directly to extraction, as proposed by bernd-mesos in MESOS-1667.

Under similar rationale it's necessary to implement hadoop fs -cat into a file descriptor, but I'm unable to deploy HDFS for testing.