[MESOS-1405] Mesos fetcher does not support S3(n)

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 0.18.2
• Fix Version/s: 0.19.0
• Assignee: Tom Arnfeld
• Reporter: Tom Arnfeld
• Created: 22/May/14 10:17
• Updated: 27/May/16 15:43
• Resolved: 23/May/14 17:17

Description

The HDFS client is able to support both S3 and S3N. Details for the difference between the two can be found here: http://wiki.apache.org/hadoop/AmazonS3.

Examples:

s3://bucket/path.tar.gz <- S3 Block Store
s3n://bucket/path.tar.gz <- S3 K/V Store

Either we can simply pass these URIs through to the HDFS client (hdfs.cpp) and let hadoop do the work, or we can integrate with S3 directly. The latter then requires we have a way of managing S3 credentials, whereas using the HDFS client will just pull credentials from HADOOP_HOME.

Issue Links

• is related to: MESOS-5472 Hadoop-free S3 fetcher