[SPARK-5348] s3a:// protocol and hadoop-aws dependency

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Not A Problem
• Affects Version/s: 1.2.0
• Assignee: Unassigned
• Reporter: Rafal Kwasny
• Created: 21/Jan/15 15:01
• Updated: 14/Jul/15 09:22
• Resolved: 23/Jan/15 10:44

Description

Hadoop 2.6.0+ includes new s3a library which is a lot better than the old one.
https://issues.apache.org/jira/browse/HADOOP-10400

Unfortunately this library is linking with amazon sdk which I think is not compatible with apache licensing. For this reason it is not a dependency for hadoop-client.

What I did is just add a profile for maven build, not sure if there is a better option to integrate this, we could also add a new hadoop-2.6 profile.

diff --git a/pom.xml b/pom.xml
index a9e7c9d..71ee776 100644
--- a/pom.xml
+++ b/pom.xml
@@ -1214,6 +1214,27 @@
     </profile>

     <profile>
+      <id>hadoop-aws</id>
+      <dependencies>
+        <dependency>
+          <groupId>org.apache.hadoop</groupId>
+          <artifactId>hadoop-aws</artifactId>
+          <version>${hadoop.version}</version>
+          <exclusions>
+            <exclusion>
+              <groupId>com.fasterxml.jackson.core</groupId>
+              <artifactId>jackson-annotations</artifactId>
+            </exclusion>
+            <exclusion>
+              <groupId>com.fasterxml.jackson.core</groupId>
+              <artifactId>jackson-databind</artifactId>
+            </exclusion>
+          </exclusions>
+        </dependency>
+      </dependencies>
+    </profile>
+
+    <profile>
       <id>java8-tests</id>
       <build>
         <plugins>

Issue Links

• duplicates: SPARK-7481 Add spark-hadoop-cloud module to pull in object store support