[SPARK-21077] Cannot access public files over S3 protocol

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Not A Problem
• Affects Version/s: 2.1.0
• Component/s: EC2

Environment

Spark 2.1.0 default installation. No existing hadoop, using the one distributed with Spark.
Added in $SPARK_HOME/jars:
hadoop-aws-2.7.3.jar and aws-java-sdk-1.7.4.jar

Added endpoint configuration in $SPARK_HOME/conf/core-site.xml (I want to access datasets hosted by organisation with CEPH; follows S3 protocols).

Ubuntu 14.04 x64.

• Assignee: Unassigned
• Reporter: Ciprian Tomoiaga
• Created: 13/Jun/17 12:43
• Updated: 12/Dec/22 18:10
• Resolved: 16/Jun/17 09:45

Description

I am trying to access a dataset with public (anonymous) credentials via the S3 (or S3a, s3n) protocol.

It fails with the error that no provider in chain can supply the credentials.
I asked our sysadmin to add some dummy credentials, and if I set them up (via link or config) then I have access.

I tried setting the config :

<property>
  <name>fs.s3a.credentials.provider</name>
  <value>org.apache.hadoop.fs.s3a.AnonymousAWSCredentialsProvider</value>
</property>

but it still doesn't work.

I suggested that it is a java-aws issue here, but they said it is not.

Any hints on how to use public S3 files from Spark ?