Details
-
Bug
-
Status: Resolved
-
Minor
-
Resolution: Invalid
-
2.9.1, 2.7.7
-
None
-
None
-
Hadoop 2.7.7 and 2.9.1
Java openJDK 8: 8u181-b13-0ubuntu0.18.04.1
Swift S3 api
Description
I'm trying to connect to an internal swift server using S3A capability of Hadoop. This server works using python boto, it contains one bucket name test and in this bucket there's one file test.file
So far it's been impossible for me to reach the server properly, each time I try it either ignore fs.s3a.endpoint or treat it incorrectly.:
site-core.xml:
<configuration>
<property>
<name>fs.s3a.access.key</name>
<description>mykey</description>
</property>
<property>
<name>fs.s3a.secret.key</name>
<description>mysecret</description>
</property>
<property>
<name>fs.s3a.endpoint</name>
<description>my.endpoint.fr:8080</description>
</property>
<property>
<name>fs.s3a.connection.ssl.enabled</name>
<value>true</value>
</property>
<property>
<name>fs.s3a.path.style.access</name>
<description>true</description>
</property>
<property>
<name>fs.s3a.impl</name>
<description>org.apache.hadoop.fs.s3a.S3AFileSystem</description>
</property>
</configuration>
To debug this issue, I've try using this tool: cloudstore which helps debug Hadoop fs.
When trying to list my bucket test :
s3a://test/
I can see that it's connecting to :
https://test.s3.amazonaws.com/
Meaning that it happen test to the original s3 server and ignoring my previous settings.
When trying to list my bucket test using this url :
s3a://my.endpoint.fr:8080/test/
I can see that it's connecting to :
Meaning that it ignore the port I set up in fs.s3a.endpoint configuration which of course doesn't work because my server is listening to port 8080.
I've tried with fs.s3a.path.style.access to false it's pretty much the same.
I'm sorry if it's not a bug but any help or consideration would be very appreciate.