[HADOOP-8545] Filesystem Implementation for OpenStack Swift

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.2.0, 2.0.3-alpha
• Fix Version/s: 2.3.0
• Component/s: fs
• Labels: hadoop patch

Release Note

<!-- markdown -->
Added file system implementation for OpenStack Swift.
There are two implementation: block and native (similar to Amazon S3 integration).
Data locality issue solved by patch in Swift, commit procedure to OpenStack is in progress.

To use implementation add to core-site.xml following:

```xml
<property>
<name>fs.swift.impl</name>
<value>com.mirantis.fs.SwiftFileSystem</value>
</property>
<property>
<name>fs.swift.block.impl</name>
<value>com.mirantis.fs.block.SwiftBlockFileSystem</value>
        </property>
```

In MapReduce job specify following configs for OpenStack Keystone authentication:

```java
conf.set("swift.auth.url", "http://172.18.66.117:5000/v2.0/tokens");
conf.set("swift.tenant", "superuser");
conf.set("swift.username", "admin1");
conf.set("swift.password", "password");
conf.setInt("swift.http.port", 8080);
conf.setInt("swift.https.port", 443);
```

Additional information specified on github: https://github.com/DmitryMezhensky/Hadoop-and-Swift-integration

• Assignee: Dmitry Mezhensky
• Reporter: Tim Miller
• Created: 29/Jun/12 21:02
• Updated: 03/Sep/15 21:35
• Resolved: 27/Sep/13 11:30

Description

,Add a filesystem implementation for OpenStack Swift object store, similar to the one which exists today for S3.

Issue Links

• depends upon: HADOOP-9258 Add stricter tests to FileSystemContractTestBase
• depends upon: HADOOP-9118 FileSystemContractBaseTest test data for read/write isn't rigorous enough
• depends upon: HADOOP-9119 Add test to FileSystemContractBaseTest to verify integrity of overwritten files
• is depended upon by: BIGTOP-866 add OpenStack swift filesystem support for Hadoop as new RPM/deb
• is related to: HADOOP-10006 Compilation failure in trunk for o.a.h.fs.swift.util.JSONUtil
• is related to: HADOOP-9385 create hadoop-common-project/hadoop-filesystem-clients subprojects for blobstore & other clients
• relates to: HADOOP-9371 Define Semantics of FileSystem more rigorously
• relates to: HADOOP-9361 Strictly define the expected behavior of filesystem APIs and write tests to verify compliance