Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-8545

Filesystem Implementation for OpenStack Swift

    Details

    • Type: New Feature
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.2.0, 2.0.3-alpha
    • Fix Version/s: 2.3.0
    • Component/s: fs
    • Labels:
    • Target Version/s:
    • Release Note:
      Hide
      <!-- markdown -->
      Added file system implementation for OpenStack Swift.
      There are two implementation: block and native (similar to Amazon S3 integration).
      Data locality issue solved by patch in Swift, commit procedure to OpenStack is in progress.

      To use implementation add to core-site.xml following:

      ```xml
      <property>
      <name>fs.swift.impl</name>
      <value>com.mirantis.fs.SwiftFileSystem</value>
      </property>
      <property>
      <name>fs.swift.block.impl</name>
      <value>com.mirantis.fs.block.SwiftBlockFileSystem</value>
              </property>
      ```

      In MapReduce job specify following configs for OpenStack Keystone authentication:

      ```java
      conf.set("swift.auth.url", "http://172.18.66.117:5000/v2.0/tokens");
      conf.set("swift.tenant", "superuser");
      conf.set("swift.username", "admin1");
      conf.set("swift.password", "password");
      conf.setInt("swift.http.port", 8080);
      conf.setInt("swift.https.port", 443);
      ```

      Additional information specified on github: https://github.com/DmitryMezhensky/Hadoop-and-Swift-integration
      Show
      <!-- markdown --> Added file system implementation for OpenStack Swift. There are two implementation: block and native (similar to Amazon S3 integration). Data locality issue solved by patch in Swift, commit procedure to OpenStack is in progress. To use implementation add to core-site.xml following: ```xml <property> <name>fs.swift.impl</name> <value>com.mirantis.fs.SwiftFileSystem</value> </property> <property> <name>fs.swift.block.impl</name> <value>com.mirantis.fs.block.SwiftBlockFileSystem</value>         </property> ``` In MapReduce job specify following configs for OpenStack Keystone authentication: ```java conf.set("swift.auth.url", " http://172.18.66.117:5000/v2.0/tokens "); conf.set("swift.tenant", "superuser"); conf.set("swift.username", "admin1"); conf.set("swift.password", "password"); conf.setInt("swift.http.port", 8080); conf.setInt("swift.https.port", 443); ``` Additional information specified on github: https://github.com/DmitryMezhensky/Hadoop-and-Swift-integration

      Description

      ,Add a filesystem implementation for OpenStack Swift object store, similar to the one which exists today for S3.

        Attachments

        1. HADOOP-8545.patch
          118 kB
          Dmitry Mezhensky
        2. HADOOP-8545-1.patch
          55 kB
          David Dobbins
        3. HADOOP-8545-javaclouds-2.patch
          77 kB
          Steve Loughran
        4. HADOOP-8545.patch
          121 kB
          Dmitry Mezhensky
        5. HADOOP-8545-2.patch
          130 kB
          Dmitry Mezhensky
        6. HADOOP-8545-3.patch
          131 kB
          Dmitry Mezhensky
        7. HADOOP-8545-4.patch
          128 kB
          Dmitry Mezhensky
        8. HADOOP-8545-5.patch
          276 kB
          Dmitry Mezhensky
        9. HADOOP-8545-6.patch
          271 kB
          Dmitry Mezhensky
        10. HADOOP-8545-7.patch
          244 kB
          Dmitry Mezhensky
        11. HADOOP-8545-8.patch
          250 kB
          Dmitry Mezhensky
        12. HADOOP-8545-9.patch
          261 kB
          Dmitry Mezhensky
        13. HADOOP-8545-10.patch
          261 kB
          Dmitry Mezhensky
        14. HADOOP-8545-11.patch
          296 kB
          Dmitry Mezhensky
        15. HADOOP-8545-12.patch
          297 kB
          Dmitry Mezhensky
        16. HADOOP-8545-13.patch
          295 kB
          Dmitry Mezhensky
        17. HADOOP-8545-14.patch
          294 kB
          Dmitry Mezhensky
        18. HADOOP-8545-15.patch
          291 kB
          Dmitry Mezhensky
        19. HADOOP-8545-16.patch
          291 kB
          Dmitry Mezhensky
        20. HADOOP-8545-17.patch
          293 kB
          Dmitry Mezhensky
        21. HADOOP-8545-18.patch
          293 kB
          Dmitry Mezhensky
        22. HADOOP-8545-19.patch
          359 kB
          Steve Loughran
        23. HADOOP-8545-20.patch
          359 kB
          Steve Loughran
        24. HADOOP-8545-21.patch
          372 kB
          Steve Loughran
        25. HADOOP-8545-22.patch
          397 kB
          Steve Loughran
        26. HADOOP-8545-23.patch
          440 kB
          Steve Loughran
        27. HADOOP-8545-24.patch
          447 kB
          Steve Loughran
        28. HADOOP-8545-25.patch
          386 kB
          Dmitry Mezhensky
        29. HADOOP-8545-026.patch
          4.68 MB
          Steve Loughran
        30. HADOOP-8545-027.patch
          481 kB
          Steve Loughran
        31. HADOOP-8545-028.patch
          491 kB
          Steve Loughran
        32. HADOOP-8545-029.patch
          487 kB
          Steve Loughran
        33. HADOOP-8545.suresh.patch
          486 kB
          Suresh Srinivas
        34. HADOOP-8545-030.patch
          498 kB
          Steve Loughran
        35. HADOOP-8545-031.patch
          510 kB
          Steve Loughran
        36. HADOOP-8545-032.patch
          510 kB
          Steve Loughran
        37. HADOOP-8545-033.patch
          512 kB
          Steve Loughran
        38. HADOOP-8545-034.patch
          513 kB
          Steve Loughran
        39. HADOOP-8545-035.patch
          513 kB
          Suresh Srinivas
        40. HADOOP-8545-035.patch
          513 kB
          Steve Loughran
        41. HADOOP-8545-036.patch
          514 kB
          Steve Loughran
        42. HADOOP-8545-037.patch
          515 kB
          Steve Loughran

          Issue Links

            Activity

              People

              • Assignee:
                dmezhensky Dmitry Mezhensky
                Reporter:
                echohead Tim Miller
              • Votes:
                6 Vote for this issue
                Watchers:
                51 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: