Hadoop Common
  1. Hadoop Common
  2. HADOOP-8545

Filesystem Implementation for OpenStack Swift

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.2.0, 2.0.3-alpha
    • Fix Version/s: 2.3.0
    • Component/s: fs
    • Labels:
    • Release Note:
      Hide
      Added file system implementation for OpenStack Swift.
      There are two implementation: block and native (similar to Amazon S3 integration).
      Data locality issue solved by patch in Swift, commit procedure to OpenStack is in progress.

      To use implementation add to core-site.xml following:
      ...
      <property>
      <name>fs.swift.impl</name>
      <value>com.mirantis.fs.SwiftFileSystem</value>
      </property>
      <property>
      <name>fs.swift.block.impl</name>
      <value>com.mirantis.fs.block.SwiftBlockFileSystem</value>
              </property>
      ...

      In MapReduce job specify following configs for OpenStack Keystone authentication:
      conf.set("swift.auth.url", "http://172.18.66.117:5000/v2.0/tokens");
      conf.set("swift.tenant", "superuser");
      conf.set("swift.username", "admin1");
      conf.set("swift.password", "password");
      conf.setInt("swift.http.port", 8080);
      conf.setInt("swift.https.port", 443);

      Additional information specified on github: https://github.com/DmitryMezhensky/Hadoop-and-Swift-integration
      Show
      Added file system implementation for OpenStack Swift. There are two implementation: block and native (similar to Amazon S3 integration). Data locality issue solved by patch in Swift, commit procedure to OpenStack is in progress. To use implementation add to core-site.xml following: ... <property> <name>fs.swift.impl</name> <value>com.mirantis.fs.SwiftFileSystem</value> </property> <property> <name>fs.swift.block.impl</name> <value>com.mirantis.fs.block.SwiftBlockFileSystem</value>         </property> ... In MapReduce job specify following configs for OpenStack Keystone authentication: conf.set("swift.auth.url", " http://172.18.66.117:5000/v2.0/tokens "); conf.set("swift.tenant", "superuser"); conf.set("swift.username", "admin1"); conf.set("swift.password", "password"); conf.setInt("swift.http.port", 8080); conf.setInt("swift.https.port", 443); Additional information specified on github: https://github.com/DmitryMezhensky/Hadoop-and-Swift-integration
    • Target Version/s:

      Description

      ,Add a filesystem implementation for OpenStack Swift object store, similar to the one which exists today for S3.

      1. HADOOP-8545-037.patch
        515 kB
        Steve Loughran
      2. HADOOP-8545-036.patch
        514 kB
        Steve Loughran
      3. HADOOP-8545-035.patch
        513 kB
        Steve Loughran
      4. HADOOP-8545-035.patch
        513 kB
        Suresh Srinivas
      5. HADOOP-8545-034.patch
        513 kB
        Steve Loughran
      6. HADOOP-8545-033.patch
        512 kB
        Steve Loughran
      7. HADOOP-8545-032.patch
        510 kB
        Steve Loughran
      8. HADOOP-8545-031.patch
        510 kB
        Steve Loughran
      9. HADOOP-8545-030.patch
        498 kB
        Steve Loughran
      10. HADOOP-8545.suresh.patch
        486 kB
        Suresh Srinivas
      11. HADOOP-8545-029.patch
        487 kB
        Steve Loughran
      12. HADOOP-8545-028.patch
        491 kB
        Steve Loughran
      13. HADOOP-8545-027.patch
        481 kB
        Steve Loughran
      14. HADOOP-8545-026.patch
        4.68 MB
        Steve Loughran
      15. HADOOP-8545-25.patch
        386 kB
        Dmitry Mezhensky
      16. HADOOP-8545-24.patch
        447 kB
        Steve Loughran
      17. HADOOP-8545-23.patch
        440 kB
        Steve Loughran
      18. HADOOP-8545-22.patch
        397 kB
        Steve Loughran
      19. HADOOP-8545-21.patch
        372 kB
        Steve Loughran
      20. HADOOP-8545-20.patch
        359 kB
        Steve Loughran
      21. HADOOP-8545-19.patch
        359 kB
        Steve Loughran
      22. HADOOP-8545-18.patch
        293 kB
        Dmitry Mezhensky
      23. HADOOP-8545-17.patch
        293 kB
        Dmitry Mezhensky
      24. HADOOP-8545-16.patch
        291 kB
        Dmitry Mezhensky
      25. HADOOP-8545-15.patch
        291 kB
        Dmitry Mezhensky
      26. HADOOP-8545-14.patch
        294 kB
        Dmitry Mezhensky
      27. HADOOP-8545-13.patch
        295 kB
        Dmitry Mezhensky
      28. HADOOP-8545-12.patch
        297 kB
        Dmitry Mezhensky
      29. HADOOP-8545-11.patch
        296 kB
        Dmitry Mezhensky
      30. HADOOP-8545-10.patch
        261 kB
        Dmitry Mezhensky
      31. HADOOP-8545-9.patch
        261 kB
        Dmitry Mezhensky
      32. HADOOP-8545-8.patch
        250 kB
        Dmitry Mezhensky
      33. HADOOP-8545-7.patch
        244 kB
        Dmitry Mezhensky
      34. HADOOP-8545-6.patch
        271 kB
        Dmitry Mezhensky
      35. HADOOP-8545-5.patch
        276 kB
        Dmitry Mezhensky
      36. HADOOP-8545-4.patch
        128 kB
        Dmitry Mezhensky
      37. HADOOP-8545-3.patch
        131 kB
        Dmitry Mezhensky
      38. HADOOP-8545-2.patch
        130 kB
        Dmitry Mezhensky
      39. HADOOP-8545.patch
        121 kB
        Dmitry Mezhensky
      40. HADOOP-8545-javaclouds-2.patch
        77 kB
        Steve Loughran
      41. HADOOP-8545-1.patch
        55 kB
        David Dobbins
      42. HADOOP-8545.patch
        118 kB
        Dmitry Mezhensky

        Issue Links

          Activity

            People

            • Assignee:
              Dmitry Mezhensky
              Reporter:
              Tim Miller
            • Votes:
              6 Vote for this issue
              Watchers:
              46 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development