Uploaded image for project: 'Hadoop Distributed Data Store'
  1. Hadoop Distributed Data Store
  2. HDDS-434

Provide an s3 compatible REST api for ozone objects

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.3.0, 0.4.0
    • Component/s: None
    • Labels:
      None
    • Target Version/s:

      Description

      S3 REST api is the de facto standard for object stores. Many external tools already support it.

      This issue is about creating a new s3gateway component which implements (most part of) the s3 API using the internal RPC calls.

      Some part of the implementation is very straightforward: we need a new service with usual REST stack and we need to implement the most commont GET/POST/PUT calls. Some other (Authorization, multi-part upload) are more tricky.

      Here I suggest to create an evaluation: first we can implement a skeleton service which could support read only requests without authorization and we can define proper specification for the upload part / authorization during the work.

      As of now the gatway service could be a new standalone application (eg. ozone s3g start) later we can modify it to work as s DatanodePlugin similar to the existing object store plugin.

        Attachments

        1. S3Gateway.pdf
          144 kB
          Anu Engineer
        1.
        Create new s3gateway daemon Sub-task Resolved Elek, Marton
        2.
        Add rest service to the s3gateway Sub-task Resolved Elek, Marton
        3.
        Support virtual-hosted style URLs Sub-task Resolved Bharat Viswanadham
        4.
        Create acceptance test to test aws cli with the s3 gateway Sub-task Resolved Elek, Marton
        5.
        Create Generic exception class to be used by S3 rest services Sub-task Resolved Bharat Viswanadham
        6.
        Implement HeadBucket REST endpoint Sub-task Resolved Bharat Viswanadham
        7.
        Support S3 buckets as first class objects in Ozone Manager - 1 Sub-task Resolved Anu Engineer
        8.
        Implement DeleteBucket REST endpoint Sub-task Resolved Bharat Viswanadham
        9.
        Handle common request identifiers in a transparent way Sub-task Resolved Elek, Marton
        10.
        Support S3 buckets as first class objects in Ozone Manager - 2 Sub-task Resolved Bharat Viswanadham
        11.
        Implement HeadObject REST endpoint Sub-task Resolved LiXin Ge
        12.
        Implement PutBucket REST endpoint Sub-task Resolved Bharat Viswanadham
        13.
        Create delete s3Bucket Sub-task Resolved Bharat Viswanadham
        14.
        Create a logger to print out all of the incoming requests Sub-task Resolved Bharat Viswanadham
        15.
        Update HeadBucket, DeleteBucket to not to have volume in path Sub-task Resolved Bharat Viswanadham
        16.
        Implement PutObject Rest endpoint Sub-task Resolved chencan
        17.
        Implement DeleteObject REST endpoint Sub-task Resolved Elek, Marton
        18.
        Implement GetObject REST endpoint Sub-task Resolved Elek, Marton
        19.
        Implement ListBucket REST endpoint Sub-task Resolved LiXin Ge
        20.
        Remove volume from the url of HeadObject and GetBucket (ListObject) endpoints Sub-task Resolved Elek, Marton
        21.
        Remove {volume} path segments from all the remaining rest endpoints Sub-task Resolved Elek, Marton
        22.
        Remove volume name parsing from VirtualHostStyleFilter Sub-task Resolved Bharat Viswanadham
        23.
        Add a shell command to provide ozone mapping for a S3Bucket Sub-task Resolved Bharat Viswanadham
        24.
        Format of Last-Modified header is invalid in HEAD Object call Sub-task Resolved Elek, Marton
        25.
        Provide web based bucket browser. Sub-task Resolved Elek, Marton
        26.
        Support key multi-delete Sub-task Resolved Elek, Marton
        27.
        Add query parameter to the constructed query in VirtualHostStyleFilter Sub-task Resolved Bharat Viswanadham
        28.
        OS3Exception resource name should be the actual resource name Sub-task Resolved Bharat Viswanadham
        29.
        Support multi-chunk signatures in s3g PUT object endpoint Sub-task Resolved Elek, Marton
        30.
        Implement CopyObject REST endpoint Sub-task Resolved Bharat Viswanadham
        31.
        Parse Authorization header in a separate filter Sub-task Resolved Bharat Viswanadham
        32.
        Support MultiDeleteRequest without XML namespace Sub-task Resolved Elek, Marton
        33.
        Make VirtualHostStyleFilter port agnostic Sub-task Resolved Danilo Perez
        34.
        S3 multi delete request should return XML header in quiet mode Sub-task Resolved Elek, Marton
        35.
        Handle object list requests (GET bucket) without prefix parameter Sub-task Resolved Elek, Marton
        36.
        Use x-amz-storage-class to specify replication type and replication factor Sub-task Resolved Bharat Viswanadham
        37.
        Implement pagination in GET bucket (object list) endpoint Sub-task Resolved Bharat Viswanadham
        38.
        Loading ozone s3 bucket browser could be failed Sub-task Resolved Elek, Marton
        39.
        Create documentation for s3 gateway to the docs Sub-task Resolved Elek, Marton
        40.
        Fix missing jenkins issue in s3gateway module Sub-task Resolved Bharat Viswanadham

          Activity

            People

            • Assignee:
              elek Elek, Marton
              Reporter:
              elek Elek, Marton
            • Votes:
              0 Vote for this issue
              Watchers:
              15 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: