Uploaded image for project: 'Apache Ozone'
  1. Apache Ozone
  2. HDDS-434

Provide an s3 compatible REST api for ozone objects

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 0.3.0, 0.4.0
    • None
    • None

    Description

      S3 REST api is the de facto standard for object stores. Many external tools already support it.

      This issue is about creating a new s3gateway component which implements (most part of) the s3 API using the internal RPC calls.

      Some part of the implementation is very straightforward: we need a new service with usual REST stack and we need to implement the most commont GET/POST/PUT calls. Some other (Authorization, multi-part upload) are more tricky.

      Here I suggest to create an evaluation: first we can implement a skeleton service which could support read only requests without authorization and we can define proper specification for the upload part / authorization during the work.

      As of now the gatway service could be a new standalone application (eg. ozone s3g start) later we can modify it to work as s DatanodePlugin similar to the existing object store plugin.

      Attachments

        1. S3Gateway.pdf
          144 kB
          Anu Engineer
        1.
        Create new s3gateway daemon Sub-task Resolved Marton Elek
        2.
        Add rest service to the s3gateway Sub-task Resolved Marton Elek
        3.
        Support virtual-hosted style URLs Sub-task Resolved Bharat Viswanadham
        4.
        Create acceptance test to test aws cli with the s3 gateway Sub-task Resolved Marton Elek
        5.
        Create Generic exception class to be used by S3 rest services Sub-task Resolved Bharat Viswanadham
        6.
        Implement HeadBucket REST endpoint Sub-task Resolved Bharat Viswanadham
        7.
        Support S3 buckets as first class objects in Ozone Manager - 1 Sub-task Resolved Anu Engineer
        8.
        Implement DeleteBucket REST endpoint Sub-task Resolved Bharat Viswanadham
        9.
        Handle common request identifiers in a transparent way Sub-task Resolved Marton Elek
        10.
        Support S3 buckets as first class objects in Ozone Manager - 2 Sub-task Resolved Bharat Viswanadham
        11.
        Implement HeadObject REST endpoint Sub-task Resolved LiXin Ge
        12.
        Implement PutBucket REST endpoint Sub-task Resolved Bharat Viswanadham
        13.
        Create delete s3Bucket Sub-task Resolved Bharat Viswanadham
        14.
        Create a logger to print out all of the incoming requests Sub-task Resolved Bharat Viswanadham
        15.
        Update HeadBucket, DeleteBucket to not to have volume in path Sub-task Resolved Bharat Viswanadham
        16.
        Implement PutObject Rest endpoint Sub-task Resolved chencan
        17.
        Implement DeleteObject REST endpoint Sub-task Resolved Marton Elek
        18.
        Implement GetObject REST endpoint Sub-task Resolved Marton Elek
        19.
        Implement ListBucket REST endpoint Sub-task Resolved LiXin Ge
        20.
        Remove volume from the url of HeadObject and GetBucket (ListObject) endpoints Sub-task Resolved Marton Elek
        21.
        Remove {volume} path segments from all the remaining rest endpoints Sub-task Resolved Marton Elek
        22.
        Remove volume name parsing from VirtualHostStyleFilter Sub-task Resolved Bharat Viswanadham
        23.
        Add a shell command to provide ozone mapping for a S3Bucket Sub-task Resolved Bharat Viswanadham
        24.
        Format of Last-Modified header is invalid in HEAD Object call Sub-task Resolved Marton Elek
        25.
        Provide web based bucket browser. Sub-task Resolved Marton Elek
        26.
        Support key multi-delete Sub-task Resolved Marton Elek
        27.
        Add query parameter to the constructed query in VirtualHostStyleFilter Sub-task Resolved Bharat Viswanadham
        28.
        OS3Exception resource name should be the actual resource name Sub-task Resolved Bharat Viswanadham
        29.
        Support multi-chunk signatures in s3g PUT object endpoint Sub-task Resolved Marton Elek
        30.
        Implement CopyObject REST endpoint Sub-task Resolved Bharat Viswanadham
        31.
        Parse Authorization header in a separate filter Sub-task Resolved Bharat Viswanadham
        32.
        Support MultiDeleteRequest without XML namespace Sub-task Resolved Marton Elek
        33.
        Make VirtualHostStyleFilter port agnostic Sub-task Resolved Danilo Perez
        34.
        S3 multi delete request should return XML header in quiet mode Sub-task Resolved Marton Elek
        35.
        Handle object list requests (GET bucket) without prefix parameter Sub-task Resolved Marton Elek
        36.
        Use x-amz-storage-class to specify replication type and replication factor Sub-task Resolved Bharat Viswanadham
        37.
        Implement pagination in GET bucket (object list) endpoint Sub-task Resolved Bharat Viswanadham
        38.
        Loading ozone s3 bucket browser could be failed Sub-task Resolved Marton Elek
        39.
        Create documentation for s3 gateway to the docs Sub-task Resolved Marton Elek
        40.
        Fix missing jenkins issue in s3gateway module Sub-task Resolved Bharat Viswanadham

        Activity

          People

            elek Marton Elek
            elek Marton Elek
            Votes:
            0 Vote for this issue
            Watchers:
            15 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: