Uploaded image for project: 'Apache Ozone'
  1. Apache Ozone
  2. HDDS-9095

Cleanup expired incomplete MPUs

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Implemented
    • None
    • 1.4.0
    • OM
    • None

    Description

      In our cluster, there are around few thousands MPU keys in multipartInfoTable, with some of them being few months old. 

      The reason is that these MPU keys are already initiated, and possibly committed few parts, but was not completed / aborted by the user. These spaces can be freed.  

      Similar to the cleanup service OM open key table (HDDS-4120), we can implement clean up on MultipartInfoTable (and related open keys in OpenKeyTable) using a background job. However, instead of using a new OM request / response, we can reuse the OM MPU abort request / response instead, which already handles the cleanup (i.e. expired inflight MPU can be aborted after a defined expiry threshold).

       

      This is similar to AbortIncompleteMultipartUpload lifecycle configurations in AWS S3 (https://aws.amazon.com/blogs/aws-cloud-financial-management/discovering-and-deleting-incomplete-multipart-uploads-to-lower-amazon-s3-costs/).

      Attachments

        Issue Links

          Activity

            People

              ivanandika Ivan Andika
              ivanandika Ivan Andika
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: