Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-15620 Über-jira: S3A phase VI: Hadoop 3.3 features
  3. HADOOP-15628

S3A Filesystem does not check return from AmazonS3Client deleteObjects

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: 2.9.1, 2.8.4, 3.1.1, 3.0.3
    • Fix Version/s: None
    • Component/s: fs/s3
    • Labels:
      None
    • Environment:

      Hadoop 3.0.2 / Hadoop 2.8.3

      Hive 2.3.2 / Hive 2.3.3 / Hive 3.0.0

      Non-AWS S3 implementation

      Description

      Deletes in S3A that use the Multi-Delete functionality in the Amazon S3 api do not check to see if all objects have been succesfully delete. In the event of a failure, the api will still return a 200 OK (which isn't checked currently):

      Delete Code from Hadoop 2.8 

      if (keysToDelete.size() == MAX_ENTRIES_TO_DELETE) {
      DeleteObjectsRequest deleteRequest =
      new DeleteObjectsRequest(bucket).withKeys(keysToDelete);
      s3.deleteObjects(deleteRequest);
      statistics.incrementWriteOps(1);
      keysToDelete.clear();
      }
      

      This should be converted to use the DeleteObjectsResult class from the S3Client: 

      Amazon Code Example

      // Verify that the objects were deleted successfully.
      DeleteObjectsResult delObjRes = s3Client.deleteObjects(multiObjectDeleteRequest); int successfulDeletes = delObjRes.getDeletedObjects().size();
      System.out.println(successfulDeletes + " objects successfully deleted.");
      

      Bucket policies can be misconfigured, and deletes will fail without warning by S3A clients.

       

       

       

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              steveatbat Steve Jacobs
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated: