Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13345 S3Guard: Improved Consistency for S3A
  3. HADOOP-14423

s3guard will set file length to -1 on a putObjectDirect(stream, -1) call

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 3.0.0-alpha4
    • HADOOP-13345
    • fs/s3
    • None

    Description

      You can pass a negative number into S3AFileSystem.putObjectDirect, which means "put until the end of the stream". S3guard has been using this len argument: it needs to be using the actual number of bytes uploaded. Also relevant with client side encryption, when the amount of data put > the amount of data in the file or stream.

      Noted in the committer branch after I added some more assertions, I've changed it there so making changes to S3AFS.putObjectDirect to pull the content length to pass in to finishedWrite() from the PutObjectResult instead. This can be picked into the s3guard branch

      Attachments

        1. HADOOP-14423-HADOOP-13345-002.patch
          5 kB
          Steve Loughran
        2. HADOOP-14423-HADOOP-13345-001.patch
          5 kB
          Steve Loughran

        Issue Links

          Activity

            People

              stevel@apache.org Steve Loughran
              stevel@apache.org Steve Loughran
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: