Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-10737

S3n silent failure on copy, data loss on rename

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 2.4.0
    • 2.5.0
    • fs/s3
    • None

    Description

      Jets3tNativeFileSystemStore.copy(String, String) handles its exceptions with handleServiceException(String, ServiceException), which behaves like:

      1) Throw FileNotFoundException if the exception's error code is NoSuchKey
      2) Otherwise, throw IOException if the exception's cause is an IOException
      3) Otherwise, LOG.debug a message and throw nothing

      So S3 exceptions other than NoSuchKey (like RequestTimeout, ServiceUnavailable) are suppressed. This makes "copy" fail while still returning as if it succeeded. Furthermore since NativeS3FileSystem's "rename" is implemented as a copy followed by a delete, this means "rename" can delete the source key even though the copy has failed.

      Attachments

        Issue Links

          Activity

            People

              stevel@apache.org Steve Loughran
              gian Gian Merlino
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: