Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-10737

S3n silent failure on copy, data loss on rename

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.4.0
    • Fix Version/s: 2.5.0
    • Component/s: fs/s3
    • Labels:
      None

      Description

      Jets3tNativeFileSystemStore.copy(String, String) handles its exceptions with handleServiceException(String, ServiceException), which behaves like:

      1) Throw FileNotFoundException if the exception's error code is NoSuchKey
      2) Otherwise, throw IOException if the exception's cause is an IOException
      3) Otherwise, LOG.debug a message and throw nothing

      So S3 exceptions other than NoSuchKey (like RequestTimeout, ServiceUnavailable) are suppressed. This makes "copy" fail while still returning as if it succeeded. Furthermore since NativeS3FileSystem's "rename" is implemented as a copy followed by a delete, this means "rename" can delete the source key even though the copy has failed.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                stevel@apache.org Steve Loughran
                Reporter:
                gian Gian Merlino
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: