Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-11556

SwiftNativeFileSystem rename being interrupted during "fs -put"

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 2.6.0
    • Fix Version/s: None
    • Component/s: fs, fs/swift
    • Labels:
      None

      Description

      When running hadoop fs -put bar swift://... }}, {{FsShell writes to bar.COPYING first, and then renames it to "bar".

      However, it fails when the file is partitioned uploaded.

      15/02/06 21:00:49 DEBUG snative.SwiftNativeFileSystemStore: Source file appears to be partitioned. copying file and deleting children
      15/02/06 21:00:49 DEBUG params.DefaultHttpParams: Set parameter http.method.retry-handler = org.apache.commons.httpclient.DefaultHttpMethodRetryHandler@26bf4e60
      15/02/06 21:00:49 DEBUG params.DefaultHttpParams: Set parameter http.connection.timeout = 15000
      15/02/06 21:00:49 DEBUG params.DefaultHttpParams: Set parameter http.socket.timeout = 60000
      15/02/06 21:00:49 DEBUG http.SwiftRestClient: COPY https://snet-storage101.iad3.clouddrive.com/v1/MossoCloudFS_946265/foo/bar._COPYING_
      X-Auth-Token: ...
      Destination: foo/bar
      User-Agent: Apache Hadoop Swift Client 3.0.0-SNAPSHOT from ... by root source checksum ...
      
      15/02/06 21:00:49 DEBUG httpclient.HttpConnection: Open connection to snet-storage101.iad3.clouddrive.com:443
      15/02/06 21:00:49 DEBUG wire.header: >> "COPY /v1/MossoCloudFS_946265/foo/bar._COPYING_ HTTP/1.1[\r][\n]"
      15/02/06 21:00:49 DEBUG httpclient.HttpMethodBase: Adding Host request header
      15/02/06 21:00:49 DEBUG wire.header: >> "X-Auth-Token: ...[\r][\n]"
      15/02/06 21:00:49 DEBUG wire.header: >> "Destination: foo/bar[\r][\n]"
      15/02/06 21:00:49 DEBUG wire.header: >> "User-Agent: Apache Hadoop Swift Client 3.0.0-SNAPSHOT from ... by root source checksum ...[\r][\n]"
      15/02/06 21:00:49 DEBUG wire.header: >> "Host: snet-storage101.iad3.clouddrive.com[\r][\n]"
      15/02/06 21:00:49 DEBUG wire.header: >> "Content-Length: 0[\r][\n]"
      15/02/06 21:00:49 DEBUG wire.header: >> "[\r][\n]"
      15/02/06 21:00:49 DEBUG methods.EntityEnclosingMethod: Request body has not been specified
      15/02/06 21:01:49 DEBUG httpclient.HttpMethodDirector: Closing the connection.
      15/02/06 21:01:49 DEBUG httpclient.HttpMethodDirector: Method retry handler returned false. Automatic recovery will not be attempted
      15/02/06 21:01:49 DEBUG httpclient.HttpConnection: Releasing connection back to connection manager.
      15/02/06 21:01:49 DEBUG httpclient.HttpConnection: Releasing connection back to connection manager.
      

      The bar file eventually shows up in Swift. However, the bar.COPYING directories and the manifest file are not deleted.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                eddyxu Lei (Eddy) Xu
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated: