[HADOOP-11556] SwiftNativeFileSystem rename being interrupted during "fs -put"

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.6.0
• Component/s: fs, fs/swift
• Assignee: Unassigned
• Reporter: Lei (Eddy) Xu
• Created: 06/Feb/15 21:57
• Updated: 13/Feb/15 01:24

Description

When running hadoop fs -put bar swift://... }}, {{FsShell writes to bar.COPYING first, and then renames it to "bar".

However, it fails when the file is partitioned uploaded.

15/02/06 21:00:49 DEBUG snative.SwiftNativeFileSystemStore: Source file appears to be partitioned. copying file and deleting children
15/02/06 21:00:49 DEBUG params.DefaultHttpParams: Set parameter http.method.retry-handler = org.apache.commons.httpclient.DefaultHttpMethodRetryHandler@26bf4e60
15/02/06 21:00:49 DEBUG params.DefaultHttpParams: Set parameter http.connection.timeout = 15000
15/02/06 21:00:49 DEBUG params.DefaultHttpParams: Set parameter http.socket.timeout = 60000
15/02/06 21:00:49 DEBUG http.SwiftRestClient: COPY https://snet-storage101.iad3.clouddrive.com/v1/MossoCloudFS_946265/foo/bar._COPYING_
X-Auth-Token: ...
Destination: foo/bar
User-Agent: Apache Hadoop Swift Client 3.0.0-SNAPSHOT from ... by root source checksum ...

15/02/06 21:00:49 DEBUG httpclient.HttpConnection: Open connection to snet-storage101.iad3.clouddrive.com:443
15/02/06 21:00:49 DEBUG wire.header: >> "COPY /v1/MossoCloudFS_946265/foo/bar._COPYING_ HTTP/1.1[\r][\n]"
15/02/06 21:00:49 DEBUG httpclient.HttpMethodBase: Adding Host request header
15/02/06 21:00:49 DEBUG wire.header: >> "X-Auth-Token: ...[\r][\n]"
15/02/06 21:00:49 DEBUG wire.header: >> "Destination: foo/bar[\r][\n]"
15/02/06 21:00:49 DEBUG wire.header: >> "User-Agent: Apache Hadoop Swift Client 3.0.0-SNAPSHOT from ... by root source checksum ...[\r][\n]"
15/02/06 21:00:49 DEBUG wire.header: >> "Host: snet-storage101.iad3.clouddrive.com[\r][\n]"
15/02/06 21:00:49 DEBUG wire.header: >> "Content-Length: 0[\r][\n]"
15/02/06 21:00:49 DEBUG wire.header: >> "[\r][\n]"
15/02/06 21:00:49 DEBUG methods.EntityEnclosingMethod: Request body has not been specified
15/02/06 21:01:49 DEBUG httpclient.HttpMethodDirector: Closing the connection.
15/02/06 21:01:49 DEBUG httpclient.HttpMethodDirector: Method retry handler returned false. Automatic recovery will not be attempted
15/02/06 21:01:49 DEBUG httpclient.HttpConnection: Releasing connection back to connection manager.
15/02/06 21:01:49 DEBUG httpclient.HttpConnection: Releasing connection back to connection manager.

The bar file eventually shows up in Swift. However, the bar.COPYING directories and the manifest file are not deleted.

Issue Links

• depends upon: HADOOP-9565 Add a Blobstore interface to add to blobstore FileSystems