[HADOOP-12471] Support Swift file (> 5GB) continuious uploading where there is a failure

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.7.1
• Component/s: fs/swift
• Assignee: Unassigned
• Reporter: Chen He
• Created: 09/Oct/15 06:54
• Updated: 12/Oct/15 16:39

Description

Current Swift FileSystem supports file larger than 5GB.
File will be chunked as large as 4.6GB (configurable). For example, if there is a 46GB file "foo" in swift,
Then the structure will look like:

foo/000001
foo/000002
foo/000003
...
foo/000010

User will not see those 00000x files if they don't specify. That means, if user does:
\> hadoop fs -ls swift://container.serviceProvidor/foo

It only shows:
dwr-r-r- 46GB foo

However, in my test, if there is a failure, during uploading the foo file, the previous uploaded chunks will be left in the object store. It will be good to support continuous uploading based on previous leftover

Issue Links

• is blocked by: HADOOP-12109 Distcp of file > 5GB to swift fails with HTTP 413 error