Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-12471

Support Swift file (> 5GB) continuious uploading where there is a failure

Add voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 2.7.1
    • Fix Version/s: None
    • Component/s: fs/swift
    • Labels:
      None

      Description

      Current Swift FileSystem supports file larger than 5GB.
      File will be chunked as large as 4.6GB (configurable). For example, if there is a 46GB file "foo" in swift,
      Then the structure will look like:

      foo/000001
      foo/000002
      foo/000003
      ...
      foo/000010

      User will not see those 00000x files if they don't specify. That means, if user does:
      \> hadoop fs -ls swift://container.serviceProvidor/foo

      It only shows:
      dwr-r-r- 46GB foo

      However, in my test, if there is a failure, during uploading the foo file, the previous uploaded chunks will be left in the object store. It will be good to support continuous uploading based on previous leftover

        Attachments

        Issue Links

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              airbots Chen He

              Dates

              • Created:
                Updated:

                Issue deployment