Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-972

distcp can timeout during rename operation to s3

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.20.1
    • 0.21.0
    • distcp, documentation
    • None
    • Reviewed

    Description

      rename() in S3 is implemented as copy + delete. The S3 copy operation can perform very slowly, which may cause task timeout.

      Attachments

        1. MAPREDUCE-972.2.patch
          3 kB
          Aaron Kimball
        2. MAPREDUCE-972.3.patch
          3 kB
          Aaron Kimball
        3. MAPREDUCE-972.4.patch
          8 kB
          Aaron Kimball
        4. MAPREDUCE-972.5.patch
          11 kB
          Aaron Kimball
        5. MAPREDUCE-972.6.patch
          2 kB
          Aaron Kimball
        6. MAPREDUCE-972.patch
          4 kB
          Aaron Kimball

        Issue Links

        is blocked by

        Improvement - An improvement or enhancement to an existing feature or task. HADOOP-6324 FileSystem API should allow progress callbacks in rename and delete

        • Major - Major loss of function.
        • Open
        is related to

        Improvement - An improvement or enhancement to an existing feature or task. MAPREDUCE-1127 distcp should timeout later during S3-based transfers

        • Major - Major loss of function.
        • Resolved

        Activity
          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            kimballa Aaron Kimball
            kimballa Aaron Kimball
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment