Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-972

distcp can timeout during rename operation to s3

          Details

          • Type: Improvement
          • Status: Closed
          • Priority: Major
          • Resolution: Fixed
          • Affects Version/s: 0.20.1
          • Fix Version/s: 0.21.0
          • Component/s: distcp, documentation
          • Labels:
            None
          • Hadoop Flags:
            Reviewed

            Description

            rename() in S3 is implemented as copy + delete. The S3 copy operation can perform very slowly, which may cause task timeout.

              Attachments

              1. MAPREDUCE-972.2.patch
                3 kB
                Aaron Kimball
              2. MAPREDUCE-972.3.patch
                3 kB
                Aaron Kimball
              3. MAPREDUCE-972.4.patch
                8 kB
                Aaron Kimball
              4. MAPREDUCE-972.5.patch
                11 kB
                Aaron Kimball
              5. MAPREDUCE-972.6.patch
                2 kB
                Aaron Kimball
              6. MAPREDUCE-972.patch
                4 kB
                Aaron Kimball

                Issue Links

                is blocked by

                Improvement - An improvement or enhancement to an existing feature or task. HADOOP-6324 FileSystem API should allow progress callbacks in rename and delete

                • Major - Major loss of function.
                • Open
                is related to

                Improvement - An improvement or enhancement to an existing feature or task. MAPREDUCE-1127 distcp should timeout later during S3-based transfers

                • Major - Major loss of function.
                • Resolved

                  Activity

                    People

                    • Assignee:
                      kimballa Aaron Kimball
                      Reporter:
                      kimballa Aaron Kimball
                    • Votes:
                      0 Vote for this issue
                      Watchers:
                      4 Start watching this issue

                      Dates

                      • Created:
                        Updated:
                        Resolved: