Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-997

Implement S3 retry mechanism for failed block transfers

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.11.0
    • Fix Version/s: 0.12.0
    • Component/s: fs
    • Labels:
      None

      Description

      HADOOP-882 improves S3FileSystem so that when certain communications problems with S3 occur the operation is retried. However, the retry mechanism cannot handle a block transfer failure, since blocks may be very large and we don't want to buffer them in memory. This improvement is to write a wrapper (using java.lang.reflect.Proxy if possible - see discussion in HADOOP-882) that can retry block transfers.

        Attachments

        1. HADOOP-997.patch
          39 kB
          Tom White
        2. HADOOP-997-v2.patch
          45 kB
          Tom White
        3. HADOOP-997-v3.patch
          46 kB
          Tom White

          Issue Links

            Activity

              People

              • Assignee:
                tomwhite Tom White
                Reporter:
                tomwhite Tom White
              • Votes:
                0 Vote for this issue
                Watchers:
                0 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: