Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-997

Implement S3 retry mechanism for failed block transfers

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.11.0
    • 0.12.0
    • fs
    • None

    Description

      HADOOP-882 improves S3FileSystem so that when certain communications problems with S3 occur the operation is retried. However, the retry mechanism cannot handle a block transfer failure, since blocks may be very large and we don't want to buffer them in memory. This improvement is to write a wrapper (using java.lang.reflect.Proxy if possible - see discussion in HADOOP-882) that can retry block transfers.

      Attachments

        1. HADOOP-997-v3.patch
          46 kB
          Thomas White
        2. HADOOP-997-v2.patch
          45 kB
          Thomas White
        3. HADOOP-997.patch
          39 kB
          Thomas White

        Issue Links

          Activity

            People

              tomwhite Thomas White
              tomwhite Thomas White
              Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: