Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-18028 High performance S3A input stream with prefetching & caching
  3. HADOOP-18218

Ensure S3A prefetching stream memory consumption scales

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 3.4.0
    • None
    • fs/s3
    • None

    Description

      A recurrent problem in cloud store IO is running out of memory because blocks are buffered in reads or writes.

      We need to make sure that data/memory is managed in the prefetch code such that it works in processes with many worker threads (hive, spark....) and does not exhaust resources.

      Attachments

        Issue Links

          is related to

          Sub-task - The sub-task of the issue HADOOP-18291 S3A prefetch - Implement LRU cache for SingleFilePerBlockCache

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              Unassigned Unassigned
              stevel@apache.org Steve Loughran
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated: