Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-8125

Support limiting the number of open FileSystem connections

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.4.0
    • Component/s: None
    • Labels:
      None

      Description

      We need a way to limit the number of streams that Flink FileSystems concurrently open.
      For example, for very small HDFS clusters with few RPC handlers, a large Flink job trying to build up many connections during a checkpoint causes failures due to rejected connections.

      I propose to add a file system that can wrap another existing file system The file system may track the progress of streams and close streams that have been inactive for too long, to avoid locked streams of taking up the complete pool.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                sewen Stephan Ewen
                Reporter:
                sewen Stephan Ewen
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: