Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-23182

Allow enabling of TCP keep alive for RPC connections

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 2.2.2, 2.4.0
    • 3.0.0
    • Spark Core
    • None

    Description

      We rely heavily on preemptible worker machines in GCP/GCE. These machines disappear without closing the TCP connections to the master which increases the number of established connections and new workers can not connect because of "Too many open files" on the master.

      To solve the problem we need to enable TCP keep alive for the RPC connections to the master but it's not possible to do so via configuration.

      Attachments

        Activity

          People

            pesho82 Petar Petrov
            pesho82 Petar Petrov
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: