Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-32663

TransportClient getting closed when there are outstanding requests to the server

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.0.0
    • 3.0.1, 3.1.0
    • Shuffle
    • None

    Description

      The implementation of removeBlocks and getHostLocalDirs in ExternalBlockStoreClient closes the client after processing a response in the callback.

      This is a cached client which will be re-used for other responses. There could be other outstanding request to the shuffle service, so it should not be closed after processing a response.
      Seems like this is a bug introduced with SPARK-27651 and SPARK-27677.

      The older methods registerWithShuffleServer and fetchBlocks didn't close the client.

      cc attilapiros vanzin mridulm80

      Attachments

        Activity

          People

            attilapiros Attila Zsolt Piros
            csingh Chandni Singh
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: