[FLINK-10390] DataDog MetricReporter leaks connections

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.6.1
• Fix Version/s: 1.6.2, 1.7.0
• Component/s: Runtime / Metrics
• Assignee: Chesnay Schepler
• Reporter: Elias Levy
• Created: 21/Sep/18 22:18
• Updated: 17/Oct/18 08:45
• Resolved: 17/Oct/18 08:45

Description

After upgrading to 1.6.1 from 1.4.2 we starting observing in the log warnings associated with the DataDog metrics reporter:

Sep 21, 2018 9:43:20 PM org.apache.flink.shaded.okhttp3.internal.platform.Platform log WARNING: A connection to https://app.datadoghq.com/ was leaked. Did you forget to close a response body? To see where this was allocated, set the OkHttpClient logger level to FINE: Logger.getLogger(OkHttpClient.class.getName()).setLevel(Level.FINE);

The metric reporter's okhttp dependency version (3.7.0) has not changed, so that does not appear to be the source of the warning.

I believe the issue is the changed made in FLINK-8553. The HTTP calls were made async. The previous code called client.newCall(r).execute().close(). The new call does nothing in the callback, even thought the Callback.onResponse documentation states:

Called when the HTTP response was successfully returned by the remote server. The callback may proceed to read the response body with Response.body. The response is still live until its response body is closed.