[SPARK-31340] No call to destroy() for filter in SparkHistory

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 2.4.5
• Component/s: Spark Core
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: thierry accart
• Created: 03/Apr/20 13:00
• Updated: 25/May/21 01:53
• Resolved: 25/May/21 01:42

Description

Adding  UI filter AuthenticationFilter (from Hadoop) causes Spark application to never end, due to threads created in this class not interrupted.

To reproduce

Start a local spark context with hadoop-auth 3.1.0
spark.ui.enabled=true
spark.ui.filters=org.apache.hadoop.security.authentication.server.AuthenticationFilter
#and all required ldap props
spark.org.apache.hadoop.security.authentication.server.AuthenticationFilter.param.ldap.*=...

What's happening :

In AuthenticationFilter's init we have the following chain:

(line.178) initializeSecretProvider(filterConfig);
(line.209) secretProvider = constructSecretProvider(...)
(line 237) provider.init(config, ctx, validity);

If no config is specified provider will be RolloverSignerSecretProvider which will (line 95) start a new thread via
scheduler = Executors.newSingleThreadScheduledExecutor();

The created thread will be stopped in destroy() method (line 106).

Unfortunately, this destroy() method is not called when SparkHistory is closed, leaving threads running.

 

This ticket is not here to address the particular case of Hadoop's authentication filter, but to ensure that any Filter added in spark.ui will have its destroy() method called.