[SPARK-24948] SHS filters wrongly some applications due to permission check

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 2.3.1
• Fix Version/s: 2.2.3, 2.3.2, 2.4.0
• Component/s: Web UI
• Target Version/s: 2.2.3, 2.3.2, 2.4.0
• Assignee: Marco Gaido
• Reporter: Marco Gaido
• Created: 27/Jul/18 16:00
• Updated: 25/Jun/19 10:29
• Resolved: 06/Aug/18 21:34

Description

SHS filters the event logs it doesn't have permissions to read. Unfortunately, this check is quite naive, as it takes into account only the base permissions (ie. user, group, other permissions). For instance, if ACL are enabled, they are ignored in this check; moreover, each filesystem may have different policies (eg. they can consider spark as a superuser who can access everything).

This results in some applications not being displayed in the SHS, despite the Spark user (or whatever user the SHS is started with) can actually read their ent logs.

Issue Links

• causes: SPARK-28157 Make SHS clear KVStore LogInfo for the blacklisted entries
• links to: [Github] Pull Request #21895 (mgaido91)
• links to: [Github] Pull Request #22021 (mgaido91)
• links to: [Github] Pull Request #22022 (mgaido91)