[SPARK-21521] History service requires user is in any group

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 2.2.0
• Component/s: Spark Core
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Adrian Bridgett
• Created: 24/Jul/17 10:40
• Updated: 21/May/19 04:13
• Resolved: 21/May/19 04:13

Description

(Regression cf. 2.0.2)

We run spark as several users, these write to the history location where the files are saved as those users with permissions of 770 (this is hardcoded in EventLoggingListener.scala).

The history service runs as root so that it has permissions on these files (see https://spark.apache.org/docs/latest/security.html).

This worked fine in v2.0.2, however in v2.2.0 the events are being skipped unless I add the root user into each users group at which point they are seen.

We currently have all acls configuration unset.

Issue Links

• is related to: SPARK-22528 History service and non-HDFS filesystems