[BIGTOP-2603] NN/RM charm should include a spark user/group

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.1.0
• Fix Version/s: 1.2.0
• Component/s: deployment
• Assignee: Kevin Monroe
• Reporter: Kevin Monroe
• Created: 22/Nov/16 23:00
• Updated: 29/Mar/17 18:46
• Resolved: 03/Feb/17 17:37

Description

When in 'yarn-client' mode, the spark charm creates a directory in hdfs with 'spark' user/group permissions.

If spark is not colocated with the NN and RM, this user/group does not exist on those machines. This causes warnings in the hdfs log like this:

WARN org.apache.hadoop.security.UserGroupInformation: No groups available for user spark

It also prevents the RM history server from accessing spark job history files.

To combat this, always include a spark user/group in the NN/RM charms. This is very similar to how we always create a 'mapred' user on the NN to prevent warnings about mapred being unknown to the namenode.

Issue Links

• links to: GitHub Pull Request #163