Uploaded image for project: 'Bigtop'
  1. Bigtop
  2. BIGTOP-2603

NN/RM charm should include a spark user/group

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 1.1.0
    • Fix Version/s: 1.2.0
    • Component/s: deployment
    • Labels:
      None

      Description

      When in 'yarn-client' mode, the spark charm creates a directory in hdfs with 'spark' user/group permissions.

      If spark is not colocated with the NN and RM, this user/group does not exist on those machines. This causes warnings in the hdfs log like this:

      WARN org.apache.hadoop.security.UserGroupInformation: No groups available for user spark

      It also prevents the RM history server from accessing spark job history files.

      To combat this, always include a spark user/group in the NN/RM charms. This is very similar to how we always create a 'mapred' user on the NN to prevent warnings about mapred being unknown to the namenode.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                kwmonroe Kevin W Monroe
                Reporter:
                kwmonroe Kevin W Monroe
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: