[SPARK-25516] spark-history and spark-shuffle-service scripts can not run in docker image

• Type: Bug
• Status: Resolved
• Priority: Trivial
• Resolution: Not A Problem
• Affects Version/s: 2.3.1
• Component/s: Kubernetes, Spark Core

Environment

Docker 18.06

• Target Version/s: 2.3.2
• Flags: Patch
• Assignee: Unassigned
• Reporter: Richard Kugler
• Created: 24/Sep/18 18:51
• Updated: 17/May/20 18:23
• Resolved: 24/Sep/18 19:54

Description

When attempting to run ./sbin/start-shuffle-service.sh the image errors with the following output:

ps: unrecognized option: p

Adding the procps package fixes this issue.

After correcting the issue with ps, the code will then error with:
{{}}

nohup: can't execute '--': No such file or directory

This is remedied with the inclusion of coreutils to the apk add statement.

 

 

The ./sbin/start-history-server.sh also requires the added packages to successfully start.

Issue Links

• links to: [Github] Pull Request #22538 (BYondRAK)