[BIGTOP-2738] spark-worker fails to start

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.2.0
• Fix Version/s: 1.2.1, 1.3.0
• Component/s: spark
• Assignee: Amir F Sanjar
• Reporter: Amir F Sanjar
• Due: 11/Apr/17
• Created: 11/Apr/17 17:40
• Updated: 18/Apr/17 18:16
• Resolved: 12/Apr/17 15:05

Description

Problem is caused by BIGTOP-2490 that assumes only puppet managed orchestration. Thus, breaks other method of deployment .
Cause of the problem:
File bigtop-packages/src/common/spark/spark-worker.svc was changed to look for value $SPARK_MASTER_URL (line 47). This value is defined only in file spark-env.sh (bigtop-deploy/puppet/modules/spark/templates/spark-env.sh) that gets only initialized during puppet deployment.

https://github.com/apache/bigtop/commit/f627c7868cb1123f6595380a7325f538ebdb8913#diff-90a09a528d21e12a96d209ffc4036f1e

I propose to remove BIGTOP-2490 and update Bigtop 1.2 repository asap, Hopefully we will have a proper fix soon.