[AMBARI-21770] Spark1 Shuffle Property Is Removed Incorrectly on a Stack Upgrade

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 2.5.2
• Component/s: ambari-server
• Assignee: Jonathan Hurley
• Reporter: Yesha Vora
• Created: 21/Aug/17 18:09
• Updated: 22/Aug/17 02:22
• Resolved: 21/Aug/17 22:57

Description

Steps to reproduce:

• Install BI 4.2.0
• Upgrade to Ambari-2.5.2.0
• Upgrade to HDP-2.6

Here, Spark-shuffle service is not configured for Spark1 component. It only configures Spark2-shuffle service

current yarn-site.xml after upgrade

yarn.nodemanager.aux-services = mapreduce_shuffle,spark2_shuffle
yarn.nodemanager.aux-services.spark2_shuffle.class = org.apache.spark.network.yarn.YarnShuffleService
yarn.nodemanager.aux-services.spark2_shuffle.classpath={{stack_root}}/${hdp.version}/spark2/aux/*
yarn.nodemanager.aux-services.spark_shuffle.class=org.apache.spark.network.yarn.YarnShuffleService
yarn.nodemanager.aux-services.spark_shuffle.classpath={{stack_root}}/${hdp.version}/spark/aux/*
yarn.timeline-service.entity-group-fs-store.group-id-plugin-classes  <EMPTY VALUE>
yarn.timeline-service.entity-group-fs-store.group-id-plugin-classpath <EMPTY VALUE>

expected yarn-site.xml

yarn.nodemanager.aux-services = mapreduce_shuffle,spark_shuffle
yarn.nodemanager.aux-services.spark_shuffle.class=org.apache.spark.network.yarn.YarnShuffleService
yarn.nodemanager.aux-services.spark_shuffle.classpath={{stack_root}}/${hdp.version}/spark/aux/*
yarn.timeline-service.entity-group-fs-store.group-id-plugin-classpath=/usr/hdp/${hdp.version}/spark/hdpLib/*
yarn.timeline-service.entity-group-fs-store.group-id-plugin-classes=org.apache.spark.deploy.history.yarn.plugin.SparkATSPlugin

Issue Links

• links to: ReviewBoard