[AMBARI-22628] YARN Shuffle Service Can't Be Found On Client-Only Nodes After New Cluster Install

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 2.6.1
• Fix Version/s: 2.6.1
• Assignee: Jonathan Hurley
• Reporter: Kishor Ramakrishnan
• Created: 11/Dec/17 14:02
• Updated: 18/Jan/18 18:26
• Resolved: 12/Dec/17 13:30

Description

Installing a new cluster can create values in yarn-site.xml which have None specified in the classpath for Spark

<property>
      <name>yarn.nodemanager.aux-services.spark2_shuffle.classpath</name>
      <value>/usr/hdp/None/spark2/aux/*</value>
    </property>

 <property>
      <name>yarn.nodemanager.aux-services.spark_shuffle.classpath</name>
      <value>/usr/hdp/None/spark/aux/*</value>
    </property>

<property>
      <name>yarn.timeline-service.entity-group-fs-store.group-id-plugin-classpath</name>
      <value>/usr/hdp/None/spark/hdpLib/*</value>
    </property>

The cause for this is that YARN Clients on hosts without daemons never get a restart command after the initial yarn-site.xml, and can never fill in the correct values. This causes problems when jobs are run on these nodes:

2017-12-04 10:16:41,789 INFO  service.AbstractService (AbstractService.java:noteFailure(272)) - Service org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices failed in state INITED; cause: java.lang.ClassNotFoundException: org.apache.spark.network.yarn.YarnShuffleService
java.lang.ClassNotFoundException: org.apache.spark.network.yarn.YarnShuffleService

Issue Links

• links to: ReviewBoard