[AMBARI-21572] Spark configuration mismatch post express upgrade

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.5.2
• Fix Version/s: 2.5.2
• Component/s: service-advisor
• Epic Link: Cross Stack Name Upgrades
• Assignee: Attila Doroszlai
• Reporter: Eric Yang
• Created: 25/Jul/17 16:13
• Updated: 31/Jul/17 08:17
• Resolved: 31/Jul/17 07:27

Description

During upgrade from IOP 4.2.x to HDP 2.6.2, there is a warning with the spark property spark.yarn.archive the value is:

hdfs:///iop/apps/4.2.5.0-0000/spark2/spark2-iop-yarn-archive.tar.gz prior to the EU
hdfs:///hdp/apps/4.2.5.0-0000/spark2/spark2-iop-yarn-archive.tar.gz post EU

If this property isn't used by HDP it should be deleted during the stack upgrade. If it is used by HDP then the version number 4.2.5.0-0000 should be updated to a param.

Spark2 alert had issues connecting to the thrift server post migration where it tried to ping port 4038 (defined by spark.thriftserver.ui.port, carried over from IOP before migration) while the process really was running on 4040. I got the alerts working again by updating spark.thriftserver.ui.port to 4040.

Issue Links

• links to: Review request