[AMBARI-14764] MapReduce, Oozie service check failures after upgrade finished

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.2.0
• Fix Version/s: 2.2.2
• Component/s: ambari-server
• Assignee: Dmitry Lysnichenko
• Reporter: Dmitry Lysnichenko
• Created: 21/Jan/16 19:15
• Updated: 05/Feb/16 22:03
• Resolved: 05/Feb/16 17:59

Description

1. MapReduce jobs were failing after upgrading and restarting MapReduce2 service (ClassNotFound exception). Also Oozie job were failing because of the same reason.
2. This was caused because mapreduce.application.classpath property had the older version of the LZO codec (/usr/hdp/current/hadoop/lib/hadoop-lzo-0.6.0.<OLD-STACK-VERSION>.jar).
3. During upgrade the path was valid because we had not switched /usr/hdp/current/hadoop symlink to new version. After upgrade, once the symlink is updated to new stack version and after restarting MapReduce2 this path becomes invalid.
4. To fix this updated the LZO codec path to /usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.${hdp.version}.jar in mapreduce.application.classpath config property. Also in oozie-env property added export HADOOP_OPTS="-Dhdp.version=$HDP_VERSION $HADOOP_OPTS”

Fix:

1. During stack upgrade, update lzo codec path in mapreduce.application.classpath to

   /usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.${hdp.version}.jar
   

Also, looks like we also need to update following property for TEZ.

<property>
<name>tez.cluster.additional.classpath.prefix</name>
<value>/usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.${hdp.version}.jar:/etc/hadoop/conf/secure</value>
<description></description>
</property>

1. Update oozie-env and add

   export HADOOP_OPTS="-Dhdp.version=$HDP_VERSION $HADOOP_OPTS"
   

2. Add upgrade pre-checks to validate if any config properties have old-stack-version hardcodings and report these config properties to be updated.

Issue Links

• links to: review_request_