[HDDS-1877] hadoop31-mapreduce fails due to wrong HADOOP_VERSION

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 0.4.1, 0.5.0
• Fix Version/s: 0.4.1
• Component/s: test
• Labels: pull-request-available
• Target Version/s: 0.4.1, 0.5.0
• Sprint: Done
• Assignee: Attila Doroszlai
• Reporter: Attila Doroszlai
• Created: 31/Jul/19 08:49
• Updated: 05/Aug/19 14:45
• Resolved: 31/Jul/19 14:08

Description

hadoop31-mapreduce fails with:

https://elek.github.io/ozone-ci/byscane/byscane-nightly-gl52x/acceptance/smokeresult/log.html#s1-s2-t2-k2-k2

JAR does not exist or is not a normal file: /opt/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.2.jar

because 3.1 test is being run with HADOOP_VERSION=3:

https://raw.githubusercontent.com/elek/ozone-ci/master/byscane/byscane-nightly-gl52x/acceptance/output.log

Creating network "hadoop31_default" with the default driver
Pulling nm (flokkr/hadoop:3)...
3: Pulling from flokkr/hadoop
Digest: sha256:62e3488e64ff8c0406752fc4f263ae2549e04fedf02534469913c496c6a89d78
Status: Downloaded newer image for flokkr/hadoop:3

which has Hadoop 3.2.0 instead of 3.1.2:

docker run -it --entrypoint /bin/bash flokkr/hadoop:3 -c 'ls -la /opt/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples*'

-rw-r--r--    1 hadoop   flokkr      316570 Jan  8  2019 /opt/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.0.jar

docker run -it --entrypoint /bin/bash flokkr/hadoop:3.1.2 -c 'ls -la /opt/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples*'

-rw-r--r--    1 hadoop   flokkr      316380 Jan 29  2019 /opt/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.2.jar

This only happens with acceptance.sh, not when directly using test-all.sh, because the former explicitly defines HADOOP_VERSION:

https://github.com/apache/hadoop/blob/d4ab9aea6f9cbcdcaf48b821e5be04b4e952b133/hadoop-ozone/dev-support/checks/acceptance.sh#L19

export HADOOP_VERSION=3

so the correct value from .env file is ignored:

https://github.com/apache/hadoop/blob/d4ab9aea6f9cbcdcaf48b821e5be04b4e952b133/hadoop-ozone/dist/src/main/compose/ozone-mr/hadoop31/.env#L21

HADOOP_VERSION=3.1.2

Issue Links

• links to: GitHub Pull Request #1193