[HADOOP-18007] Use platform specific endpoints for CI report

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.4.0
• Component/s: build
• Labels: pull-request-available

Environment

Centos 7, Centos 8, Debian 10, Ubuntu Focal

Language

groovy

• Assignee: Gautham Banasandra
• Reporter: Gautham Banasandra
• Due: 24/Dec/21
• Created: 12/Nov/21 16:04
• Updated: 20/Feb/22 14:49

Description

Consider the Github comment by hadoop-yetus for different platform runs -

Platform	hadoop-yetus report
Centos 7	https://github.com/apache/hadoop/pull/3563#issuecomment-947151110
Centos 8	https://github.com/apache/hadoop/pull/3563#issuecomment-947239385
Debian 10	https://github.com/apache/hadoop/pull/3563#issuecomment-947340406
Ubuntu Focal	https://github.com/apache/hadoop/pull/3563#issuecomment-947464004

 
Notice that the Docker subsystem points to the same file for all the platforms -

To illustrate the issue here, upon clicking on the link to Dockerfile in Centos 7's hadoop-yetus report will navigate to Ubuntu Focal's Dockerfile (since Ubuntu Focal wass the most recently run platform for that pipeline run).
Please note that this issue applies to all the links that contain "out/" that appear in the Yetus summary.

As a side note, this issue doesn't happen when the Yetus run fails. Consider https://github.com/apache/hadoop/pull/3563#issuecomment-946215556. Here the Yetus run has failed for Centos 7. The link to Dockerfile points to Centos 7's Dockerfile since Centos 7 is the last platform that was run for that pipeline.

---

To fix this issue, we need to do the following two things -
Archive the artifacts even when the Yetus run is successful
Currently, we're archiving the artifacts only when the run fails - https://github.com/apache/hadoop/blob/bccf2f3ef4c8f09f010656f9061a4e323daf132b/dev-support/Jenkinsfile#L142-L148 (Hence, we're able to navigate to the right Dockerfile when the Yetus run fails for a platform). We need to archive the artifacts even when the run succeeds so that we're able to navigate to it irrespective of success/failure of the Yetus run.

Expose different endpoints for each platform
At the end of the Yetus's pre-commit run, we see the following table -

Subsystem	Report/Notes
cc	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3650/1/*artifact/out*/results-compile-cc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt
unit	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3650/1/*artifact/out*/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt

We need a way to change the URL to /artifact/centos-7/out/, /artifact/ubuntu-focal/out/ and so on.

Issue Links

• links to: GitHub Pull Request #4004