[HBASE-24936] review Jenkins build artifacts

• Type: Task
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Assignee: Duo Zhang
• Reporter: Sean Busbey
• Created: 23/Aug/20 14:20
• Updated: 01/Feb/22 16:04
• Resolved: 01/Feb/22 16:04

Description

Post move to the ci-hadoop build servers we are now the biggest user of space. That is not a problem in and of itself, but the master node has run out of disk space twice now. As of this snapshot we are using 125GB of storage and the next largest project is only using 20GB.

https://paste.apache.org/kyrds

We should review our builds for any issues and come up with expectations for what our steady-state disk usage should look like

• we are supposed to compress any test logs (usually this gets us 90-99% space savings)
• we are supposed to clean up workspaces when jobs are done
• we are supposed to keep a fixed window of prior builds (either by days or number of runs)

If all of our jobs are currently following these guidelines, another possibility is to push the artifacts we need over to nightlies.a.o. Barring that, we should formally request asf infra set up a plugin for storing artifact on s3.

Issue Links

• is related to: HBASE-24930 Reduce the number builds to keep for our jenkins jobs

Sub-Tasks

• Do not publish workspace in flaky find job: Resolved
• Compress the surefire report for flaky tests jenkins job: Resolved
• Change the reference url to flaky list in our jenkins jobs: Resolved
• Publish the test logs for our nightly jobs to nightlies.apache.org: Resolved
• Publish the test logs for pre commit jenkins job to nightlies: Resolved
• Publish test logs for flaky jobs to nightlies: Resolved
• Make flaky find job track more builds: Resolved
• SshPublisher will skip uploading artifacts if the build is failure: Resolved
• Upload dashboard html for flaky find job to nightlies: Resolved
• Use gen_redirect_html for linking test logs: Resolved
• Upload mvn site artifacts for nightly build to nightlies: Resolved
• Use gen_redirect_html for linking flaky test logs: Resolved
• Upload src tarball to nightlies for nightly jobs: Resolved