[HIVE-28359] Discard old builds in Jenkins to avoid disk space exhaustion

• Type: Task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 4.1.0
• Component/s: Testing Infrastructure
• Labels: pull-request-available
• Assignee: Stamatis Zampetakis
• Reporter: Stamatis Zampetakis
• Created: 03/Jul/24 11:42
• Updated: 19/Jul/24 09:14
• Resolved: 19/Jul/24 09:12

Description

Currently Jenkins retains the builds from all active branches/PRs.

for b in `find var/jenkins_home/jobs -name "builds"`; do echo -n $b" " ; ls -l $b | wc -l; done | sort -k2 -rn > builds.txt

Some PRs (e.g., PR-5216) with an excessive number of builds (i.e., 66) can easily consume many GBs of data (PR-5216 uses 13GB for the builds). The first build for PR-5216 was saved on April 26, 2024 and it is now more than 2 months old.

For master, we currently have all builds since January 2023 (previous builds where manually removed as part of HIVE-28013). The builds for master occupy currently 50GB of space.

Due to the above the disk space (persistent volume) cannot be reclaimed and currently it is almost full (91% /var/jenkins_home).

kubectl exec jenkins-6858ddb664-l4xfg -- bash -c "df"
Filesystem     1K-blocks      Used Available Use% Mounted on
overlay         98831908   4675004  94140520   5% /
tmpfs              65536         0     65536   0% /dev
tmpfs            6645236         0   6645236   0% /sys/fs/cgroup
/dev/sdb       308521792 278996208  29509200  91% /var/jenkins_home
/dev/sda1       98831908   4675004  94140520   5% /etc/hosts
shm                65536         0     65536   0% /dev/shm
tmpfs           10801128        12  10801116   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs            6645236         0   6645236   0% /proc/acpi
tmpfs            6645236         0   6645236   0% /proc/scsi
tmpfs            6645236         0   6645236   0% /sys/firmware

Without a discard policy in place we are going to hit again HIVE-28013 or other disk related issues pretty soon.

Issue Links

• Discovered while testing: HIVE-28339 Upgrade Jenkins version in CI from 2.332.3 to 2.452.2
• relates to: HIVE-28013 No space left on device when running precommit tests
• links to: GitHub Pull Request #5332