[ARROW-18128] [Java][CI] Java nightlies does not remove binaries keeping the newest ones

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 11.0.0
• Component/s: Continuous Integration, Java
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/33325
• Assignee: Raúl Cumplido
• Reporter: Raúl Cumplido
• Created: 21/Oct/22 13:42
• Updated: 11/Jan/23 11:58
• Resolved: 28/Oct/22 21:03

Description

After some investigation on why some of our jars did not have a correct 10.0.0-SNAPSHOT folder on the repository, i.e:
https://nightlies.apache.org/arrow/java/org/apache/arrow/arrow-vector/

https://nightlies.apache.org/arrow/java/org/apache/arrow/flight-core/

It seems that when pruning the old artifacts we are not ordering based on newer ones:

      - name: Prune Repository
        shell: bash
        env:
          KEEP: ${{ github.event.inputs.keep || 14 }}
        run: |
          for i in `ls -t repo/org/apache/arrow`; do
            find repo/org/apache/arrow/$i -mindepth 1 -maxdepth 1 -type d -print0 \
            | xargs -0 ls -t -d \
            | tail -n +$((KEEP + 1)) \
            | xargs rm -rf
          done 

that makes us delete based on the output from find and having things like:

[DIR] 2022-09-21/                   2022-09-21 15:43    -   
[DIR] 2022-09-22/                   2022-09-22 20:53    -   
[DIR] 2022-09-23/                   2022-09-23 15:14    -   
[DIR] 2022-10-11/                   2022-10-11 15:18    -   
[DIR] 2022-10-12/                   2022-10-12 18:04    -   
[DIR] 2022-10-13/                   2022-10-13 20:35    -   
[DIR] 2022-10-14/                   2022-10-14 17:28    -   
[DIR] 2022-10-15/                   2022-10-15 14:10    -   
[DIR] 2022-10-16/                   2022-10-16 14:13    -   
[DIR] 2022-10-17/                   2022-10-17 14:21    -   
[DIR] 2022-10-18/                   2022-10-18 16:24    -   
[DIR] 2022-10-19/                   2022-10-19 14:31    -   
[DIR] 2022-10-20/                   2022-10-20 17:09    -   

See artifacts for 21st-23rd September and then jumping to 11th October.

We should fix how we prune the older artifacts.

Issue Links

• links to: GitHub Pull Request #14496