[ARROW-16404] [R][CI] Research alternative binary hosting

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Done
• Fix Version/s: 9.0.0
• Component/s: Continuous Integration, R
• External issue URL: https://github.com/apache/arrow/issues/31781
• Assignee: Jacob Wujciak
• Reporter: Jacob Wujciak
• Created: 28/Apr/22 15:27
• Updated: 11/Jan/23 11:43
• Resolved: 09/May/22 19:38

Description

Find a way to host the R nightly binaries in some form of artifactory under ASF umbrella. Currently they are hosted on s3. See ARROW-16401

Python wheels are hosted gemfury.io

cc: kszucs kou amol- raulcd 

Possible solutions, both of which could be extended for other Components that don't need an active server for their binary repos:

• Host nightly builds on apache artifactory
  • https://apache.jfrog.io/artifactory
    • Is this even possible as nightlies are not signed official releases?
    • Size considerations (-> limit to 5 days as with conda?)
  • https://nightlies.apache.org/
    • Made for this purpose, extendable to other components like java that currently have no "nice" solution
    • Entirely under ASF control and on-brand{{}}
    • We could host M1 binaries compiled on Crossbow (by downloading them in arrow ci)
• Host nightly builds on Github via Github Pages using either drat or manually creating the repo structure.
  • To prevent insane repo sizes this will require constant pruning/rewriting of git history 
  • Building and hosting on apache/arrow: 
    • The existing R CI could easily be extended to also commit the binaries to a gh_pages based repository, either on push to master or as a cron job. 
    • Entirely under ASF control and on-brand for users: install.packages("arrow", repos = "https://apache.github.io/arrow/r") vs. install.packages("arrow", repos = "https://arrow-r-nightly.s3.amazonaws.com")
    • Github Pages can be turned on via .asf.yaml - no ticket needed for Infra?
    • Not possible to create M1 binaries (for now?)
  • Hosting on a newly created utility repository apache/arrow-nightly:
    • Build binaries in Crossbow and upload via PAT with write access (not an option for apache/arrow due to security concerns)
    • We could host M1 binaries compiled on Crossbow
    • Entirely under ASF control and on-brand for users: install.packages("arrow", repos = "https://apache.github.io/arrow-nightly/") vs. install.packages("arrow", repos = "https://arrow-r-nightly.s3.amazonaws.com")
    • Does INFRA even allow such "utility" repos?
  • Building and hosting on ursacomputing/crossbow:
    • Neither under ASF control nor on-brand
    • We could host M1 binaries

Issue Links

• blocks: ARROW-16405 [R][CI] Use nightlies.apache.org as dev repo