[YUNIKORN-1865] Shim: Update build process to follow standard conventions

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.4.0
• Component/s: shim - kubernetes
• Labels: pull-request-available
• Target Version: 1.4.0
• Assignee: Craig Condit
• Reporter: Craig Condit
• Created: 14/Jul/23 22:12
• Updated: 20/Nov/23 16:42
• Resolved: 18/Jul/23 02:43

Description

Our current build process does several things that are suboptimal:

• Temporary build files are created within source location (Docker content)
• Output goes in _output instead of more standard locations (i.e. build)
• "clean" target is overused, resulting in far slower builds than necessary
• Build artifacts are inconsistently named, leading to confusion
• Build targets are all marked .PHONY meaning they cannot be cached properly

Additionally, using (for example) a different build of go should be possible.

We should update the build process as follows:

• Move Dockerfiles into docker/ {imagename}/
  - Use output directory for all generated content
  - Use output/dev for development-time binaries
  - Use output/bin for production-time binaries
  - Use output/docker/{imagename}

  for temporary Docker build environments

• Allow setting GO env var to override go binary name (i.e. GO=go1.20.5 or GO=/opt/go1.20.5/bin/go)
• Use proper dependencies for Makefile build targets
• Name output binaries as: yunikorn-scheduler, yunikorn-scheduler-plugin, and yunikorn-admission-controller

Experimentally, doing this results in significant speedups for deployments. An incremental "make image" with no changes now completes in a few seconds.

Issue Links

• requires: YUNIKORN-1862 Shim: Use privately installed build tools
• links to: GitHub Pull Request #635