[YETUS-15] build environment

• Type: Improvement
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 0.1.0
• Fix Version/s: 0.9.0
• Component/s: Precommit

Release Note

<!-- markdown -->
Apache Yetus has been converted to use Apache Maven as a build tool. As a result, many changes have taken place that directly impacts the project.

* Source directories have been re-arranged and re-named:
  * All bash code is now in (feature)/src/main/shell
  * All python code is now in (feature)/src/main/python
* audience-annotations is mostly unchanged.
* releasedocmaker and shelldocs are now available as Jython-built jars.
* Introduction of the yetus-minimaven-plugin and yetus-maven-plugins. The yetus-minimaven-plugin is used to build Apache Yetus. yetus-maven-plugin is an end-user artifact that gives access some Apache Yetus features to Apache Maven and compatible build systems without needing any external help (e.g., yetus-wrapper)
* Middleman is still used for creating the static website, however, it is now tied into the 'mvn site' command. 'mvn install' MUST be executed before running 'mvn site' as website generation depends upon the yetus-minimaven-plugin.
* The content of yetus-project is now in the root of the source tree.
* The new yetus-dist module handles the creation of a complete distribution. The artifacts are now in the yetus-dist/target directory. The artifact contents are largely unchanged. New yetus-assemblies module and various Apache Maven configuration files have been added to create distribution parity.
* The website is also available as a tar.gz tarball in the yetus-dist artifact area.
* The jdiff module is now always built.
* Version handling has been modified in several different locations and the executables themselves.

Also, other changes introduced:

* start-build-env.sh has been added to create a Docker-ized development environment. In particular, this imports the .ssh and .gnupg directories and has all pre-requisites for building Apache Yetus and making releases.
* A Dockerfile in root has been added for hub.docker.com and CI-system integration.
* The old Dockerfile (previously located at precommit/test-patch-docker and now located at precommit/src/main/shell/test-patch-docker) has been changed to be able also to run releasedocmaker.
* Some ruby dependencies for the website have been updated for security reasons.
* JDK8 is now the minimum version of Java used to build the Apache Yetus Java components.
* precommit's shellcheck.sh now recognizes src/main/shell as containing shell code to check.
* releasedocmaker and shelldocs now explicitly call for python2

• Flags: Important
• Assignee: Allen Wittenauer
• Reporter: Allen Wittenauer
• Created: 27/Jun/15 16:17
• Updated: 11/Dec/19 05:06
• Resolved: 11/Nov/18 22:27

Description

We need to have some way to build (esp the documentation!) and unit tests.

Issue Links

• blocks: YETUS-683 Publish apache/yetus docker images on hub.docker.com
• blocks: YETUS-681 Add robots plugin type
• blocks: YETUS-6 reorganize repo layout for break from Hadoop code base
• causes: YETUS-739 yetus' mvn site fails when run with built-in dockerifle
• causes: YETUS-783 download links for 0.8.0 vs 0.9.0 are different
• causes: YETUS-897 Remove duplicate jdk.tools dependency from audience-annotations module
• contains: YETUS-676 releasedocmaker and shelldocs should specify python2
• contains: YETUS-125 Set up project-wide build
• contains: YETUS-314 update docker image to build releases
• contains: YETUS-316 document publishing convenience artifacts
• contains: YETUS-317 Website build instructions should recommend deploying Ruby via RVM.
• incorporates: YETUS-313 mvn clean doesn't clean jdiff from parent
• incorporates: YETUS-491 build.sh silently fails if middleman isn't installed
• incorporates: YETUS-673 Write documentation on how to EOL releases
• is blocked by: YETUS-568 break apart versioning of files and directories
• is depended upon by: YETUS-578 Jenkins Precommit Plugin
• relates to: YETUS-54 Add Yetus parent POM and top-level POM