[SPARK-32008] 3.0.0 release build fails

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.0.0
• Component/s: Build, Documentation
• Assignee: Unassigned
• Reporter: Philipp Dallig
• Created: 16/Jun/20 20:35
• Updated: 17/Jun/20 08:40
• Resolved: 17/Jun/20 08:40

Description

Hi,
I try to build the spark release 3.0.0 by myself.

I got the following error.

                                          
20/06/16 15:20:49 WARN PrefixSpan: Input data is not cached.
20/06/16 15:20:50 WARN Instrumentation: [b307b568] regParam is zero, which might cause numerical instability and overfitting.
Error: processing vignette 'sparkr-vignettes.Rmd' failed with diagnostics:
'vignetteInfo' is not an exported object from 'namespace:tools'
Execution halted

I can reproduce this error with a small Dockerfile.

EDIT: Issue is solved with ubuntu 20.04 as base image.

FROM ubuntu:20.04 as builder

ENV MVN_VERSION=3.6.3 \
    M2_HOME=/opt/apache-maven \
    MAVEN_HOME=/opt/apache-maven \
    MVN_HOME=/opt/apache-maven \
    MVN_SHA512=c35a1803a6e70a126e80b2b3ae33eed961f83ed74d18fcd16909b2d44d7dada3203f1ffe726c17ef8dcca2dcaa9fca676987befeadc9b9f759967a8cb77181c0 \
    MAVEN_OPTS="-Xmx3g -XX:ReservedCodeCacheSize=1g" \
    R_HOME=/usr/lib/R \
    GIT_REPO=https://github.com/apache/spark.git \
    GIT_BRANCH=v3.0.0 \
    SPARK_DISTRO_NAME=hadoop3.2 \
    SPARK_LOCAL_HOSTNAME=localhost

# Preparation
RUN /usr/bin/apt-get update && \
    # APT
    INSTALL_PKGS="openjdk-8-jdk-headless git wget python3 python3-pip python3-setuptools r-base r-base-dev pandoc pandoc-citeproc libcurl4-openssl-dev libssl-dev libxml2-dev texlive qpdf language-pack-en" && \
    DEBIAN_FRONTEND=noninteractive /usr/bin/apt-get -y install --no-install-recommends $INSTALL_PKGS && \
    rm -rf /var/lib/apt/lists/* && \
    Rscript -e "install.packages(c('knitr', 'rmarkdown', 'devtools', 'testthat', 'e1071', 'survival'), repos='https://cloud.r-project.org/')" && \
    # Maven
    /usr/bin/wget -nv -O apache-maven.tar.gz "https://www.apache.org/dyn/mirrors/mirrors.cgi?action=download&filename=maven/maven-3/${MVN_VERSION}/binaries/apache-maven-${MVN_VERSION}-bin.tar.gz" && \
    echo "${MVN_SHA512} apache-maven.tar.gz" > apache-maven.sha512 && \
    sha512sum --strict -c apache-maven.sha512 && \
    tar -xvzf apache-maven.tar.gz -C /opt && \
    rm -v apache-maven.sha512 apache-maven.tar.gz && \
    /bin/ln -vs /opt/apache-maven-${MVN_VERSION} /opt/apache-maven && \
    /bin/ln -vs /opt/apache-maven/bin/mvn /usr/bin/mvn

# Spark Distribution Build
RUN mkdir -p /workspace && \
    cd /workspace && \
    git clone --branch ${GIT_BRANCH} ${GIT_REPO} && \
    cd /workspace/spark && \
    ./dev/make-distribution.sh --name ${SPARK_DISTRO_NAME} --pip --r --tgz -Psparkr -Phadoop-3.2 -Phive-2.3 -Phive-thriftserver -Pyarn -Pkubernetes

I am very grateful to all helpers.