[SPARK-49875] error build spark 3.5.3

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.5.3
• Component/s: Build

Environment

docker image in description

• Assignee: Unassigned
• Reporter: Arsen Gumin
• Created: 04/Oct/24 10:40
• Updated: 04/Oct/24 10:41

Description

i try create make-distribution.sh and get error. maybe I did something wrong?

what did you try to do:

1. Update scala-maven-plugin verison (all upper 4.8)
2. Downgrade scala-maven-plugin (4.7.*)
3. Build without my patch (patch -p1 < $SPARK_HOME/spark-42411.patch)
4. change scala and java version(8,11,17)

 

// docker image with build

FROM eclipse-temurin:17

ENV MAVEN_OPTS="-Xss64m -Xmx2g -XX:ReservedCodeCacheSize=1g"
ENV SCALA_VERSION=2.13
ENV SPARK_VERSION=3.5.3
ENV SPARK_HOME=/opt/spark
ENV SPARK_TGZ_URL=https://archive.apache.org/dist/spark/spark-${SPARK_VERSION}/spark-${SPARK_VERSION}.tgz

WORKDIR /opt

ARG spark_uid=185

RUN groupadd --system --gid=${spark_uid} spark && \
    useradd --system --uid=${spark_uid} --gid=spark spark

RUN apt-get update; \
    apt-get install -y wget patch gettext-base gnupg2 bash tini libc6 libpam-modules krb5-user libnss3 procps net-tools gosu libnss-wrapper; \
    rm -rf /var/lib/apt/lists/*


RUN set -ex; \
    mkdir -p $SPARK_HOME; \
    wget -nv -O /opt/spark.tgz "$SPARK_TGZ_URL"; \
    tar -zxf /opt/spark.tgz --strip-components=1 --directory=$SPARK_HOME; \
    chown -R spark:spark .;


WORKDIR $SPARK_HOME

COPY mtls/spark-42411.patch .  
RUN patch -p1 < $SPARK_HOME/spark-42411.patch

RUN ./dev/change-scala-version.sh ${SCALA_VERSION}
RUN ./dev/make-distribution.sh \
      --name spark-mtls \
      --pip \
      -P"scala-${SCALA_VERSION}" \
      -Pconnect \
      -Pkubernetes 
 

 

 

error

 

 

//
[INFO] Compiler bridge file: /root/.sbt/1.0/zinc/org.scala-sbt/org.scala-sbt-compiler-bridge_2.13-1.9.5-bin_2.13.8__61.0-1.9.5_20230915T135648.jar
[INFO] Compiler bridge file is not installed yet
Downloading from gcs-maven-central-mirror: https://maven-central.storage-download.googleapis.com/maven2/org/scala-sbt/compiler-bridge_2.13/1.9.5/compiler-bridge_2.13-1.9.5-sources.jar
Downloaded from gcs-maven-central-mirror: https://maven-central.storage-download.googleapis.com/maven2/org/scala-sbt/compiler-bridge_2.13/1.9.5/compiler-bridge_2.13-1.9.5-sources.jar (53 kB at 228 kB/s)
Downloading from gcs-maven-central-mirror: https://maven-central.storage-download.googleapis.com/maven2/org/jline/jline/3.22.0/jline-3.22.0.pom
Downloaded from gcs-maven-central-mirror: https://maven-central.storage-download.googleapis.com/maven2/org/jline/jline/3.22.0/jline-3.22.0.pom (18 kB at 99 kB/s)
Downloading from gcs-maven-central-mirror: https://maven-central.storage-download.googleapis.com/maven2/org/jline/jline-parent/3.22.0/jline-parent-3.22.0.pom
Downloaded from gcs-maven-central-mirror: https://maven-central.storage-download.googleapis.com/maven2/org/jline/jline-parent/3.22.0/jline-parent-3.22.0.pom (28 kB at 137 kB/s)
Downloading from gcs-maven-central-mirror: https://maven-central.storage-download.googleapis.com/maven2/org/jline/jline/3.22.0/jline-3.22.0.jar
Downloaded from gcs-maven-central-mirror: https://maven-central.storage-download.googleapis.com/maven2/org/jline/jline/3.22.0/jline-3.22.0.jar (1.1 MB at 1.2 MB/s)
warning: 4 deprecations (since 2.12.0)
warning: 1 deprecation (since 2.12.5)
warning: 2 deprecations (since 2.12.9)
warning: 2 deprecations (since 2.13)
warning: 8 deprecations (since 2.13.0)
warning: 1 deprecation (since 2.13.4)
warning: 18 deprecations in total; re-run with -deprecation for details
warning: 2 feature warnings; re-run with -feature for details
8 warnings
[INFO] Compiler bridge installed
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Spark Project Parent POM 3.5.3:
[INFO] 
[INFO] Spark Project Parent POM ........................... SUCCESS [03:47 min]
[INFO] Spark Project Sketch ............................... FAILURE [ 20.618 s]
[INFO] Spark Project Local DB ............................. SKIPPED
[INFO] Spark Project Networking ........................... SKIPPED
[INFO] Spark Project Shuffle Streaming Service ............ SKIPPED
[INFO] Spark Project Unsafe ............................... SKIPPED
[INFO] Spark Project Common Utils ......................... SKIPPED
[INFO] Spark Project Tags ................................. SKIPPED
[INFO] Spark Project Core ................................. SKIPPED
[INFO] Spark Project GraphX ............................... SKIPPED
[INFO] Spark Project ML Library ........................... SKIPPED
[INFO] Spark Project ML Local Library ..................... SKIPPED
[INFO] Spark Project Tools ................................ SKIPPED
[INFO] Spark Project Streaming ............................ SKIPPED
[INFO] Spark Project SQL API .............................. SKIPPED
[INFO] Spark Project Catalyst ............................. SKIPPED
[INFO] Spark Project SQL .................................. SKIPPED
[INFO] Spark Project Hive ................................. SKIPPED
[INFO] Spark Project Assembly ............................. SKIPPED
[INFO] Spark Project Examples ............................. SKIPPED
[INFO] Spark Project REPL ................................. SKIPPED
[INFO] Spark Project Launcher ............................. SKIPPED
[INFO] Kafka 0.10+ Token Provider for Streaming ........... SKIPPED
[INFO] Spark Integration for Kafka 0.10 ................... SKIPPED
[INFO] Spark Integration for Kafka 0.10 Assembly .......... SKIPPED
[INFO] Kafka 0.10+ Source for Structured Streaming ........ SKIPPED
[INFO] Spark Avro ......................................... SKIPPED
[INFO] Spark Project Connect Server ....................... SKIPPED
[INFO] Spark Project Connect Common ....................... SKIPPED
[INFO] Spark Project Connect Client ....................... SKIPPED
[INFO] Spark Protobuf ..................................... SKIPPED
[INFO] Spark Project Kubernetes ........................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  04:09 min
[INFO] Finished at: 2024-10-04T10:16:17Z
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:4.9.0:compile (scala-compile-first) on project spark-sketch_2.12: Execution scala-compile-first of goal net.alchim31.maven:scala-maven-plugin:4.9.0:compile failed: org.apache.commons.exec.ExecuteException: Process exited with an error: 1 (Exit value: 1) -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <args> -rf :spark-sketch_2.12
Error: building at STEP "RUN ./dev/make-distribution.sh       --name spark-mtls       --pip       -P"scala-${SCALA_VERSION}"       -Pconnect       -Pkubernetes       -Dscala-maven-plugin.version=4.9.0": while running runtime: exit status 1