Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
3.1.1
-
None
-
None
Description
build-classpath could not get correct classpath with -pl:
$ git clone https://github.com/wangyum/spark.git $ cd spark && git checkout CARMEL-363 $ build/mvn -q versions:set -DnewVersion=CARMEL-363 -DgenerateBackupPoms=false -Phadoop-3.1 $ build/mvn -Phadoop-3.1 jar:jar jar:test-jar install:install clean $ build/mvn -Phadoop-3.1 dependency:build-classpath -pl assembly
It always add orc-core-1.5.4-nohive.jar and orc-mapreduce-1.5.4-nohive.jar to classpath:
RoaringBitmap-0.5.11.jar accessors-smart-1.2.jar activation-1.1.1.jar aircompressor-0.10.jar antlr4-runtime-4.7.1.jar aopalliance-repackaged-2.4.0-b34.jar arpack_combined_all-0.1.jar arrow-format-0.12.0.jar arrow-memory-0.12.0.jar arrow-vector-0.12.0.jar audience-annotations-0.5.0.jar avro-1.8.2.jar avro-ipc-1.8.2.jar avro-mapred-1.8.2-hadoop2.jar breeze-macros_2.12-0.13.2.jar breeze_2.12-0.13.2.jar chill-java-0.9.3.jar chill_2.12-0.9.3.jar commons-beanutils-1.9.3.jar commons-cli-1.2.jar commons-codec-1.10.jar commons-collections-3.2.2.jar commons-compiler-3.0.11.jar commons-compress-1.9.jar commons-configuration2-2.1.1.jar commons-crypto-1.0.0.jar commons-io-2.4.jar commons-lang-2.6.jar commons-lang3-3.8.1.jar commons-math3-3.4.1.jar commons-net-3.1.jar commons-text-1.6.jar compress-lzf-1.0.3.jar core-1.1.2.jar curator-client-2.13.0.jar curator-framework-2.13.0.jar curator-recipes-2.13.0.jar dnsjava-2.1.7.jar flatbuffers-java-1.9.0.jar gson-2.2.4.jar guava-14.0.1.jar hadoop-annotations-3.2.0.jar hadoop-auth-3.2.0.jar hadoop-client-3.2.0.jar hadoop-common-3.2.0.jar hadoop-hdfs-client-3.2.0.jar hadoop-mapreduce-client-common-3.2.0.jar hadoop-mapreduce-client-core-3.2.0.jar hadoop-mapreduce-client-jobclient-3.2.0.jar hadoop-yarn-api-3.2.0.jar hadoop-yarn-client-3.2.0.jar hadoop-yarn-common-3.2.0.jar hive-common-2.3.4.jar hive-llap-client-2.3.4.jar hive-llap-common-2.3.4.jar hive-serde-2.3.4.jar hive-service-rpc-2.3.4.jar hive-shims-0.23-2.3.4.jar hive-shims-2.3.4.jar hive-shims-common-2.3.4.jar hive-shims-scheduler-2.3.4.jar hive-storage-api-2.6.0.jar hk2-api-2.4.0-b34.jar hk2-locator-2.4.0-b34.jar hk2-utils-2.4.0-b34.jar hppc-0.7.2.jar htrace-core4-4.1.0-incubating.jar httpclient-4.5.6.jar httpcore-4.4.10.jar ivy-2.4.0.jar jackson-annotations-2.9.6.jar jackson-core-2.9.6.jar jackson-core-asl-1.9.13.jar jackson-databind-2.9.6.jar jackson-jaxrs-base-2.9.5.jar jackson-jaxrs-json-provider-2.9.5.jar jackson-mapper-asl-1.9.13.jar jackson-module-jaxb-annotations-2.9.6.jar jackson-module-paranamer-2.9.6.jar jackson-module-scala_2.12-2.9.6.jar janino-3.0.11.jar javassist-3.18.1-GA.jar javax.annotation-api-1.2.jar javax.inject-2.4.0-b34.jar javax.servlet-api-3.1.0.jar javax.ws.rs-api-2.0.1.jar jaxb-api-2.2.11.jar jcip-annotations-1.0-1.jar jcl-over-slf4j-1.7.16.jar jersey-client-2.22.2.jar jersey-common-2.22.2.jar jersey-container-servlet-2.22.2.jar jersey-container-servlet-core-2.22.2.jar jersey-guava-2.22.2.jar jersey-media-jaxb-2.22.2.jar jersey-server-2.22.2.jar jetty-webapp-9.3.24.v20180605.jar jetty-xml-9.3.24.v20180605.jar jline-2.14.6.jar joda-time-2.9.3.jar json-1.8.jar json-smart-2.3.jar json4s-ast_2.12-3.5.3.jar json4s-core_2.12-3.5.3.jar json4s-jackson_2.12-3.5.3.jar json4s-scalap_2.12-3.5.3.jar jsp-api-2.1.jar jsr305-3.0.0.jar jtransforms-2.4.0.jar jul-to-slf4j-1.7.16.jar kerb-admin-1.0.1.jar kerb-client-1.0.1.jar kerb-common-1.0.1.jar kerb-core-1.0.1.jar kerb-crypto-1.0.1.jar kerb-identity-1.0.1.jar kerb-server-1.0.1.jar kerb-simplekdc-1.0.1.jar kerb-util-1.0.1.jar kerby-asn1-1.0.1.jar kerby-config-1.0.1.jar kerby-pkix-1.0.1.jar kerby-util-1.0.1.jar kerby-xdr-1.0.1.jar kryo-shaded-4.0.2.jar leveldbjni-all-1.8.jar log4j-1.2.17.jar lz4-java-1.5.0.jar machinist_2.12-0.6.1.jar macro-compat_2.12-1.1.1.jar metrics-core-3.1.5.jar metrics-graphite-3.1.5.jar metrics-json-3.1.5.jar metrics-jvm-3.1.5.jar minlog-1.3.0.jar netty-3.9.9.Final.jar netty-all-4.1.30.Final.jar nimbus-jose-jwt-4.41.1.jar objenesis-2.5.1.jar okhttp-2.7.5.jar okio-1.6.0.jar opencsv-2.3.jar orc-core-1.5.4-nohive.jar orc-core-1.5.4.jar orc-mapreduce-1.5.4-nohive.jar orc-shims-1.5.4.jar oro-2.0.8.jar osgi-resource-locator-1.0.1.jar paranamer-2.8.jar parquet-column-1.10.1.jar parquet-common-1.10.1.jar parquet-encoding-1.10.1.jar parquet-format-2.4.0.jar parquet-hadoop-1.10.1.jar parquet-jackson-1.10.1.jar protobuf-java-2.5.0.jar py4j-0.10.8.1.jar pyrolite-4.13.jar re2j-1.1.jar scala-compiler-2.12.8.jar scala-library-2.12.8.jar scala-parser-combinators_2.12-1.1.0.jar scala-reflect-2.12.8.jar scala-xml_2.12-1.0.5.jar shapeless_2.12-2.3.2.jar slf4j-api-1.7.16.jar slf4j-log4j12-1.7.16.jar snappy-java-1.1.7.1.jar spark-catalyst_2.12-CARMEL-363.jar spark-core_2.12-CARMEL-363.jar spark-graphx_2.12-CARMEL-363.jar spark-kvstore_2.12-CARMEL-363.jar spark-launcher_2.12-CARMEL-363.jar spark-mllib-local_2.12-CARMEL-363.jar spark-mllib_2.12-CARMEL-363.jar spark-network-common_2.12-CARMEL-363.jar spark-network-shuffle_2.12-CARMEL-363.jar spark-repl_2.12-CARMEL-363.jar spark-sketch_2.12-CARMEL-363.jar spark-sql_2.12-CARMEL-363.jar spark-streaming_2.12-CARMEL-363.jar spark-tags_2.12-CARMEL-363.jar spark-unsafe_2.12-CARMEL-363.jar spire-macros_2.12-0.13.0.jar spire_2.12-0.13.0.jar stax2-api-3.1.4.jar stream-2.7.0.jar token-provider-1.0.1.jar univocity-parsers-2.7.3.jar validation-api-1.1.0.Final.jar woodstox-core-5.0.3.jar xbean-asm7-shaded-4.12.jar xz-1.5.jar zookeeper-3.4.13.jar zstd-jni-1.3.2-2.jar
But if remove -pl assembly and navigator to org.apache.spark:spark-assembly_2.12.
build/mvn -Phadoop-3.1 dependency:build-classpath
The classpath is:
RoaringBitmap-0.5.11.jar accessors-smart-1.2.jar activation-1.1.1.jar aircompressor-0.10.jar antlr4-runtime-4.7.1.jar aopalliance-repackaged-2.4.0-b34.jar arpack_combined_all-0.1.jar arrow-format-0.12.0.jar arrow-memory-0.12.0.jar arrow-vector-0.12.0.jar audience-annotations-0.5.0.jar avro-1.8.2.jar avro-ipc-1.8.2.jar avro-mapred-1.8.2-hadoop2.jar breeze-macros_2.12-0.13.2.jar breeze_2.12-0.13.2.jar chill-java-0.9.3.jar chill_2.12-0.9.3.jar commons-beanutils-1.9.3.jar commons-cli-1.2.jar commons-codec-1.10.jar commons-collections-3.2.2.jar commons-compiler-3.0.11.jar commons-compress-1.9.jar commons-configuration2-2.1.1.jar commons-crypto-1.0.0.jar commons-io-2.4.jar commons-lang-2.6.jar commons-lang3-3.8.1.jar commons-math3-3.4.1.jar commons-net-3.1.jar commons-text-1.6.jar compress-lzf-1.0.3.jar core-1.1.2.jar curator-client-2.13.0.jar curator-framework-2.13.0.jar curator-recipes-2.13.0.jar dnsjava-2.1.7.jar flatbuffers-java-1.9.0.jar gson-2.2.4.jar guava-14.0.1.jar hadoop-annotations-3.2.0.jar hadoop-auth-3.2.0.jar hadoop-client-3.2.0.jar hadoop-common-3.2.0.jar hadoop-hdfs-client-3.2.0.jar hadoop-mapreduce-client-common-3.2.0.jar hadoop-mapreduce-client-core-3.2.0.jar hadoop-mapreduce-client-jobclient-3.2.0.jar hadoop-yarn-api-3.2.0.jar hadoop-yarn-client-3.2.0.jar hadoop-yarn-common-3.2.0.jar hive-common-2.3.4.jar hive-llap-client-2.3.4.jar hive-llap-common-2.3.4.jar hive-serde-2.3.4.jar hive-service-rpc-2.3.4.jar hive-shims-0.23-2.3.4.jar hive-shims-2.3.4.jar hive-shims-common-2.3.4.jar hive-shims-scheduler-2.3.4.jar hive-storage-api-2.6.0.jar hk2-api-2.4.0-b34.jar hk2-locator-2.4.0-b34.jar hk2-utils-2.4.0-b34.jar hppc-0.7.2.jar htrace-core4-4.1.0-incubating.jar httpclient-4.5.6.jar httpcore-4.4.10.jar ivy-2.4.0.jar jackson-annotations-2.9.6.jar jackson-core-2.9.6.jar jackson-core-asl-1.9.13.jar jackson-databind-2.9.6.jar jackson-jaxrs-base-2.9.5.jar jackson-jaxrs-json-provider-2.9.5.jar jackson-mapper-asl-1.9.13.jar jackson-module-jaxb-annotations-2.9.6.jar jackson-module-paranamer-2.9.6.jar jackson-module-scala_2.12-2.9.6.jar janino-3.0.11.jar javassist-3.18.1-GA.jar javax.annotation-api-1.2.jar javax.inject-2.4.0-b34.jar javax.servlet-api-3.1.0.jar javax.ws.rs-api-2.0.1.jar jaxb-api-2.2.11.jar jcip-annotations-1.0-1.jar jcl-over-slf4j-1.7.16.jar jersey-client-2.22.2.jar jersey-common-2.22.2.jar jersey-container-servlet-2.22.2.jar jersey-container-servlet-core-2.22.2.jar jersey-guava-2.22.2.jar jersey-media-jaxb-2.22.2.jar jersey-server-2.22.2.jar jetty-webapp-9.3.24.v20180605.jar jetty-xml-9.3.24.v20180605.jar jline-2.14.6.jar joda-time-2.9.3.jar json-1.8.jar json-smart-2.3.jar json4s-ast_2.12-3.5.3.jar json4s-core_2.12-3.5.3.jar json4s-jackson_2.12-3.5.3.jar json4s-scalap_2.12-3.5.3.jar jsp-api-2.1.jar jsr305-3.0.0.jar jtransforms-2.4.0.jar jul-to-slf4j-1.7.16.jar kerb-admin-1.0.1.jar kerb-client-1.0.1.jar kerb-common-1.0.1.jar kerb-core-1.0.1.jar kerb-crypto-1.0.1.jar kerb-identity-1.0.1.jar kerb-server-1.0.1.jar kerb-simplekdc-1.0.1.jar kerb-util-1.0.1.jar kerby-asn1-1.0.1.jar kerby-config-1.0.1.jar kerby-pkix-1.0.1.jar kerby-util-1.0.1.jar kerby-xdr-1.0.1.jar kryo-shaded-4.0.2.jar leveldbjni-all-1.8.jar log4j-1.2.17.jar lz4-java-1.5.0.jar machinist_2.12-0.6.1.jar macro-compat_2.12-1.1.1.jar metrics-core-3.1.5.jar metrics-graphite-3.1.5.jar metrics-json-3.1.5.jar metrics-jvm-3.1.5.jar minlog-1.3.0.jar netty-3.9.9.Final.jar netty-all-4.1.30.Final.jar nimbus-jose-jwt-4.41.1.jar objenesis-2.5.1.jar okhttp-2.7.5.jar okio-1.6.0.jar opencsv-2.3.jar orc-core-1.5.4.jar orc-mapreduce-1.5.4.jar orc-shims-1.5.4.jar oro-2.0.8.jar osgi-resource-locator-1.0.1.jar paranamer-2.8.jar parquet-column-1.10.1.jar parquet-common-1.10.1.jar parquet-encoding-1.10.1.jar parquet-format-2.4.0.jar parquet-hadoop-1.10.1.jar parquet-jackson-1.10.1.jar protobuf-java-2.5.0.jar py4j-0.10.8.1.jar pyrolite-4.13.jar re2j-1.1.jar scala-compiler-2.12.8.jar scala-library-2.12.8.jar scala-parser-combinators_2.12-1.1.0.jar scala-reflect-2.12.8.jar scala-xml_2.12-1.0.5.jar shapeless_2.12-2.3.2.jar slf4j-api-1.7.16.jar slf4j-log4j12-1.7.16.jar snappy-java-1.1.7.1.jar spark-catalyst_2.12-CARMEL-363.jar spark-core_2.12-CARMEL-363.jar spark-graphx_2.12-CARMEL-363.jar spark-kvstore_2.12-CARMEL-363.jar spark-launcher_2.12-CARMEL-363.jar spark-mllib-local_2.12-CARMEL-363.jar spark-mllib_2.12-CARMEL-363.jar spark-network-common_2.12-CARMEL-363.jar spark-network-shuffle_2.12-CARMEL-363.jar spark-repl_2.12-CARMEL-363.jar spark-sketch_2.12-CARMEL-363.jar spark-sql_2.12-CARMEL-363.jar spark-streaming_2.12-CARMEL-363.jar spark-tags_2.12-CARMEL-363.jar spark-unsafe_2.12-CARMEL-363.jar spire-macros_2.12-0.13.0.jar spire_2.12-0.13.0.jar stax2-api-3.1.4.jar stream-2.7.0.jar token-provider-1.0.1.jar univocity-parsers-2.7.3.jar validation-api-1.1.0.Final.jar woodstox-core-5.0.3.jar xbean-asm7-shaded-4.12.jar xz-1.5.jar zookeeper-3.4.13.jar zstd-jni-1.3.2-2.jar