[SPARK-26299] Shaded test jar in spark-streaming cause deploy test jar twice

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.3.2, 2.4.0
• Component/s: Deploy
• Assignee: Unassigned Assign to me
• Reporter: Liu, Linhong
• Created: 07/Dec/18 02:21
• Updated: 08/Dec/18 20:19

Description

In spark root pom, there is a test-jar execution for all sub-projects including spark-streaming. This will attach an artifact: org.apache.spark:spark-streaming_2.11:test-jar:tests:2.3.2

Also, in streaming pom, there is a shade test configuration, it will attach an artifact: org.apache.spark:spark-streaming_2.11:jar:tests:2.3.2

But two artifacts actually point to the same file: spark-streaming_2.11-2.3.2-tests.jar

So, when deploy spark to nexus using mvn deploy, maven will upload the test.jar twice since it belongs to 2 artifacts. Then deploy fails due to nexus don't allow overrides existing file for non-SNAPSHOT release.

What's more, after checking the spark-streaming, shaded test jar is exactly same to original test jar. It seems we can just delete the related shade config.

<build>
  <outputDirectory>target/scala-${scala.binary.version}/classes</outputDirectory>
  <testOutputDirectory>target/scala-${scala.binary.version}/test-classes</testOutputDirectory>
  <plugins>
    <plugin>
      <groupId>org.apache.maven.plugins</groupId>
      <artifactId>maven-shade-plugin</artifactId>
      <configuration>
        <shadeTestJar>true</shadeTestJar>
      </configuration>
    </plugin>
  </plugins>
</build>