Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-26299

Shaded test jar in spark-streaming cause deploy test jar twice

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 2.3.2, 2.4.0
    • Fix Version/s: None
    • Component/s: Deploy
    • Labels:
      None

      Description

      In spark root pom, there is a test-jar execution for all sub-projects including spark-streaming. This will attach an artifact: org.apache.spark:spark-streaming_2.11:test-jar:tests:2.3.2

      Also, in streaming pom, there is a shade test configuration, it will attach an artifact: org.apache.spark:spark-streaming_2.11:jar:tests:2.3.2

      But two artifacts actually point to the same file: spark-streaming_2.11-2.3.2-tests.jar

      So, when deploy spark to nexus using mvn deploy, maven will upload the test.jar twice since it belongs to 2 artifacts. Then deploy fails due to nexus don't allow overrides existing file for non-SNAPSHOT release.

      What's more, after checking the spark-streaming, shaded test jar is exactly same to original test jar. It seems we can just delete the related shade config.

      <build>
        <outputDirectory>target/scala-${scala.binary.version}/classes</outputDirectory>
        <testOutputDirectory>target/scala-${scala.binary.version}/test-classes</testOutputDirectory>
        <plugins>
          <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-shade-plugin</artifactId>
            <configuration>
              <shadeTestJar>true</shadeTestJar>
            </configuration>
          </plugin>
        </plugins>
      </build>

       

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              liulinhong Liu, Linhong
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: