Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-24895

Spark 2.4.0 Snapshot artifacts has broken metadata due to mismatched filenames

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.4.0
    • Fix Version/s: 2.4.0
    • Component/s: Build
    • Labels:
      None
    • Target Version/s:

      Description

      Spark 2.4.0 has Maven build errors because artifacts uploaded to apache maven repo has mismatched filenames:

      [ERROR] Failed to execute goal org.apache.maven.plugins:maven-enforcer-plugin:1.4.1:enforce (enforce-banned-dependencies) on project spark_2.4: Execution enforce-banned-dependencies of goal org.apache.maven.plugins:maven-enforcer-plugin:1.4.1:enforce failed: org.apache.maven.shared.dependency.graph.DependencyGraphBuilderException: Could not resolve following dependencies: [org.apache.spark:spark-mllib-local_2.11:jar:2.4.0-SNAPSHOT (compile), org.apache.spark:spark-network-shuffle_2.11:jar:2.4.0-SNAPSHOT (compile), org.apache.spark:spark-sketch_2.11:jar:2.4.0-SNAPSHOT (compile)]: Could not resolve dependencies for project com.databricks:spark_2.4:pom:1: The following artifacts could not be resolved: org.apache.spark:spark-mllib-local_2.11:jar:2.4.0-SNAPSHOT, org.apache.spark:spark-network-shuffle_2.11:jar:2.4.0-SNAPSHOT, org.apache.spark:spark-sketch_2.11:jar:2.4.0-SNAPSHOT: Could not find artifact org.apache.spark:spark-mllib-local_2.11:jar:2.4.0-20180723.232411-177 in apache-snapshots ([https://repository.apache.org/snapshots/]) -> [Help 1]
      

       

      If you check the artifact metadata you will see the pom and jar files are 2.4.0-20180723.232411-177 instead of 2.4.0-20180723.232410-177:

      <metadata modelVersion="1.1.0">
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-mllib-local_2.11</artifactId>
        <version>2.4.0-SNAPSHOT</version>
        <versioning>
          <snapshot>
            <timestamp>20180723.232411</timestamp>
            <buildNumber>177</buildNumber>
          </snapshot>
          <lastUpdated>20180723232411</lastUpdated>
          <snapshotVersions>
            <snapshotVersion>
              <extension>jar</extension>
              <value>2.4.0-20180723.232411-177</value>
              <updated>20180723232411</updated>
            </snapshotVersion>
            <snapshotVersion>
              <extension>pom</extension>
              <value>2.4.0-20180723.232411-177</value>
              <updated>20180723232411</updated>
            </snapshotVersion>
            <snapshotVersion>
              <classifier>tests</classifier>
              <extension>jar</extension>
              <value>2.4.0-20180723.232410-177</value>
              <updated>20180723232411</updated>
            </snapshotVersion>
            <snapshotVersion>
              <classifier>sources</classifier>
              <extension>jar</extension>
              <value>2.4.0-20180723.232410-177</value>
              <updated>20180723232411</updated>
            </snapshotVersion>
            <snapshotVersion>
              <classifier>test-sources</classifier>
              <extension>jar</extension>
              <value>2.4.0-20180723.232410-177</value>
              <updated>20180723232411</updated>
            </snapshotVersion>
          </snapshotVersions>
        </versioning>
      </metadata>
      

       
      This behavior is very similar to this issue: https://issues.apache.org/jira/browse/MDEPLOY-221

      Since 2.3.0 snapshots work with the same maven 3.3.9 version and maven deploy 2.8.2 plugin, it is highly possible that we introduced a new plugin that causes this. 

      The most recent addition is the spot-bugs plugin, which is known to have incompatibilities with other plugins: https://github.com/spotbugs/spotbugs-maven-plugin/issues/21

      We may want to try building without it to sanity check.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                ericfchang Eric Chang
                Reporter:
                ericfchang Eric Chang
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: