Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-36835

Spark 3.2.0 POMs are no longer "dependency reduced"

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 3.2.0
    • Fix Version/s: 3.2.0
    • Component/s: Build
    • Labels:
      None

      Description

      It looks like Spark 3.2.0's POMs are no longer "dependency reduced". As a result, applications may pull in additional unnecessary dependencies when depending on Spark.

      Spark uses the Maven Shade plugin to create effective POMs and to bundle shaded versions of certain libraries with Spark (namely, Jetty, Guava, and JPPML). By default, the Maven Shade plugin generates simplified POMs which remove dependencies on artifacts that have been shaded.

      SPARK-33212 / b6f46ca29742029efea2790af7fdefbc2fcf52de changed the configuration of the Maven Shade plugin, setting createDependencyReducedPom to false.

      As a result, the generated POMs now include compile-scope dependencies on the shaded libraries. For example, compare the org.eclipse.jetty dependencies in:

      I think we should revert back to generating "dependency reduced" POMs to ensure that Spark declares a proper set of dependencies and to avoid "unknown unknown" consequences of changing our generated POM format.

      /cc Chao Sun

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                csun Chao Sun
                Reporter:
                joshrosen Josh Rosen
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: