Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-29329

maven incremental builds not working

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 3.0.0
    • None
    • Build
    • None

    Description

      It looks like since we Upgraded scala-maven-plugin to 4.2.0 https://issues.apache.org/jira/browse/SPARK-28759 spark incremental builds stop working.  Everytime you build its building all files, which takes forever.

      It would be nice to fix this.

       

      To reproduce, just build spark once ( I happened to be using the command below):

      build/mvn -Phadoop-3.2 -Phive-thriftserver -Phive -Pyarn -Pkinesis-asl -Pkubernetes -Pmesos -Phadoop-cloud -Pspark-ganglia-lgpl package -DskipTests

      Then build it again and you will see that it compiles all the files and takes 15-30 minutes. With incremental it skips all unnecessary files and takes closer to 5 minutes.

      Attachments

        Issue Links

          is caused by

          Sub-task - The sub-task of the issue SPARK-28759 Upgrade scala-maven-plugin to 4.2.0

          • Major - Major loss of function.
          • Resolved
          relates to

          Sub-task - The sub-task of the issue SPARK-29528 Upgrade scala-maven-plugin to 4.3.0 for Scala 2.13.1

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              Unassigned Unassigned
              tgraves Thomas Graves
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated: