Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-34757

Spark submit should ignore cache for SNAPSHOT dependencies

    XMLWordPrintableJSON

Details

    • Bug
    • Status: In Progress
    • Minor
    • Resolution: Unresolved
    • 3.1.1
    • None
    • Deploy, Spark Core
    • None

    Description

      When spark-submit is executed with --packages, it will not download the dependency jars when they are available in cache (e.g. ivy cache), even when the dependencies are SNAPSHOTs.

      This might block developers who work on external modules in Spark (e.g. spark-avro), since they need to remove the cache manually every time when they update the code during developments (which generates SNAPSHOT jars). Without knowing this, they could be blocked wondering why their code changes are not reflected in spark-submit executions.

      Attachments

        Activity

          People

            apachespark Apache Spark
            bozhang Bo Zhang
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated: