Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-24867

Add AnalysisBarrier to DataFrameWriter

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Blocker
    • Resolution: Fixed
    • 2.3.0, 2.3.1
    • 2.3.2
    • SQL
    • None

    Description

            val udf1 = udf({(x: Int, y: Int) => x + y})
            val df = spark.range(0, 3).toDF("a")
              .withColumn("b", udf1($"a", udf1($"a", lit(10))))
            df.cache()
            df.write.saveAsTable("t")
            df.write.saveAsTable("t1")
      

      Cache is not being used because the plans do not match with the cached plan. This is a regression caused by the changes we made in AnalysisBarrier, since not all the Analyzer rules are idempotent. We need to fix it to Spark 2.3.2

      Attachments

        Activity

          People

            smilegator Xiao Li
            smilegator Xiao Li
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: