Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-25727

makeCopy failed in InMemoryRelation

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.4.0
    • Fix Version/s: 2.4.0
    • Component/s: SQL
    • Labels:
      None

      Description

          val data = Seq(100).toDF("count").cache()
          data.queryExecution.optimizedPlan.toJSON
      

      The above code can generate the following error:

      assertion failed: InMemoryRelation fields: output, cacheBuilder, statsOfPlanToCache, outputOrdering, values: List(count#178), CachedRDDBuilder(true,10000,StorageLevel(disk, memory, deserialized, 1 replicas),*(1) Project [value#176 AS count#178]
      +- LocalTableScan [value#176]
      ,None), Statistics(sizeInBytes=12.0 B, hints=none)
      java.lang.AssertionError: assertion failed: InMemoryRelation fields: output, cacheBuilder, statsOfPlanToCache, outputOrdering, values: List(count#178), CachedRDDBuilder(true,10000,StorageLevel(disk, memory, deserialized, 1 replicas),*(1) Project [value#176 AS count#178]
      +- LocalTableScan [value#176]
      ,None), Statistics(sizeInBytes=12.0 B, hints=none)
      	at scala.Predef$.assert(Predef.scala:170)
      	at org.apache.spark.sql.catalyst.trees.TreeNode.jsonFields(TreeNode.scala:611)
      	at org.apache.spark.sql.catalyst.trees.TreeNode.org$apache$spark$sql$catalyst$trees$TreeNode$$collectJsonValue$1(TreeNode.scala:599)
      	at org.apache.spark.sql.catalyst.trees.TreeNode.jsonValue(TreeNode.scala:604)
      	at org.apache.spark.sql.catalyst.trees.TreeNode.toJSON(TreeNode.scala:590)
      

        Attachments

          Activity

            People

            • Assignee:
              smilegator Xiao Li
              Reporter:
              smilegator Xiao Li
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: