Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-26063

CatalystDataToAvro gives "UnresolvedException: Invalid call to dataType on unresolved object" when requested for numberedTreeString

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: 2.4.0
    • Fix Version/s: None
    • Component/s: SQL
    • Labels:
      None

      Description

      The following gives org.apache.spark.sql.catalyst.analysis.UnresolvedException: Invalid call to dataType on unresolved object, tree: 'id:

      // ./bin/spark-shell --packages org.apache.spark:spark-avro_2.12:2.4.0
      scala> spark.version
      res0: String = 2.4.0
      
      import org.apache.spark.sql.avro._
      val q = spark.range(1).withColumn("to_avro_id", to_avro('id))
      val logicalPlan = q.queryExecution.logical
      
      scala> logicalPlan.expressions.drop(1).head.numberedTreeString
      org.apache.spark.sql.catalyst.analysis.UnresolvedException: Invalid call to dataType on unresolved object, tree: 'id
      at org.apache.spark.sql.catalyst.analysis.UnresolvedAttribute.dataType(unresolved.scala:105)
      at org.apache.spark.sql.avro.CatalystDataToAvro.simpleString(CatalystDataToAvro.scala:56)
      at org.apache.spark.sql.catalyst.expressions.Expression.verboseString(Expression.scala:233)
      at org.apache.spark.sql.catalyst.trees.TreeNode.generateTreeString(TreeNode.scala:548)
      at org.apache.spark.sql.catalyst.trees.TreeNode.generateTreeString(TreeNode.scala:569)
      at org.apache.spark.sql.catalyst.trees.TreeNode.treeString(TreeNode.scala:472)
      at org.apache.spark.sql.catalyst.trees.TreeNode.treeString(TreeNode.scala:469)
      at org.apache.spark.sql.catalyst.trees.TreeNode.numberedTreeString(TreeNode.scala:483)
      ... 51 elided

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                jlaskowski Jacek Laskowski
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: