Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-2066

Better error message for non-aggregated attributes with aggregates

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 1.0.0
    • Fix Version/s: 1.2.0
    • Component/s: SQL
    • Labels:
      None
    • Target Version/s:

      Description

      Michael Armbrust

      Run the following query

      scala> c.hql("select key, count(*) from src").collect()
      

      Got the following exception at runtime

      org.apache.spark.sql.catalyst.errors.package$TreeNodeException: No function to evaluate expression. type: AttributeReference, tree: key#61
      	at org.apache.spark.sql.catalyst.expressions.AttributeReference.eval(namedExpressions.scala:157)
      	at org.apache.spark.sql.catalyst.expressions.Projection.apply(Projection.scala:35)
      	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$1.apply(Aggregate.scala:154)
      	at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$1.apply(Aggregate.scala:134)
      	at org.apache.spark.rdd.RDD$$anonfun$12.apply(RDD.scala:558)
      	at org.apache.spark.rdd.RDD$$anonfun$12.apply(RDD.scala:558)
      	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
      	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:261)
      	at org.apache.spark.rdd.RDD.iterator(RDD.scala:228)
      	at org.apache.spark.rdd.MappedRDD.compute(MappedRDD.scala:31)
      	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:261)
      	at org.apache.spark.rdd.RDD.iterator(RDD.scala:228)
      	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:111)
      	at org.apache.spark.scheduler.Task.run(Task.scala:51)
      	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:187)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:744)
      

      This should either fail in analysis time, or pass at runtime. Definitely shouldn't fail at runtime.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                lian cheng Cheng Lian
                Reporter:
                rxin Reynold Xin
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: