Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-11032

Failure to resolve having correctly

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 1.5.1, 1.6.0
    • Fix Version/s: 1.5.2, 1.6.0
    • Component/s: SQL
    • Labels:
      None

      Description

      This is a regression from Spark 1.4

      Seq(("michael", 30)).toDF("name", "age").registerTempTable("people")
      sql("SELECT MIN(t0.age) FROM (SELECT * FROM PEOPLE WHERE age > 0) t0 HAVING(COUNT(1) > 0)").explain(true)
      == Parsed Logical Plan ==
      'Filter cast(('COUNT(1) > 0) as boolean)
       'Project [unresolvedalias('MIN('t0.age))]
        'Subquery t0
         'Project [unresolvedalias(*)]
          'Filter ('age > 0)
           'UnresolvedRelation [PEOPLE], None
      
      == Analyzed Logical Plan ==
      _c0: int
      Filter cast((count(1) > cast(0 as bigint)) as boolean)
       Aggregate [min(age#6) AS _c0#9]
        Subquery t0
         Project [name#5,age#6]
          Filter (age#6 > 0)
           Subquery people
            Project [_1#3 AS name#5,_2#4 AS age#6]
             LocalRelation [_1#3,_2#4], [[michael,30]]
      
      == Optimized Logical Plan ==
      Filter (count(1) > 0)
       Aggregate [min(age#6) AS _c0#9]
        Project [_2#4 AS age#6]
         Filter (_2#4 > 0)
          LocalRelation [_1#3,_2#4], [[michael,30]]
      
      == Physical Plan ==
      Filter (count(1) > 0)
       TungstenAggregate(key=[], functions=[(min(age#6),mode=Final,isDistinct=false)], output=[_c0#9])
        TungstenExchange SinglePartition
         TungstenAggregate(key=[], functions=[(min(age#6),mode=Partial,isDistinct=false)], output=[min#12])
          TungstenProject [_2#4 AS age#6]
           Filter (_2#4 > 0)
            LocalTableScan [_1#3,_2#4], [[michael,30]]
      
      Code Generation: true
      
      Caused by: java.lang.UnsupportedOperationException: Cannot evaluate expression: count(1)
      	at org.apache.spark.sql.catalyst.expressions.Unevaluable$class.eval(Expression.scala:188)
      	at org.apache.spark.sql.catalyst.expressions.Count.eval(aggregates.scala:156)
      	at org.apache.spark.sql.catalyst.expressions.BinaryExpression.eval(Expression.scala:327)
      	at org.apache.spark.sql.catalyst.expressions.InterpretedPredicate$$anonfun$create$2.apply(predicates.scala:38)
      	at org.apache.spark.sql.catalyst.expressions.InterpretedPredicate$$anonfun$create$2.apply(predicates.scala:38)
      	at org.apache.spark.sql.execution.Filter$$anonfun$4$$anonfun$apply$4.apply(basicOperators.scala:117)
      	at org.apache.spark.sql.execution.Filter$$anonfun$4$$anonfun$apply$4.apply(basicOperators.scala:115)
      	at scala.collection.Iterator$$anon$14.hasNext(Iterator.scala:390)
      	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
      

        Attachments

          Activity

            People

            • Assignee:
              cloud_fan Wenchen Fan
              Reporter:
              marmbrus Michael Armbrust
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: