Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-10623

NoSuchElementException thrown when ORC predicate push-down is turned on

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 1.4.0, 1.4.1, 1.5.0
    • Fix Version/s: 1.5.1, 1.6.0
    • Component/s: SQL
    • Labels:
      None

      Description

      Turning on predicate pushdown for ORC datasources results in a NoSuchElementException:

      scala> val df = sqlContext.sql("SELECT name FROM people WHERE age < 15")
      df: org.apache.spark.sql.DataFrame = [name: string]
      
      scala> sqlContext.setConf("spark.sql.orc.filterPushdown", "true")
      
      scala> df.explain
      == Physical Plan ==
      java.util.NoSuchElementException
      

      Disabling the pushdown makes things work again:

      scala> sqlContext.setConf("spark.sql.orc.filterPushdown", "false")
      
      scala> df.explain
      == Physical Plan ==
      Project [name#6]
       Filter (age#7 < 15)
        Scan OrcRelation[file:/home/mydir/spark-1.5.0-SNAPSHOT/test/people][name#6,age#7]
      

        Attachments

          Activity

            People

            • Assignee:
              zhazhan Zhan Zhang
              Reporter:
              rams Ram Sriharsha
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: