Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-38182

Fix NoSuchElementException if pushed filter does not contain any references

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.3.0
    • 3.3.0
    • SQL
    • None

    Description

      reproduce:

      CREATE TABLE t (c1 int) USING PARQUET;
      
      SET spark.sql.optimizer.excludedRules=org.apache.spark.sql.catalyst.optimizer.BooleanSimplification;
      
      SELECT * FROM t WHERE c1 = 1 AND 2 > 1;
      

      and the error msg:

      java.util.NoSuchElementException: next on empty iterator
      	at scala.collection.Iterator$$anon$2.next(Iterator.scala:41)
      	at scala.collection.Iterator$$anon$2.next(Iterator.scala:39)
      	at scala.collection.mutable.LinkedHashSet$$anon$1.next(LinkedHashSet.scala:89)
      	at scala.collection.IterableLike.head(IterableLike.scala:109)
      	at scala.collection.IterableLike.head$(IterableLike.scala:108)
      	at org.apache.spark.sql.catalyst.expressions.AttributeSet.head(AttributeSet.scala:69)
      	at org.apache.spark.sql.execution.datasources.PartitioningAwareFileIndex.$anonfun$listFiles$3(PartitioningAwareFileIndex.scala:85)
      	at scala.Option.map(Option.scala:230)
      	at org.apache.spark.sql.execution.datasources.PartitioningAwareFileIndex.listFiles(PartitioningAwareFileIndex.scala:84)
      	at org.apache.spark.sql.execution.FileSourceScanExec.selectedPartitions$lzycompute(DataSourceScanExec.scala:249)
      

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            ulysses XiDuo You
            ulysses XiDuo You
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment