[SPARK-26128] filter breaks input_file_name

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Cannot Reproduce
• Affects Version/s: 2.3.2
• Component/s: Spark Shell
• Assignee: Unassigned
• Reporter: Paul Praet
• Created: 20/Nov/18 13:01
• Updated: 12/Dec/22 18:11
• Resolved: 02/Mar/19 20:10

Description

This works:

scala> spark.read.parquet("/tmp/newparquet").select(input_file_name).show(5,false)
+-----------------------------------------------------------------------------------------------------------------------------------------------------+
|input_file_name()                                                                                                                                    |
+-----------------------------------------------------------------------------------------------------------------------------------------------------+
|file:///tmp/newparquet/parquet-5-PT6H/junit/data/tenant=NA/year=2017/month=201704/day=20170406/hour=2017040618/data.eu-west-1b.290.PT6H.FINAL.parquet|
|file:///tmp/newparquet/parquet-5-PT6H/junit/data/tenant=NA/year=2017/month=201704/day=20170406/hour=2017040618/data.eu-west-1b.290.PT6H.FINAL.parquet|
|file:///tmp/newparquet/parquet-5-PT6H/junit/data/tenant=NA/year=2017/month=201704/day=20170406/hour=2017040618/data.eu-west-1b.290.PT6H.FINAL.parquet|
|file:///tmp/newparquet/parquet-5-PT6H/junit/data/tenant=NA/year=2017/month=201704/day=20170406/hour=2017040618/data.eu-west-1b.290.PT6H.FINAL.parquet|
|file:///tmp/newparquet/parquet-5-PT6H/junit/data/tenant=NA/year=2017/month=201704/day=20170406/hour=2017040618/data.eu-west-1b.290.PT6H.FINAL.parquet|
+-----------------------------------------------------------------------------------------------------------------------------------------------------+

When adding a filter:

scala> spark.read.parquet("/tmp/newparquet").where("key.station='XYZ'").select(input_file_name()).show(5,false)
+-----------------+
|input_file_name()|
+-----------------+
| |
| |
| |
| |
| |
+-----------------+