[FLINK-19903] Allow to read metadata in filesystem connector

• Type: New Feature
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 1.15.0
• Component/s: Connectors / FileSystem, Table SQL / Ecosystem
• Labels: auto-deprioritized-major pull-request-available
• Assignee: Francesco Guardiani
• Reporter: Ruben Laguna
• Created: 30/Oct/20 13:00
• Updated: 19/Nov/21 08:37
• Resolved: 19/Nov/21 08:37

Description

Use case: 

I have a dataset where they embedded some information in the filenames
(200k files) and I need to extract that as a new column.

In Spark I could `
.withColumn("id",f.split(f.reverse(f.split(f.input_file_name(),'/'))[0],'\.')[0])`
 but I don't see how can I do the same with Flink.

 

Apparently there is FLIP-107 which would allow SQL connectors and formats to expose metadata. 

 

So it would be great for the Filesystem SQL connector to expose the path. 

Ideally for me the path could be exposed via a function that read the metadata. So I could write  something akin to `SELECT input_file_name(),* FROM table1`

 

 

[1]: https://cwiki.apache.org/confluence/display/FLINK/FLIP-107%3A+Handling+of+metadata+in+SQL+connectors

[2]: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Can-I-get-the-filename-as-a-column-td39096.html

Issue Links

• links to: GitHub Pull Request #13954

Sub-Tasks

• Add infrastructure to support metadata in filesystem connector: Closed
• Expose all metadata in FileStatus: Closed
• Support partition keys through metadata: Resolved