Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-41151

Keep built-in file _metadata column nullable value consistent

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.3.0, 3.3.1, 3.3.2
    • 3.3.2, 3.4.0
    • SQL
    • None

    Description

      In FileSourceStrategy, we add an Alias node to wrap the file metadata fields (e.g. file_name, file_size) in a NamedStruct (here). But `CreateNamedStruct` has an override `nullable` value `false` (here), which is different from the `_metadata` struct `nullable` value `true` (here). 

       

      We should keep the nullable value the same, otherwise, the downstream optimization rules might use the nullability here and cause unexpected behaviors.

      Attachments

        Activity

          People

            yaohua Yaohua Zhao
            yaohua Yaohua Zhao
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: