Major Description
There are two types or Metadata in Spark
- Metadata that is constant per file (file_name, file_size, ...)
- Metadata that is not contant (currently only row_index)
The two types are generated differently
- File constant metadata is appended to the output after scan
- non-constant metadata is generated during the scan
The proposal here is to create different metadata attributes to distinguish those different types throughout the code
