[IMPALA-10332] Add file formats to HdfsScanNode's thrift representation and codegen for those

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Implemented
• Fix Version/s: Impala 4.0.0
• Component/s: Backend, Frontend
• Epic Color: ghx-label-10
• Assignee: Daniel Becker
• Reporter: Daniel Becker
• Created: 17/Nov/20 11:58
• Updated: 24/Nov/20 16:48
• Resolved: 24/Nov/20 10:02

Description

List all file formats that a HdfsScanNode needs to process in any fragment instance. It is possible that some file formats will not be needed in all fragment instances.

This is a step towards sharing codegen between different impala backends. Using the file formats provided in the thrift file, a backend can codegen code for file formats that are not needed in its own process but are needed in other fragment instances running on other backends, and the resulting binary can be shared between multiple backends.

Codegenning for file formats will be done based on the thrift message and not on what is needed for the actual backend. This leads to some extra work in case a file format is not needed for the current backend and codegen sharing is not available (at this point it is not implemented). However, the overall number of such cases is low.

Also adding the file formats to the node's explain string.

Issue Links

• is depended upon by: IMPALA-9660 Distributed codegen