[IMPALA-6941] Allow loading more text scanner plugins

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: Impala 2.13.0, Impala 3.1.0
• Component/s: Backend
• Target Version: Impala 3.1.0
• Epic Color: ghx-label-1
• Assignee: Tim Armstrong
• Reporter: Tim Armstrong
• Created: 26/Apr/18 22:34
• Updated: 14/May/19 15:29
• Resolved: 18/May/18 17:56

Description

It would be nice if Impala supported loading plugins for scanning additional text formats aside from LZO - the current logic is fairly specialized but could easily be extended to load libraries for codecs like LZ4 and ZSTD if available. It's kind of weird that we only support that one format.

This might help a bit with IMPALA-6941 and IMPALA-3898 since we could test the plugin-loading mechanism without relying on the external Impala-lzo codebase.

Issue Links

• Blocked: IMPALA-8408 Failure to load metadata for .deflate compressed text files
• breaks: IMPALA-7068 Failed test: metadata.test_partition_metadata.TestPartitionMetadataUncompressedTextOnly.test_unsupported_text_compression on S3
• incorporates: IMPALA-7167 Catalogd should reject files with unsupported formats
• is depended upon by: IMPALA-6963 Consider allowing extension of the builtin list of compression types
• relates to: IMPALA-8549 Add support for scanning DEFLATE text files