DelimitedTextFile directly parses line delimited text files and parses each line into CSV or TSV field. It has many limits when we deal with custom text-based file format.
This patch enables DelimitedTextFile to use a pluggable line (de) serializer.
First of all, I add an abstract class for user-defined line serde class as follows:
I also added a table property text.serde.class which allows users to specify a custom line serder. This table property affects only TEXT file format. You can specify your own line serder as follows: