Major Description
For the Spark Connect Scala Client we need Encoders. We cannot use the current ExpressionEncoder infrastructure because of the following reasons:
- Connect uses Arrow instead of internal rows
- We don't want to have a dependency on Catalyst in the client.
- We don't want to be sending the expressions that expression encoders use over the wire.
An additional challenge is that we cannot send TypeTags over the wire.
The proposal is to create an intermediate encoder. This encoder describes exactly how which external type maps to an internal type, but it does not prescribe which format is used.
An upside of this approach is that we can reduce the cost of encoder creation by quite a margin.
