Major Description
in https://github.com/apache/spark/pull/10747, we decided to use expressions instead of encoders in query plans, to make the dataset easier to optimize and make encoder resolution less hacky.
However, that PR didn't finish the migration completely and cause some regressions which are hidden due to the lack of end-to-end test.
We should fix it by completing the migration and add end-to-end test for it
