Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
Description
Sometimes one Substrait function will map to multiple Arrow functions. For example, the Substrait add function might be referring to Arrow's add or add_checked. We need to figure out how to register this correctly (e.g. one possible approach would be a substrait_add meta function).
Other times a substrait function will encode something Arrow considers an "option" as a function argument. For example, the is_in Arrow function is unary with an option for the lookup set. The substrait function is binary but the second argument must be constant and be the lookup set. Neither of which is to be confused with a truly binary is_in function which takes in a different set at every row.
It's possible there is no work to do here other than adding a bunch of substrait_ meta functions in Arrow. In that case all the work will be done in other JIRAs. Or, it is possible that there is some kind of extension we can make to the function registry that bypasses the need for the meta functions. I'm leaving this JIRA open so future contributors can consider this second option.
