[DRILL-6169] Table functions do not allow list arguments

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.12.0
• Assignee: Unassigned
• Reporter: Paul Rogers
• Created: 19/Feb/18 21:12
• Updated: 21/Jan/19 13:43

Description

Consider DRILL-6167 which describes a regex storage plugin. The plugin definition wants to take a list of columns. This can be expressed in the storage plugin config as a list:

@JsonTypeName("regex")
@JsonInclude(Include.NON_DEFAULT)
public class RegexFormatConfig implements FormatPluginConfig {
  ...
  public List<String> fields = new ArrayList<>();

The list can be defined in JSON using the Drill web console. All works fine.

Drill also provides table functions that allow us to specify format plugin config options at query time. Because the function is defined as SQL, there appears to be no way to express an argument which is a list.

Let's try to define the list in sql:

   SELECT * FROM table(cp.`regex/simple.log2`
      (type => 'regex',
       regex => '(\\d\\d\\d\\d)-(\\d\\d)-(\\d\\d) .*',
       fields => ('a', 'b', 'c')))

The syntax above is legal SQL (see below), but is not wired up correctly to map to a list argument. When run:

org.apache.calcite.runtime.CalciteContextException:
  From line 1, column 24 to line 4, column 27:
  No match found for function signature
    regex/simple.log2(type => <CHARACTER>,
      regex => <CHARACTER>,
      fields => <RecordType(CHAR(1) EXPR$0, CHAR(1) EXPR$1, CHAR(1) EXPR$2)>)

This error seems to say that:

• The syntax is OK. (Check parser.jj, if you work down the rules starting from TableFunctionCall, you'll see that ParenthesizedQueryOrCommaList defines the above list as valid syntax for a function argument.
• The type of the argument appears to be a RecordType.
• No rules are in place to realize that, for a plugin config table function, the RecordType can be translated to a String list.

The only workaround appears to be to redefine the string list to be a single string. In this case, rather than a list of fields, require a string with a comma-separated list of files which is parsed by the plugin at run time.

If Drill provides lists in the config, and provides table functions to set the properties in a config, then it should provide a way to set lists.

Issue Links

• Is contained by: DRILL-6986 Table function improvements / issues (UMBRELLA JIRA)