Uploaded image for project: 'Apache Drill'
  1. Apache Drill
  2. DRILL-6673

Drill table functions cannot use plugin configs with a constructor

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: 1.13.0
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Consider an example format plugin, such as the regex one used in the Drill book. (GitHub reference needed.) Format configurations should be immutable. So, we make the members private final and define a constructor like this:

      public class RegexFormatConfig implements FormatPluginConfig {
      
        private final String regex;
      
        // Should be a List<String>. But, table functions don't support
        // lists, so we store the fields as single string that contains
        // a comma-delimited list: a, b, c. Spaces are optional.
      
        private final String fields;
        private final String extension;
      
        public RegexFormatConfig(
            @JsonProperty("extension") String extension,
            @JsonProperty("regex") String regex,
            @JsonProperty("fields") String fields
            ) {
          this.regex = regex;
          this.fields = fields;
          this.extension = extension;
        }
      

      We can then create a plugin configuration using the Drill Web console, the bootstrap-storage-plugins.json and so on. All work fine.

      Suppose we try to define a configuration using a Drill table function:

            final String sql = "SELECT * FROM table(cp.`regex/simple.log2`\n" +
                "(type => 'regex',\n" +
                " extension => 'log2',\n" +
                " regex => '(\\\\d\\\\d\\\\d\\\\d)-(\\\\d\\\\d)-(\\\\d\\\\d) .*',\n" +
                " fields => 'a, b, c, d'))";
      

      We get this error:

      org.apache.drill.common.exceptions.UserRemoteException: PARSE ERROR: 
      configuration for format of type regex can not be created 
      (class: org.apache.drill.exec.store.easy.regex.RegexFormatConfig)
      
      table regex/simple.log2
      
      

      They fail because the code that resolves the plugin configuration does not know about the Jackson constructor conventions: it requires an default constructor:

      package org.apache.drill.exec.store.dfs;
      ...
      final class FormatPluginOptionsDescriptor {
        ...
        FormatPluginConfig createConfigForTable(TableInstance t) {
          ...
          try {
            config = pluginConfigClass.newInstance();
          } catch (InstantiationException | IllegalAccessException e) {
            throw UserException.parseError(e)
                .message(
                    "configuration for format of type %s can not be created (class: %s)",
                    this.typeName, pluginConfigClass.getName())
               ...
      

      The workaround would seem to be to define setters using Jackson standards. But, that does not work either due to DRILL-6672 which says that Drill table functions don't allow them. The only real workaround is to make the fields public and omit the constructor.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                paul-rogers Paul Rogers
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated: