Uploaded image for project: 'Apache Storm'
  1. Apache Storm
  2. STORM-1870

Allow FluxShellBolt/Spout set custom "componentConfig" via yaml.

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.0.0, 1.1.0
    • Component/s: Flux
    • Labels:
      None

      Description

      FluxShellBolt/Spout should have option to provide custom config option when importing topology from YAML file.

      We use this to provide custom configuration for example to our python RabbitMQ bolts. (Passing strings and list of strings).

      We are using this code:

      FluxShellBolt.java
      public class FluxShellBolt extends ShellBolt implements IRichBolt {
          
          //... (rest of the class)
      
          public void addComponentConfig(String key, Object value) {
              this.componentConfig.put(key, value);
          }
      
          public void addComponentConfig(String key, Object[] value) {
              this.componentConfig.put(key, value);
          }
      }
      

      And our YAML file:

      topology.yaml
      bolts:
      - className: org.apache.storm.flux.wrappers.bolts.FluxShellBolt
        configMethods:
        - name: addComponentConfig
          args: [rabbitmq.configfile, etc/rabbit.yml]
        - name: addComponentConfig
          args:
          - publisher.data_paths
          - [actions]
          
      ... (rest of yaml file)
      

      It works fine, however it produces this type of warning:

      WARN  o.a.s.f.FluxBuilder - Found multiple invokable methods for class class org.apache.storm.flux.wrappers.bolts.FluxShellBolt, method addComponentConfig, given arguments [publisher.data_paths, [actions]]. Using the last one found.
      

      Which fortunately happens to be correct method, but it is not correct solution.

      Any ideas?

      It is quite needed to provide custom config to ShellSpout/Bolt, since we run all spouts/bolts in python via this option. It would be nice to have this option in official release.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                kabhwan Jungtaek Lim
                Reporter:
                darkless Pavel Grochal
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 40m
                  40m