[STORM-1910] One topology can't use hdfs spout to read from two locations

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.0.1
• Fix Version/s: 2.0.0, 1.1.0
• Component/s: storm-hdfs
• Epic Link: Release Apache Storm 1.1.0
• Assignee: Roshan Naik
• Reporter: Raghav Kumar Gautam
• Created: 16/Jun/16 22:06
• Updated: 14/Jul/16 23:03
• Resolved: 14/Jul/16 21:10

Description

The hdfs uri is passed using config:

    conf.put(Configs.HDFS_URI, hdfsUri);

I see two problems with this approach:
1. If someone wants to used two hdfsUri in same or different spouts - then that does not seem feasible.
https://github.com/apache/storm/blob/d17b3b9c3cbc89d854bfb436d213d11cfd4545ec/examples/storm-starter/src/jvm/storm/starter/HdfsSpoutTopology.java#L117-L117
https://github.com/apache/storm/blob/d17b3b9c3cbc89d854bfb436d213d11cfd4545ec/external/storm-hdfs/src/main/java/org/apache/storm/hdfs/spout/HdfsSpout.java#L331-L331

    if ( !conf.containsKey(Configs.SOURCE_DIR) ) {
      LOG.error(Configs.SOURCE_DIR + " setting is required");
      throw new RuntimeException(Configs.SOURCE_DIR + " setting is required");
    }
    this.sourceDirPath = new Path( conf.get(Configs.SOURCE_DIR).toString() );

2. It does not fail fast i.e. at the time of topology submissing. We can fail fast if the hdfs path is invalid or credentials/permissions are not ok. Such errors at this time can only be detected at runtime by looking at the worker logs.
https://github.com/apache/storm/blob/d17b3b9c3cbc89d854bfb436d213d11cfd4545ec/external/storm-hdfs/src/main/java/org/apache/storm/hdfs/spout/HdfsSpout.java#L297-L297

Issue Links

• links to: GitHub Pull Request #1558