Uploaded image for project: 'Apache Storm'
  1. Apache Storm
  2. STORM-2482

Refactor the Storm auto credential plugins to be more usable


    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.0.0, 1.2.0
    • Component/s: None
    • Labels:


      Currently, the auto credential plugins are part of the respective external modules like storm-hdfs, storm-hbase etc. If users want to use it, they need to place the jars (storm-hdfs, storm-hbase) and its dependencies into ext lib. Currently these plugins does not accept any hadoop configuration programatically. These are set by placing config files like hdfs-site.xml in the class path and this does not scale well nor does it allow users to connect and fetch tokens from different clusters (say two different name nodes) with a single topology.

      To make the auto cred plugins more usable,

      1. Refactor the AutoHdfs, AutoHbase etc into a separate storm external module (say storm-autocreds). This jars along with its dependencies can be packaged and extracted to a folder like lib-autocreds which can be loaded into the class path when storm runs in secure mode (e.g. by setting STORM_EXT_CLASSPATH). The required plugins would be loaded by nimubs/workers based on the user configuration in storm.yaml.

      2. Modify the plugins to accept "configKeys" via topology config. "configKeys" would be a list of string "keys" that the user would pass in the topology config.

         // for hdfs
         topoConf.set("hdfsCredentialsConfigKeys", Arrays.asList(new String[] {"cluster1Key", "cluster2Key"}));
         // put respective config map for the config keys,
         topoConf.set("cluster1Key", configMap1);
         topoConf.set("cluster2Key", configMap2);

      This way we can support credentials from multiple clusters.

      3. During topology submission, nimbus invokes "populateCredentials". If "configKeys" are specified, the plugins will login to hadoop for each config key and fetch the credentials (delegation tokens) and store it with respective keys in the storm cluster state. Cluster state already stores the credentials as a Map<String, String> so no changes are needed there.

      The workers will download the credentials and invoke "populateSubject". The plugin would populate all the credentials for all the configured "configKeys" into the subject. Similar steps would be performed during "updateSubject"

      4. Nimbus periodically invokes "renew" credentials. At this time the plugin will fetch the credentials for the configured "configKeys" (i.e. for the users from different clusters) and renew the respective credentials.

      5. The user could specify different principal and keytab within the config key map so that the plugin will use appropriate user for logging into the respective cluster.

      We also need to enhance the auto cred by adding more plugins. E.g for hbase and kafka delegation tokens which are missing currently (this could be a separate JIRAs).


          Issue Links



              • Assignee:
                arunmahadevan Arun Mahadevan
                arunmahadevan Arun Mahadevan
              • Votes:
                0 Vote for this issue
                3 Start watching this issue


                • Created:

                  Time Tracking

                  Original Estimate - Not Specified
                  Not Specified
                  Remaining Estimate - 0h
                  Time Spent - 3h 10m
                  3h 10m