Major Description
Currently, the auto credential plugins are part of the respective external modules like storm-hdfs, storm-hbase etc. If users want to use it, they need to place the jars (storm-hdfs, storm-hbase) and its dependencies into ext lib. Currently these plugins does not accept any hadoop configuration programatically. These are set by placing config files like hdfs-site.xml in the class path and this does not scale well nor does it allow users to connect and fetch tokens from different clusters (say two different name nodes) with a single topology.
To make the auto cred plugins more usable,
1. Refactor the AutoHdfs, AutoHbase etc into a separate storm external module (say storm-autocreds). This jars along with its dependencies can be packaged and extracted to a folder like lib-autocreds which can be loaded into the class path when storm runs in secure mode (e.g. by setting STORM_EXT_CLASSPATH). The required plugins would be loaded by nimubs/workers based on the user configuration in storm.yaml.
2. Modify the plugins to accept "configKeys" via topology config. "configKeys" would be a list of string "keys" that the user would pass in the topology config.
// for hdfs
topoConf.set("hdfsCredentialsConfigKeys", Arrays.asList(new String[] {"cluster1Key", "cluster2Key"}));
// put respective config map for the config keys,
topoConf.set("cluster1Key", configMap1);
topoConf.set("cluster2Key", configMap2);
This way we can support credentials from multiple clusters.
3. During topology submission, nimbus invokes "populateCredentials". If "configKeys" are specified, the plugins will login to hadoop for each config key and fetch the credentials (delegation tokens) and store it with respective keys in the storm cluster state. Cluster state already stores the credentials as a Map<String, String> so no changes are needed there.
The workers will download the credentials and invoke "populateSubject". The plugin would populate all the credentials for all the configured "configKeys" into the subject. Similar steps would be performed during "updateSubject"
4. Nimbus periodically invokes "renew" credentials. At this time the plugin will fetch the credentials for the configured "configKeys" (i.e. for the users from different clusters) and renew the respective credentials.
5. The user could specify different principal and keytab within the config key map so that the plugin will use appropriate user for logging into the respective cluster.
We also need to enhance the auto cred by adding more plugins. E.g for hbase and kafka delegation tokens which are missing currently (this could be a separate JIRAs).
Attachments
Issue Links
- is depended upon by
-
AMBARI-21045 Enable Storm's AutoTGT configs in secure mode
-
- Resolved
-
- links to
