Uploaded image for project: 'Apache Storm'
  1. Apache Storm
  2. STORM-2482

Refactor the Storm auto credential plugins to be more usable

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 2.0.0, 1.2.0
    • None
    • None

    Description

      Currently, the auto credential plugins are part of the respective external modules like storm-hdfs, storm-hbase etc. If users want to use it, they need to place the jars (storm-hdfs, storm-hbase) and its dependencies into ext lib. Currently these plugins does not accept any hadoop configuration programatically. These are set by placing config files like hdfs-site.xml in the class path and this does not scale well nor does it allow users to connect and fetch tokens from different clusters (say two different name nodes) with a single topology.

      To make the auto cred plugins more usable,

      1. Refactor the AutoHdfs, AutoHbase etc into a separate storm external module (say storm-autocreds). This jars along with its dependencies can be packaged and extracted to a folder like lib-autocreds which can be loaded into the class path when storm runs in secure mode (e.g. by setting STORM_EXT_CLASSPATH). The required plugins would be loaded by nimubs/workers based on the user configuration in storm.yaml.

      2. Modify the plugins to accept "configKeys" via topology config. "configKeys" would be a list of string "keys" that the user would pass in the topology config.

         // for hdfs
         topoConf.set("hdfsCredentialsConfigKeys", Arrays.asList(new String[] {"cluster1Key", "cluster2Key"}));
         // put respective config map for the config keys,
         topoConf.set("cluster1Key", configMap1);
         topoConf.set("cluster2Key", configMap2);
        
      

      This way we can support credentials from multiple clusters.

      3. During topology submission, nimbus invokes "populateCredentials". If "configKeys" are specified, the plugins will login to hadoop for each config key and fetch the credentials (delegation tokens) and store it with respective keys in the storm cluster state. Cluster state already stores the credentials as a Map<String, String> so no changes are needed there.

      The workers will download the credentials and invoke "populateSubject". The plugin would populate all the credentials for all the configured "configKeys" into the subject. Similar steps would be performed during "updateSubject"

      4. Nimbus periodically invokes "renew" credentials. At this time the plugin will fetch the credentials for the configured "configKeys" (i.e. for the users from different clusters) and renew the respective credentials.

      5. The user could specify different principal and keytab within the config key map so that the plugin will use appropriate user for logging into the respective cluster.

      We also need to enhance the auto cred by adding more plugins. E.g for hbase and kafka delegation tokens which are missing currently (this could be a separate JIRAs).

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            arunmahadevan Arun Mahadevan
            arunmahadevan Arun Mahadevan
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 3h 10m
                3h 10m

                Slack

                  Issue deployment