Uploaded image for project: 'Apache Storm'
  1. Apache Storm
  2. STORM-3494

Use UserGroupInformation to login to HDFS only once per process

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.0, 2.1.0
    • Fix Version/s: 2.2.0
    • Component/s: None

      Description

      UserGroupInformation (UGI) loginUserFromKeytab should be used only once in a process to login to hdfs because it overrides static fields. Also loginUserFromKeytabAndReturnUGI function is also problematic according to hadoop team. So the correct way to connect to hdfs is to use UGI loginUserFromKeytab once and only in a process.

      Currently we only use HDFS in hdfs-blobstore. It works correctly. But the code is implemented in the hdfs-blobstore plugin. It will be problematic if we want to add another plugin which also needs to connect to HDFS.

      So the proposal here is to remove the login piece of code from hdfs-blobstore. And explicitly login to hdfs once and only once when the server (nimbus, supervisor, etc) or the client (storm cli command) launches. It can guarantee one login per process.

      The plugins like hdfs-blobstore then simply assume the process has already logged in.

        Attachments

          Activity

            People

            • Assignee:
              ethanli Ethan Li
              Reporter:
              ethanli Ethan Li
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 3h 20m
                3h 20m