Details
-
Improvement
-
Status: Closed
-
Major
-
Resolution: Fixed
-
2.0.0, 2.1.0
-
None
Description
UserGroupInformation (UGI) loginUserFromKeytab should be used only once in a process to login to hdfs because it overrides static fields. Also loginUserFromKeytabAndReturnUGI function is also problematic according to hadoop team. So the correct way to connect to hdfs is to use UGI loginUserFromKeytab once and only in a process.
Currently we only use HDFS in hdfs-blobstore. It works correctly. But the code is implemented in the hdfs-blobstore plugin. It will be problematic if we want to add another plugin which also needs to connect to HDFS.
So the proposal here is to remove the login piece of code from hdfs-blobstore. And explicitly login to hdfs once and only once when the server (nimbus, supervisor, etc) or the client (storm cli command) launches. It can guarantee one login per process.
The plugins like hdfs-blobstore then simply assume the process has already logged in.
