Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-3336

FileSystem memory leak in DelegationTokenRenewer



    • Bug
    • Status: Closed
    • Critical
    • Resolution: Fixed
    • None
    • 2.7.0
    • resourcemanager
    • None
    • Reviewed


      FileSystem memory leak in DelegationTokenRenewer.
      Every time DelegationTokenRenewer#obtainSystemTokensForUser is called, a new FileSystem entry will be added to FileSystem#CACHE which will never be garbage collected.
      This is the implementation of obtainSystemTokensForUser:

        protected Token<?>[] obtainSystemTokensForUser(String user,
            final Credentials credentials) throws IOException, InterruptedException {
          // Get new hdfs tokens on behalf of this user
          UserGroupInformation proxyUser =
          Token<?>[] newTokens =
              proxyUser.doAs(new PrivilegedExceptionAction<Token<?>[]>() {
                public Token<?>[] run() throws Exception {
                  return FileSystem.get(getConfig()).addDelegationTokens(
                    UserGroupInformation.getLoginUser().getUserName(), credentials);
          return newTokens;

      The memory leak happened when FileSystem.get(getConfig()) is called with a new proxy user.
      Because createProxyUser will always create a new Subject.
      The calling sequence is
      FileSystem.get(getConfig())=>FileSystem.get(getDefaultUri(conf), conf)=>FileSystem.CACHE.get(uri, conf)=>FileSystem.CACHE.getInternal(uri, conf, key)=>FileSystem.CACHE.map.get(key)=>createFileSystem(uri, conf)

      public static UserGroupInformation createProxyUser(String user,
            UserGroupInformation realUser) {
          if (user == null || user.isEmpty()) {
            throw new IllegalArgumentException("Null user");
          if (realUser == null) {
            throw new IllegalArgumentException("Null real user");
          Subject subject = new Subject();
          Set<Principal> principals = subject.getPrincipals();
          principals.add(new User(user));
          principals.add(new RealUser(realUser));
          UserGroupInformation result =new UserGroupInformation(subject);
          return result;

      FileSystem#Cache#Key.equals will compare the ugi

            Key(URI uri, Configuration conf, long unique) throws IOException {
              scheme = uri.getScheme()==null?"":uri.getScheme().toLowerCase();
              authority = uri.getAuthority()==null?"":uri.getAuthority().toLowerCase();
              this.unique = unique;
              this.ugi = UserGroupInformation.getCurrentUser();
            public boolean equals(Object obj) {
              if (obj == this) {
                return true;
              if (obj != null && obj instanceof Key) {
                Key that = (Key)obj;
                return isEqual(this.scheme, that.scheme)
                       && isEqual(this.authority, that.authority)
                       && isEqual(this.ugi, that.ugi)
                       && (this.unique == that.unique);
              return false;        

      UserGroupInformation.equals will compare subject by reference.

        public boolean equals(Object o) {
          if (o == this) {
            return true;
          } else if (o == null || getClass() != o.getClass()) {
            return false;
          } else {
            return subject == ((UserGroupInformation) o).subject;

      So in this case, every time createProxyUser and FileSystem.get(getConfig()) are called, a new FileSystem will be created and a new entry will be added to FileSystem.CACHE.


        1. YARN-3336.000.patch
          2 kB
          Zhihai Xu
        2. YARN-3336.001.patch
          2 kB
          Zhihai Xu
        3. YARN-3336.002.patch
          5 kB
          Zhihai Xu
        4. YARN-3336.003.patch
          5 kB
          Zhihai Xu
        5. YARN-3336.004.patch
          4 kB
          Zhihai Xu

        Issue Links



              zxu Zhihai Xu
              zxu Zhihai Xu
              0 Vote for this issue
              11 Start watching this issue