Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-3336

FileSystem memory leak in DelegationTokenRenewer

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.7.0
    • Component/s: resourcemanager
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      FileSystem memory leak in DelegationTokenRenewer.
      Every time DelegationTokenRenewer#obtainSystemTokensForUser is called, a new FileSystem entry will be added to FileSystem#CACHE which will never be garbage collected.
      This is the implementation of obtainSystemTokensForUser:

        protected Token<?>[] obtainSystemTokensForUser(String user,
            final Credentials credentials) throws IOException, InterruptedException {
          // Get new hdfs tokens on behalf of this user
          UserGroupInformation proxyUser =
              UserGroupInformation.createProxyUser(user,
                UserGroupInformation.getLoginUser());
          Token<?>[] newTokens =
              proxyUser.doAs(new PrivilegedExceptionAction<Token<?>[]>() {
                @Override
                public Token<?>[] run() throws Exception {
                  return FileSystem.get(getConfig()).addDelegationTokens(
                    UserGroupInformation.getLoginUser().getUserName(), credentials);
                }
              });
          return newTokens;
        }
      

      The memory leak happened when FileSystem.get(getConfig()) is called with a new proxy user.
      Because createProxyUser will always create a new Subject.
      The calling sequence is
      FileSystem.get(getConfig())=>FileSystem.get(getDefaultUri(conf), conf)=>FileSystem.CACHE.get(uri, conf)=>FileSystem.CACHE.getInternal(uri, conf, key)=>FileSystem.CACHE.map.get(key)=>createFileSystem(uri, conf)

      public static UserGroupInformation createProxyUser(String user,
            UserGroupInformation realUser) {
          if (user == null || user.isEmpty()) {
            throw new IllegalArgumentException("Null user");
          }
          if (realUser == null) {
            throw new IllegalArgumentException("Null real user");
          }
          Subject subject = new Subject();
          Set<Principal> principals = subject.getPrincipals();
          principals.add(new User(user));
          principals.add(new RealUser(realUser));
          UserGroupInformation result =new UserGroupInformation(subject);
          result.setAuthenticationMethod(AuthenticationMethod.PROXY);
          return result;
        }
      

      FileSystem#Cache#Key.equals will compare the ugi

            Key(URI uri, Configuration conf, long unique) throws IOException {
              scheme = uri.getScheme()==null?"":uri.getScheme().toLowerCase();
              authority = uri.getAuthority()==null?"":uri.getAuthority().toLowerCase();
              this.unique = unique;
              this.ugi = UserGroupInformation.getCurrentUser();
            }
            public boolean equals(Object obj) {
              if (obj == this) {
                return true;
              }
              if (obj != null && obj instanceof Key) {
                Key that = (Key)obj;
                return isEqual(this.scheme, that.scheme)
                       && isEqual(this.authority, that.authority)
                       && isEqual(this.ugi, that.ugi)
                       && (this.unique == that.unique);
              }
              return false;        
            }
      

      UserGroupInformation.equals will compare subject by reference.

        public boolean equals(Object o) {
          if (o == this) {
            return true;
          } else if (o == null || getClass() != o.getClass()) {
            return false;
          } else {
            return subject == ((UserGroupInformation) o).subject;
          }
        }
      

      So in this case, every time createProxyUser and FileSystem.get(getConfig()) are called, a new FileSystem will be created and a new entry will be added to FileSystem.CACHE.

        Attachments

        1. YARN-3336.004.patch
          4 kB
          Zhihai Xu
        2. YARN-3336.003.patch
          5 kB
          Zhihai Xu
        3. YARN-3336.002.patch
          5 kB
          Zhihai Xu
        4. YARN-3336.001.patch
          2 kB
          Zhihai Xu
        5. YARN-3336.000.patch
          2 kB
          Zhihai Xu

          Issue Links

            Activity

              People

              • Assignee:
                zxu Zhihai Xu
                Reporter:
                zxu Zhihai Xu
              • Votes:
                0 Vote for this issue
                Watchers:
                11 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: